So it turned out that switching my LDN dose to the morning basically solved my problems with it.