I'd guess the problem is that the daytime pictures allow for easy feature detection (tree, building etc) but the nightime pictures are washed out- We humans loo

I'd guess the problem is that the daytime pictures allow for easy feature detection (tree, building etc) but the nightime pictures are washed out- We humans look at the daytime picture first, then say "that nighttime picture must have a tree there" which involves feature detection across both pictures (in the training phase) I suspect a neural network better specialized for this task (i.e. that has the data interlaced for both day and nighttime during training) would have no problem feature detecting trees and leaving them unlit.