Could this be an artifact of just not reshuffling the dataset and how the weight regime is? What if you reversed the dataset in the second epoch, under the memo

Could this be an artifact of just not reshuffling the dataset and how the weight regime is? What if you reversed the dataset in the second epoch, under the memory hypothesis the training loss would not plummet if it has not learnt anything during the epoch after the first 10%. Yes? The report mentions there is no reshuffling: > We’re not re-shuffling the dataset at the start of the epoch, so those first batches of the second epoch are when the learning rate was still warming up.