4 ms·
Reposting comment from last time since I'm still curious: This is really fun to see. I love toy experiments like this. I see that each plot is always using the
by mchinen 3y ago
Reposting comment from last time since I'm still curious:
This is really fun to see. I love toy experiments like this. I see that each plot is always using the same initialization of weights, which presumably makes it possible to have more smoothness between each pixel. I also would guess it's using the same random seed for training (shuffling data).
I'd be curious to know what the plots would look like with a different randomness/shuffling of each pixel's dataset. I'd guess for the high learning rates it would be too noisy, but you might see fractal behavior at more typical and practical learning rates. You could also do the same with the random initialization of each dataset. This would get at if the chaotic boundary also exists in more practical use cases.
- thomashop 3y agoexactly. if they always use the same initialization seed then this isn't very surprising. one would have to do many runs for each point in the grid and average them or something. but i didn't read the paper so maybe they did.
- londons_explore 3y agoI think if you used a random seed for weights and training data order, and reran the experiment enough times to average out the noise, then the resulting charts would then be smooth with no fractal patterns.
- baq 3y agoAn interesting conjecture, well worth a paper in response.
- catlifeonmars 3y agoDo you consider the random seed (or by extension the randomized initial weights) a hyperparameter?