4 ms·
While there is much to suggest that sleep is important for learning, why contrastive learning in particular? It seems to me that the "deactive IO" argument coul
by bjourne 2mo ago
While there is much to suggest that sleep is important for learning, why contrastive learning in particular? It seems to me that the "deactive IO" argument could apply to any learning technique. Similar to how railroad maintenance is carried out at night because it is too unsafe to do during the day when trains are whizzing by.
- smallmancontrov 2mo agoContrastive learning provides an unusually strong push towards an offline solution, even though the offline solution is expensive. Road maintenance isn't a good analogy; it _can_ be done "online" (with traffic, during the day) and in fact _is_ often done online, despite the pressure to do it online being less than "road workers will be eaten by monsters if they venture out between 8pm and 6am." This is the opposite circumstance we see with sleep. Animals always sleep despite the steep cost. If we want to fix the analogy, we could imagine a world in which all roadwork was done between 8pm and 6am despite workers regularly getting eaten by black-furred creatures of the night. Wouldn't that beg the question "why not do it during the day?" If you learned that every city and state and country did the same crazy overnight work schedules despite losing workers to them, wouldn't you suspect that there was a strong hidden reason?