3 ms·
My long-standing observation has been that while nature may abhor a vacuum, she also really, really loves sigmoids. That performance vs. training data is not l
by hprotagonist 2y ago
My long-standing observation has been that while nature may abhor a vacuum, she also really, really loves sigmoids.
That performance vs. training data is not linear, but logarithmic, doesn't exactly come as a surprise.
- bilsbie 2y agoEvery exponential is really an s curve?
- hprotagonist 2y agoin physical systems, it’s very often the case!
- mr_mitm 2y agoPretty much always. One exception might be the expansion of the universe
- jskherman 2y agoSymmetries in Nature strike again! It's just like in Noether's theorem.
- eru 2y agoNoether's theorem has nothing to do with S-curves.
- ixaxaar 2y agoThe physical world has limits, that's why sigmoids everywhere.
- CuriouslyC 2y agoThe question that is unanswered, is the logarithmic performance improvement the result of better sampling of the underlying distribution over time, or related to just doing more training with slight variations to effectively regularize the model so it generalizes better? If it's the former, that indicates that we could achieve small models that are every bit as smart as large ones in limited domains, and if that's the case, it radically changes the landscape of what an optimal model architecture is. I suspect from the success of Phi3 that it is in fact the former.