4 ms·
If you work with model architecture and read papers, how could not know there are a flood of new ideas? Only few yield interesting results though. I kind of wo
by strangescript 1y ago
If you work with model architecture and read papers, how could not know there are a flood of new ideas? Only few yield interesting results though.
I kind of wonder if libraries like pytorch have hurt experimental development. So many basic concepts no one thinks about anymore because they just use the out of the box solutions. And maybe those solutions are great and those parts are "solved", but I am not sure. How many models are using someone else's tokenizer, or someone else's strapped on vision model just to check a box in the model card?
- kevmo314 1y agoThe people who don't think about such things probably wouldn't develop experimentally sans pytorch either.
- thenaturalist 1y agoThat's been the very normal way of the human world. When the foundation layer at a given moment doesn't yield an ROI on intellectual exploration - say because you can overcompensate with VC funded raw compute and make more progess elsewhere -, few(er) will go there. But inevitably, as other domains reach diminishing returns, bright minds will take a look around where significant gains for their effort can be found. And so will the next generation of PyTorch or foundational technologies evolve.
- mardifoufs 1y agoYeah and even then, it's been like ~ 2-3 years since the last rather major Architectural improvement, major enough for a lot of people to actually hear about it and use it daily. I think some people lose perspective on how short of a time frame 3 years is. But yes, there's a ton of interesting and useful stuff (beyond datasets and data related improvements) going on right now, and I'm not even talking about LLMs. I don't do anything related to LLM and even then I still see tons of new stuff popping up regularly.
- _giorgio_ 1y agoIt's the opposite. Frameworks like pytorch are really flexible. You can implement any architecture, and if it's not enough, you can learn CUDA. Keras it's the opposite, it's probably like you describe things.
- delifue 1y agoThe hardware(GPU)'s architectural limitations may slow research more than PyTorch. The hardware lottery https://hardwarelottery.github.io/ https://hardwarelottery.github.io/
- mikewarot 1y agoI've been trying to get BitGrid [1] a hardware lottery ticket for a decade now. [1] https://hn.algolia.com/?dateRange=all&page=0&prefix=false&query=BitGrid%20&sort=byPopularity&type=all https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...