3 ms·

>The hardware is already here. The gaming PCs and laptops we use every day are powerful enough to run these systems if we optimize the software correctly. I ag

by jmclnx 10mo ago

>The hardware is already here. The gaming PCs and laptops we use every day are powerful enough to run these systems if we optimize the software correctly. I agree with this but there is one issue, AFAIK, the languages used do not lend themselves to optimization. And I expect the databases in use have the same issue. It is almost like you need to put optimizations in the hardware kind of like what IBM does with its mainframes for transaction processing. Instead, AI companies is doing the usual 'race to be there first', ignoring about the consequences of the design.

godelski 10mo ago
I don't think it is so much the languages as the algorithms themselves. I'll put it one way, my most cited paper continually got rejected from AI conferences because 1) it wasn't novel enough 2) "why would you want to train a transformer from scratch?". The big reason for most of my citations is because researchers outside ML wanted to use transformers and either tuning or distilling a large model was insufficient (either computationally intractable or their problem didn't transfer learn very well so they had to do lots of training anyways). The ML research community is very focused on scaling. As an example that doesn't (fully) deanonymize me look at how people reacted to things like KAN or MAMBA. Even in HN comments questions about scale are always front and center. Don't get me wrong, scale is an important question, but the context matters. These questions are out of place because they are not giving the new frameworks a chance to scale. As any sane researcher would do, you scale iteratively. To be resource efficient you test your ideas at a small scale and then move up, fixing other problems along the way. It's not even just a matter of resource efficiency, but even for solving the problems. By jumping straight to scale you add a lot of complexity into the mix and make it difficult to decouple the problems. This hyper-fixation on making everything bigger is really hindering us. Yeah, sure there are papers that do make it out (as I even gave examples) but these are still harder for smaller labs to pursue and get through review (review isn't completely blind and we'd be ignorant to ignore the politics that goes on).

by jmclnx 10mo ago

godelski 10mo ago

I don't think it is so much the languages as the algorithms themselves. I'll put it one way, my most cited paper continually got rejected from AI conferences because 1) it wasn't novel enough 2) "why would you want to train a transformer from scratch?". The big reason for most of my citations is because researchers outside ML wanted to use transformers and either tuning or distilling a large model was insufficient (either computationally intractable or their problem didn't transfer learn very well so they had to do lots of training anyways). The ML research community is very focused on scaling. As an example that doesn't (fully) deanonymize me look at how people reacted to things like KAN or MAMBA. Even in HN comments questions about scale are always front and center. Don't get me wrong, scale is an important question, but the context matters. These questions are out of place because they are not giving the new frameworks a chance to scale. As any sane researcher would do, you scale iteratively. To be resource efficient you test your ideas at a small scale and then move up, fixing other problems along the way. It's not even just a matter of resource efficiency, but even for solving the problems. By jumping straight to scale you add a lot of complexity into the mix and make it difficult to decouple the problems. This hyper-fixation on making everything bigger is really hindering us. Yeah, sure there are papers that do make it out (as I even gave examples) but these are still harder for smaller labs to pursue and get through review (review isn't completely blind and we'd be ignorant to ignore the politics that goes on).