3 ms·

Other countries need to invest collectively in open alternatives, and AI must be considered critical infrastructure rather than a commercial venture. Building s

by fny 7mo ago

Other countries need to invest collectively in open alternatives, and AI must be considered critical infrastructure rather than a commercial venture. Building small firms to compete against behemoths will not accomplish that. And by open I mean open weights AND open training pipelines.

PaulHoule 7mo ago
It's complicated. The obvious path is the Grok path. That is, anybody with a big pile of money can read some papers and hire some people and make a model which is at or near the frontier. Beating current models by a hair at current benchmarks is not as hard as it looks because you will be building the system to beat those benchmarks from the beginning. [1] Six months or a year later people will start to realize that you're not really improving or making progress though because that's something entirely different. Now real advances in the long term are going to come out of smaller companies working on things like https://en.wikipedia.org/wiki/Mamba_(deep_learning_architecture) https://en.wikipedia.org/wiki/Mamba_(deep_learning_architect... [2] Like the frontier models are just too expensive to do the experimental work which will lead to advances in the science, the science is going to advance through work on little models whereas companies like OpenAI and Anthropic are very committed to maximize the performance of their existing systems in the short term and it is the intense competition that will keep them in an "Innovator's Dilemma" situation where their customers are going to reject anything really new which doesn't perform the same. [3] [1] ... and even if you don't cheat the model will cheat for you [2] ... not necessarily that one in particular, but something like that [3] companies like Microsoft that "disrupt themselves" ignoring their customers are afraid of an Innovator's Dilemma situation but are paradoxically not stuck in it because they are monopolies who can force their customers to do something they don't like.

by fny 7mo ago

PaulHoule 7mo ago

It's complicated. The obvious path is the Grok path. That is, anybody with a big pile of money can read some papers and hire some people and make a model which is at or near the frontier. Beating current models by a hair at current benchmarks is not as hard as it looks because you will be building the system to beat those benchmarks from the beginning. [1] Six months or a year later people will start to realize that you're not really improving or making progress though because that's something entirely different. Now real advances in the long term are going to come out of smaller companies working on things like https://en.wikipedia.org/wiki/Mamba_(deep_learning_architecture) https://en.wikipedia.org/wiki/Mamba_(deep_learning_architect... [2] Like the frontier models are just too expensive to do the experimental work which will lead to advances in the science, the science is going to advance through work on little models whereas companies like OpenAI and Anthropic are very committed to maximize the performance of their existing systems in the short term and it is the intense competition that will keep them in an "Innovator's Dilemma" situation where their customers are going to reject anything really new which doesn't perform the same. [3] [1] ... and even if you don't cheat the model will cheat for you [2] ... not necessarily that one in particular, but something like that [3] companies like Microsoft that "disrupt themselves" ignoring their customers are afraid of an Innovator's Dilemma situation but are paradoxically not stuck in it because they are monopolies who can force their customers to do something they don't like.