4 ms·
One other thing to add is large-scale RLHF. Big Tech can pay literally hundreds of technically-sophisticated people throughout the world (e.g. college grads in
by nicklecompte 2y ago
One other thing to add is large-scale RLHF. Big Tech can pay literally hundreds of technically-sophisticated people throughout the world (e.g. college grads in developing countries) to improve LLM performance on all sorts of specific problems. It is not a viable way to get AGI, but it means your LLM can learn tons of useful tricks that real people might want, and helps avoid embarrassing "mix broken glass into your baby formula" mistakes. (Obviously it is not foolproof.)
I suspect GPT-4's "secret sauce" in terms of edging out competitors is that OpenAI is better about managing data contractors than the other folks. Of course it's a haze of NDAs to learn specifics, and clearly the contractors are severely underpaid compared to OpenAI employees/executives. But a lone genius with a platinum credit card can't create a new world-class LLM without help from others.
- stephc_int13 2y agoYes, this is the secret sauce and the moat. Not as easy as buying more compute with unlimited budget. … built on the back of a disposable workforce… There is something grim and dystopian, thinking about the countless small hands feeding the machine.
- factormeta 2y ago>There is something grim and dystopian, thinking about the countless small hands feeding the machine. Dystopian indeed, this is pretty much how Manhattan Project and CERN were done, with many independent contractors doing different parts, and only a few has the overview. A page out of corporate management book, it very much allows concentration of power in the hands of a few.
- pagekicker 2y agoVery generous to compare to Manhattan Project or CERN.
- factormeta 2y ago>Comparing LLMs to the Manhattan Project based on budget alone is stupid and arrogant Just want to clarify. The comparison to Manhattan Project or CERN is referencing "the countless small hands feeding the machine." In projects such as these, roles and jobs are divided into small parts, that people who are working on it don't really see the forest from the tree, and that only a few that has the picture of the whole project.
- fragmede 2y agodon't buy into the hype, but when Facebook has spent around as much on GPUs as the Manhattan project (but not the Apollo program), the comparison kinda makes itself. https://twitter.com/emollick/status/1786213463456448900 https://twitter.com/emollick/status/1786213463456448900 $22 in 2008 -> $33 today https://data.bls.gov/cgi-bin/cpicalc.pl?cost1=22&year1=200801&year2=202404 https://data.bls.gov/cgi-bin/cpicalc.pl?cost1=22&year1=20080...
- ladzoppelin 2y agoI read this last week and its terrifying. If the world lets Facebook become an AI leader its on us as we all know how that story will play out.
- thelittleone 2y agoWe must summon a fellowship of the AI ring with one hobbit capable of withstanding the corrupting allure of it all.
- kreeben 2y agoDon't torment the hobbits! Send the eagles right away!
- nicklecompte 2y agoThe Big Dig (Boston highway overhaul) cost $22bn in 2024 dollars. The Three Gorges dam cost $31bn. These are expensive infrastructure projects (including the infrastructure for data centers). It doesn't say anything about how important they are for society. Comparing LLMs to the Manhattan Project based on budget alone is stupid and arrogant. The comparison only "makes itself" because Ethan Mollick is a childish and unscientific person.
- wodenokoto 2y agoSince when is CERN a dystopian project?
- nicklecompte 2y agoBig Government Socialism won't let you build your own 25km-circumference particle accelerator. Bureaucrats make you fill out "permits" and "I-9s for the construction workers instead of hiring undocumented day laborers." I am wondering if "CERN was pushed on the masses by the few" is an oblique reference to public fears that the LHC would destroy the world.
- bzzzt 2y agoMaybe it's the only way. Companies that don't have that concentrated power will probably fall apart.
- littlestymaar 2y agoThe big difference is that CERN or Manhattan projects where done by local contractors with often more than decent wages, which isn't the case when you pay people from Madagascar a couple dollar a day.
- fire_lake 2y agoHard to defend because once your model is out there other companies can train on its output.
- stephc_int13 2y agoYes, but the output is only one third of the data. You also need the input and the annotations.
- kleton 2y agoOpenAI is heavily relying on Scale AI for training data (contractors).