6 ms·

What Meta learned from Galactica, the doomed model

112 points by swyx 3y ago

MenhirMike 3y ago
Rookie mistake, it should have been named Galactus. Though of course, even though Galactus has omniscience, it doesn't have futuresight, so perhaps it would still have been doomed.
- atleastoptimal 3y ago
  The all knowing user service provider aggregator
  - pastrami_panda 3y ago
    Sure, but it still doesn't support ISO timestamps: https://github.com/acchiao/galactus/issues/34 https://github.com/acchiao/galactus/issues/34
- codethief 3y ago
  Reference for everyone else: https://m.youtube.com/watch?v=y8OnoxKotPQ https://m.youtube.com/watch?v=y8OnoxKotPQ
harryquach 3y ago
Gives me a chuckle every time https://youtu.be/y8OnoxKotPQ?si=xZsDfNwAivhdOBGD https://youtu.be/y8OnoxKotPQ?si=xZsDfNwAivhdOBGD
- bagels 3y ago
  I think I was in this meeting at Facebook. Yikes.
devjab 3y ago
Maybe I’m not understanding the article correctly, but what was the lesson? That Facebook should have told people it would lie, similar to how OpenAI did? I’m actually interested in what the internal lessons for Facebook would’ve been to see their product so replaced by a similar product only two weeks later, but unless I’m misunderstanding everything, the article doesn’t really seem to touch in it. At least not beyond the fact that people still want the model, and, that it’s part of llama and Facebooks new push for more open models.
- thaumasiotes 3y ago
  They do state the lesson. It just isn't interesting: > “The gap between the expectation, and where the research was, was too big.” > Overall, Pineau said, “If I was to do it today, we would just manage the release.”
riffraff 3y ago
Maybe I misremember, but isn't the fault about expectations vs reality due to how this was introduced? I had the feeling Meta released this with great fanfare saying "do science with it" so of course there was backlash when it was shown to hallucinate. I mean the website still sounds like it https://galactica.org/mission/ https://galactica.org/mission/
- mst 3y ago
  "We got the introduction completely wrong" seems to've been one of the takeaways according to the article. I suspect the underlying problem was the limitations being obvious to the research team and them being excited that it was -less- limited than previous attempts in the same direction and not realising how people who didn't find the limitations it still had obvious would react. All very predictable in hindsight but because they didn't see it as a 'real' launch I don't think people with enough of an outside view were involved, and ... I think we've all been there with the "but that's obvious" problem, so (a) you're right (b) I'm still kinda sympathetic.
dr_kiszonka 3y ago
This model had exactly the feature I have been looking for the last three days! What I need is something to semi-reliably "Translate Python code to Math." Sadly, it doesn't seem available anymore and I can't find anything I could use. (There are only tools for translation in the opposite direction, i.e., math to code.)
- _kb 3y ago
  Which is interesting (at least my simpleton perspective). The mathematical form is designed to communicate the essence of an idea rather than exact process, so seems much more suited to LLM sourced expression.
  - dr_kiszonka 3y ago
    Your comment made me realize that we probably have much more publicly available code than math, and we have only started writing code en masse in the last few decades. This, in turn, dovetails nicely with your point that math communicates the essence.
janalsncm 3y ago
I think the reason is because expectations were different. ChatGPT was released to the general public for general use. Galactica was really only noticed by the egg head press and egg head professionals (like myself) who rightly identified the failure of these statistical language models to have a grounding in facts. People whose job it is to notice details are going to push your model to the limit. But it’s not like ChatGPT was better. If I recall, RLHF actually made hallucinations slightly worse. Even today OpenAI wouldn’t claim to be able to accurately summarize papers. It’s just that there was an endless supply of “write X in the style of Y” that was like catnip for journalists and kept them busy for months.
- otabdeveloper4 3y ago
  ChatGPT relies very much on confirmation bias to do its magic. You ask something trivial you already know (and could have probably Googled in 15 seconds), you get back something glib and smooth in reply and you are wowed by how smart GPT seems to be. It is significantly less impressive when you ask something you don't already know and can't Google easily.
  - withinboredom 3y ago
    I actually love bing bot when searching for something very nuanced ("no, not like that ... [explanation]"). I haven't used Google's AI search yet; why learn a new tool?
  - danielbln 3y ago
    How would you even know that it's less impressive if you don't know the subject matter?
    - rafaelmn 3y ago
      Because it can be easy to test the result even if you don't know how to come up with a solution ? If I asked you to build a light switch I don't need to know anything about EE to see that the light turns on/off when I flip the switch. LLMs often fail to provide functional solutions for non trivial stuff (not impressive) - and once you know more about the domain, you usually figure out that the light switch will likely burn down your house in few hours because it's overheating.
quickthrower2 3y ago
All knowing user info provider service aggregator?
deleted 3y ago
[deleted]
a-dub 3y ago
i think the only real mistake they made was taking twitter drama seriously.
- corethree 3y ago
  Exactly. There was the exact same push back on chatgpt but they just didn't care.
- moffkalast 3y ago
  So say we all.
Topfi 3y ago
I feel that the article doesn't really cover what aspects made Galactica more open to criticism and led to its reception, outside of pointing to a lack of awareness for hallucinations in the wider public, which were equally present during the initial launch of ChatGPT (as most users never encountered OpenAIs disclosures on that topic, heck, we had lawyers relying on these models months after hallucinations as a phenomenon had been more widely covered). Maybe I missed it, but honestly, the article doesn't contain much information about what Meta learned specifically from Galactica (over general reception to LLMs across the industry) other than that misadventure being the claimed reason for Llama 1's release strategy, which has already been superseded by being less restrictive on Llama 2 access. For me, the main reasons why Galactica was more heavily criticized than ChatGPT were twofold. For one, the more scientific aspirations that this model had from the outset, proclaimed boldly in the associated paper [0] made any factual errors or problematic output far less justifiable, compared to a model that, through its branding and design, was more conversational in nature. That, I feel, leads directly into the second important reason. The initial launch of ChatGPT, very cleverly, targeted users of all backgrounds, making it more likely that the initial interaction the majority of users had with the model was more conversational and humorous than the type of systematic picking apart the more scientifically inclined crowd tends to partake in. This led to initial coverage of ChatGPT being filled with amazement by laypeople, drowning out a lot of criticism and more measured reactions in a sea of hype, whereas Galactica was consistently bombarded with less favorable reactions. Overall, I still feel that the approach they took in building Galactica is one that should be explored further, and I am somewhat saddened that more focused LLMs have become less favored by researchers. I remain hopeful that as we explore the limitations of universal, conversational models, there will be a resurgence of more specialized LLMs similar to Galactica. [0] https://galactica.org/static/paper.pdf https://galactica.org/static/paper.pdf
- epups 3y ago
  I think you are spot on, a LLM focused on science and medicine has a much higher bar to pass when it comes to accuracy. I tried Galactica when it came out, and I have to say that subjectively at least the results looked much inferior to what the benchmarks suggest. In the paper they claim to be substantially better than GPT-3 on their larger models, while in my personal experience even some generous queries produced garbage output. I cannot remember whether the version available at the site was the largest model, however.
pmontra 3y ago
The problem was that they told us that it was an AI for science and when you confabulate science you definitely do a very bad impression.
bagels 3y ago
I remember the outcry when this thing was released, thinking, did we learn nothing from Microsoft's "racist" Tay bot? How did they not anticipate that journalists and twitter were going to try to make it parrot things that make Facebook look bad?
mplewis 3y ago
This article is absolutely whitewashing how bad the Galactica project is. On day one, people were using the scientific paper generator to make fake papers and wiki pages with real-looking citations supporting racist, sexist, homophobic, and genocidal ends. This thing didn’t actually do much aside from hallucinate about the world in a scientific voice. The bot was allegedly supposed to help scientists explore connections between different complex theoretical fields, but it didn’t have a concept of what was real or practical, and the citations were all fake. What value did they think was going to come from this aside from misinformation? At least ChatGPT could do your homework for you.
moomin 3y ago
From the article, it seems that the lesson they learned was that public feedback is bad, which was the wrong lesson to learn.
seydor 3y ago
Anyone know if they plan to continue work on galactica? openai sucks at reviewing papers
supermatt 3y ago
Huh? Is this implying that Galactica was there “first” and yet meta lost out because they wouldn’t tolerate hallucinations whereas OpenAI does? Does venturebeat realise there was GPT3 before ChatGPT?
Havoc 3y ago
> A week after its release, Llama’s model weights were leaked by someone who posted the download link to 4chan You filled in a form and downloaded them… Not really a leak in my books
smrtinsert 3y ago
Not related to galactus the all knowing user provider service aggregator
msjet 3y ago
Galaxus did nothing wrong. They released a language model to the one community well equipped to handle the limitations of such models. It was advertised as experimental. I think it is far better than releasing a model to the public at large, which in aggregate is less able to understand that hallucinations can occur. They got trashed because (i) it came from meta and (ii) it got targeted by (maybe well-meaning, yet...) short-sighted scientists on Twitter. On the first point, big companies are scrutinised more and tend to be criticised more, it is unlikely that any big company could have released chatgpt without a huge backlash. I'm not sure "move fast and break things" would still work for meta today. OpenAI had a privileged position for thay kind of moonshot. Then there are the well documented Twitter clashes between Yann Lecun and opponents, which I believe made Galactica the perfect target when it was advertised on Twitter by Lecun. It felt a bit ridiculous at the time, like a wave of identity politics reaching science: my enemy did that so I'm going to work hard on trashing it regardless, and I'll double down on my opinions irrespective of new evidence.
- dekhn 3y ago
  Disagree. They claimed it did useful stuff; many people (including ex-scientists like me who are not short-sighted) saw the examples and how easily it was made to produce inaccurate information. In science, truthiness matters a lot and saying you have a LLM that can summarize scientific articles, users can reasonably expect that the LLM produces accurate information at a much higher rate.
- kgwgk 3y ago
  > They released a language model to the one community well equipped to handle the limitations of such models. It was advertised as experimental. [...] Then there are the well documented Twitter clashes between Yann Lecun and opponents, which I believe made Galactica the perfect target when it was advertised on Twitter by Lecun. This is how it was advertised on Twitter by Lecun: "Type a text and http://galactica.ai http://galactica.ai will generate a paper with relevant references, formulas, and everything." If he wanted to advertise it as experimental and make clear its limitations he could have written something like "Type a text and http://galactica.ai http://galactica.ai will generate something that looks like a paper with made up references, incorrect formulas, and anything."

112 points by swyx 3y ago

Rookie mistake, it should have been named Galactus. Though of course, even though Galactus has omniscience, it doesn't have futuresight, so perhaps it would still have been doomed.

atleastoptimal 3y ago
The all knowing user service provider aggregator
- pastrami_panda 3y ago
  Sure, but it still doesn't support ISO timestamps: https://github.com/acchiao/galactus/issues/34 https://github.com/acchiao/galactus/issues/34
codethief 3y ago
Reference for everyone else: https://m.youtube.com/watch?v=y8OnoxKotPQ https://m.youtube.com/watch?v=y8OnoxKotPQ

harryquach 3y ago

Gives me a chuckle every time https://youtu.be/y8OnoxKotPQ?si=xZsDfNwAivhdOBGD https://youtu.be/y8OnoxKotPQ?si=xZsDfNwAivhdOBGD

bagels 3y ago
I think I was in this meeting at Facebook. Yikes.

devjab 3y ago

Maybe I’m not understanding the article correctly, but what was the lesson? That Facebook should have told people it would lie, similar to how OpenAI did? I’m actually interested in what the internal lessons for Facebook would’ve been to see their product so replaced by a similar product only two weeks later, but unless I’m misunderstanding everything, the article doesn’t really seem to touch in it. At least not beyond the fact that people still want the model, and, that it’s part of llama and Facebooks new push for more open models.

thaumasiotes 3y ago
They do state the lesson. It just isn't interesting: > “The gap between the expectation, and where the research was, was too big.” > Overall, Pineau said, “If I was to do it today, we would just manage the release.”

riffraff 3y ago

Maybe I misremember, but isn't the fault about expectations vs reality due to how this was introduced? I had the feeling Meta released this with great fanfare saying "do science with it" so of course there was backlash when it was shown to hallucinate. I mean the website still sounds like it https://galactica.org/mission/ https://galactica.org/mission/

mst 3y ago
"We got the introduction completely wrong" seems to've been one of the takeaways according to the article. I suspect the underlying problem was the limitations being obvious to the research team and them being excited that it was -less- limited than previous attempts in the same direction and not realising how people who didn't find the limitations it still had obvious would react. All very predictable in hindsight but because they didn't see it as a 'real' launch I don't think people with enough of an outside view were involved, and ... I think we've all been there with the "but that's obvious" problem, so (a) you're right (b) I'm still kinda sympathetic.

dr_kiszonka 3y ago

This model had exactly the feature I have been looking for the last three days! What I need is something to semi-reliably "Translate Python code to Math." Sadly, it doesn't seem available anymore and I can't find anything I could use. (There are only tools for translation in the opposite direction, i.e., math to code.)

_kb 3y ago
Which is interesting (at least my simpleton perspective). The mathematical form is designed to communicate the essence of an idea rather than exact process, so seems much more suited to LLM sourced expression.
- dr_kiszonka 3y ago
  Your comment made me realize that we probably have much more publicly available code than math, and we have only started writing code en masse in the last few decades. This, in turn, dovetails nicely with your point that math communicates the essence.

janalsncm 3y ago

I think the reason is because expectations were different. ChatGPT was released to the general public for general use. Galactica was really only noticed by the egg head press and egg head professionals (like myself) who rightly identified the failure of these statistical language models to have a grounding in facts. People whose job it is to notice details are going to push your model to the limit. But it’s not like ChatGPT was better. If I recall, RLHF actually made hallucinations slightly worse. Even today OpenAI wouldn’t claim to be able to accurately summarize papers. It’s just that there was an endless supply of “write X in the style of Y” that was like catnip for journalists and kept them busy for months.

otabdeveloper4 3y ago
ChatGPT relies very much on confirmation bias to do its magic. You ask something trivial you already know (and could have probably Googled in 15 seconds), you get back something glib and smooth in reply and you are wowed by how smart GPT seems to be. It is significantly less impressive when you ask something you don't already know and can't Google easily.
- withinboredom 3y ago
  I actually love bing bot when searching for something very nuanced ("no, not like that ... [explanation]"). I haven't used Google's AI search yet; why learn a new tool?
- danielbln 3y ago
  How would you even know that it's less impressive if you don't know the subject matter?
  - rafaelmn 3y ago
    Because it can be easy to test the result even if you don't know how to come up with a solution ? If I asked you to build a light switch I don't need to know anything about EE to see that the light turns on/off when I flip the switch. LLMs often fail to provide functional solutions for non trivial stuff (not impressive) - and once you know more about the domain, you usually figure out that the light switch will likely burn down your house in few hours because it's overheating.

quickthrower2 3y ago

All knowing user info provider service aggregator?

deleted 3y ago

[deleted]

a-dub 3y ago

i think the only real mistake they made was taking twitter drama seriously.

corethree 3y ago
Exactly. There was the exact same push back on chatgpt but they just didn't care.
moffkalast 3y ago
So say we all.

Topfi 3y ago

I feel that the article doesn't really cover what aspects made Galactica more open to criticism and led to its reception, outside of pointing to a lack of awareness for hallucinations in the wider public, which were equally present during the initial launch of ChatGPT (as most users never encountered OpenAIs disclosures on that topic, heck, we had lawyers relying on these models months after hallucinations as a phenomenon had been more widely covered). Maybe I missed it, but honestly, the article doesn't contain much information about what Meta learned specifically from Galactica (over general reception to LLMs across the industry) other than that misadventure being the claimed reason for Llama 1's release strategy, which has already been superseded by being less restrictive on Llama 2 access. For me, the main reasons why Galactica was more heavily criticized than ChatGPT were twofold. For one, the more scientific aspirations that this model had from the outset, proclaimed boldly in the associated paper [0] made any factual errors or problematic output far less justifiable, compared to a model that, through its branding and design, was more conversational in nature. That, I feel, leads directly into the second important reason. The initial launch of ChatGPT, very cleverly, targeted users of all backgrounds, making it more likely that the initial interaction the majority of users had with the model was more conversational and humorous than the type of systematic picking apart the more scientifically inclined crowd tends to partake in. This led to initial coverage of ChatGPT being filled with amazement by laypeople, drowning out a lot of criticism and more measured reactions in a sea of hype, whereas Galactica was consistently bombarded with less favorable reactions. Overall, I still feel that the approach they took in building Galactica is one that should be explored further, and I am somewhat saddened that more focused LLMs have become less favored by researchers. I remain hopeful that as we explore the limitations of universal, conversational models, there will be a resurgence of more specialized LLMs similar to Galactica. [0] https://galactica.org/static/paper.pdf https://galactica.org/static/paper.pdf

epups 3y ago
I think you are spot on, a LLM focused on science and medicine has a much higher bar to pass when it comes to accuracy. I tried Galactica when it came out, and I have to say that subjectively at least the results looked much inferior to what the benchmarks suggest. In the paper they claim to be substantially better than GPT-3 on their larger models, while in my personal experience even some generous queries produced garbage output. I cannot remember whether the version available at the site was the largest model, however.

pmontra 3y ago

The problem was that they told us that it was an AI for science and when you confabulate science you definitely do a very bad impression.

bagels 3y ago

I remember the outcry when this thing was released, thinking, did we learn nothing from Microsoft's "racist" Tay bot? How did they not anticipate that journalists and twitter were going to try to make it parrot things that make Facebook look bad?

mplewis 3y ago

This article is absolutely whitewashing how bad the Galactica project is. On day one, people were using the scientific paper generator to make fake papers and wiki pages with real-looking citations supporting racist, sexist, homophobic, and genocidal ends. This thing didn’t actually do much aside from hallucinate about the world in a scientific voice. The bot was allegedly supposed to help scientists explore connections between different complex theoretical fields, but it didn’t have a concept of what was real or practical, and the citations were all fake. What value did they think was going to come from this aside from misinformation? At least ChatGPT could do your homework for you.

moomin 3y ago

From the article, it seems that the lesson they learned was that public feedback is bad, which was the wrong lesson to learn.

seydor 3y ago

Anyone know if they plan to continue work on galactica? openai sucks at reviewing papers

supermatt 3y ago

Huh? Is this implying that Galactica was there “first” and yet meta lost out because they wouldn’t tolerate hallucinations whereas OpenAI does? Does venturebeat realise there was GPT3 before ChatGPT?

Havoc 3y ago

> A week after its release, Llama’s model weights were leaked by someone who posted the download link to 4chan You filled in a form and downloaded them… Not really a leak in my books

smrtinsert 3y ago

Not related to galactus the all knowing user provider service aggregator

msjet 3y ago

Galaxus did nothing wrong. They released a language model to the one community well equipped to handle the limitations of such models. It was advertised as experimental. I think it is far better than releasing a model to the public at large, which in aggregate is less able to understand that hallucinations can occur. They got trashed because (i) it came from meta and (ii) it got targeted by (maybe well-meaning, yet...) short-sighted scientists on Twitter. On the first point, big companies are scrutinised more and tend to be criticised more, it is unlikely that any big company could have released chatgpt without a huge backlash. I'm not sure "move fast and break things" would still work for meta today. OpenAI had a privileged position for thay kind of moonshot. Then there are the well documented Twitter clashes between Yann Lecun and opponents, which I believe made Galactica the perfect target when it was advertised on Twitter by Lecun. It felt a bit ridiculous at the time, like a wave of identity politics reaching science: my enemy did that so I'm going to work hard on trashing it regardless, and I'll double down on my opinions irrespective of new evidence.

dekhn 3y ago
Disagree. They claimed it did useful stuff; many people (including ex-scientists like me who are not short-sighted) saw the examples and how easily it was made to produce inaccurate information. In science, truthiness matters a lot and saying you have a LLM that can summarize scientific articles, users can reasonably expect that the LLM produces accurate information at a much higher rate.
kgwgk 3y ago
> They released a language model to the one community well equipped to handle the limitations of such models. It was advertised as experimental. [...] Then there are the well documented Twitter clashes between Yann Lecun and opponents, which I believe made Galactica the perfect target when it was advertised on Twitter by Lecun. This is how it was advertised on Twitter by Lecun: "Type a text and http://galactica.ai http://galactica.ai will generate a paper with relevant references, formulas, and everything." If he wanted to advertise it as experimental and make clear its limitations he could have written something like "Type a text and http://galactica.ai http://galactica.ai will generate something that looks like a paper with made up references, incorrect formulas, and anything."