4 ms·
I feel that the article doesn't really cover what aspects made Galactica more open to criticism and led to its reception, outside of pointing to a lack of aware
by Topfi 3y ago
I feel that the article doesn't really cover what aspects made Galactica more open to criticism and led to its reception, outside of pointing to a lack of awareness for hallucinations in the wider public, which were equally present during the initial launch of ChatGPT (as most users never encountered OpenAIs disclosures on that topic, heck, we had lawyers relying on these models months after hallucinations as a phenomenon had been more widely covered). Maybe I missed it, but honestly, the article doesn't contain much information about what Meta learned specifically from Galactica (over general reception to LLMs across the industry) other than that misadventure being the claimed reason for Llama 1's release strategy, which has already been superseded by being less restrictive on Llama 2 access.
For me, the main reasons why Galactica was more heavily criticized than ChatGPT were twofold. For one, the more scientific aspirations that this model had from the outset, proclaimed boldly in the associated paper [0] made any factual errors or problematic output far less justifiable, compared to a model that, through its branding and design, was more conversational in nature.
That, I feel, leads directly into the second important reason. The initial launch of ChatGPT, very cleverly, targeted users of all backgrounds, making it more likely that the initial interaction the majority of users had with the model was more conversational and humorous than the type of systematic picking apart the more scientifically inclined crowd tends to partake in. This led to initial coverage of ChatGPT being filled with amazement by laypeople, drowning out a lot of criticism and more measured reactions in a sea of hype, whereas Galactica was consistently bombarded with less favorable reactions.
Overall, I still feel that the approach they took in building Galactica is one that should be explored further, and I am somewhat saddened that more focused LLMs have become less favored by researchers. I remain hopeful that as we explore the limitations of universal, conversational models, there will be a resurgence of more specialized LLMs similar to Galactica.
[0] https://galactica.org/static/paper.pdf https://galactica.org/static/paper.pdf
- epups 3y agoI think you are spot on, a LLM focused on science and medicine has a much higher bar to pass when it comes to accuracy. I tried Galactica when it came out, and I have to say that subjectively at least the results looked much inferior to what the benchmarks suggest. In the paper they claim to be substantially better than GPT-3 on their larger models, while in my personal experience even some generous queries produced garbage output. I cannot remember whether the version available at the site was the largest model, however.
- ipsum2 3y agoGPT-3 was prior to ChatGPT. Since it was a non-instruction tuned model, the output was quite bad unless prompted precisely.
- danielbln 3y agoGPT-3 was instruct finetuned a while before ChatGPT was even released (see InstructGPT paper and announcement). What ChatGPT went through is RLHF to align its responses to a more conversational level, but you could give GPT-3 (davinci) instructions long before ChatGPT.
- visarga 3y ago> the initial interaction the majority of users had with the model was more conversational and humorous than the type of systematic picking apart the more scientifically inclined crowd tends to partake in If I remember correctly chatGPT was grilled and prompt hacked for months, it was the Twitter obsession of the day.