20 ms·

Snowflake Arctic Instruct (128x3B MoE), largest open source model

315 points by cuuupid 2y ago

pointlessone 2y ago
4k context. With a sliding window in the works. Is this for chats only?
cs702 2y ago
Wow, 128 experts in a single model. That's a lot more than everyone else. The Snowflake team has a blog post explaining why they did that: https://www.snowflake.com/blog/arctic-open-efficient-foundation-language-models-snowflake/ https://www.snowflake.com/blog/arctic-open-efficient-foundat... But the most interesting aspect about this, for me, is that every tech company seems to be coming out with a free open model claiming to be better than the others at this thing or that thing. The number of choices is overwhelming. As of right now, Huggingface is hosting over 600,000 different pretrained open models. Lots of money has been forever burned training or finetuning all those open models. Even more money has been forever burned training or finetuning all the models that have not been publicly released. It's like a giant bonfire, with Nvidia supplying most of the (very expensive) chopped wood. Who's going to recoup all that investment? When? How? What's the rationale for releasing all these models to the public? Do all these tech companies know something we don't? Why are they doing this? --- EDIT: Changed "0.6 million" to "600,000," which seems clearer. Added "or finetuning".
- throwup238 2y ago
  > Who's going to recoup all that investment? When? How? What's the long-term strategy AI of all these tech companies? Do they know something we don't? The first droid armies will rapidly recoup the cost when the final wars for world domination begin…
  - rvnx 2y ago
    Even before that, elections are coming end of the year, chat bots are great for telling whom to vote for. 2020's elections costed 15B USD in total, so we can't afford to lose (we are the good guys, right ?)
    - N0b8ez 2y ago
      How will the LLMs be used for this? They can't solve captchas, and they're not smart enough to navigate the internet by themselves. All they do is generate text.
      - kaibee 2y ago
        Transformers can definitely solve captchas. Not sure why you think otherwise.
        N0b8ez 2y ago
        So captchas are obsolete now?
        kaibee 2y ago
        For a while now, even before the latest AI models. Paid services exist (~2$/1k solves https://deathbycaptcha.com/ https://deathbycaptcha.com/)
- guessmyname 2y ago
  Why 0.6 million and not +600k ?
  - cs702 2y ago
    You're right. I changed it. Thanks!
- lewi 2y ago
  > over 0.6 million What a peculiar way to say: 600,000
  - cs702 2y ago
    You're right. I changed it. Thanks!
- cornholio 2y ago
  The model seems to be "build something fast, get users, engagement, and venture capital, hope you can grow fast enough to still be around after the Great AI cull". > offers over 0.6 million different pretrained open models. One estimate I saw was that training GPT3 released 500 tons of CO2 back in 2020. Out of those 600k models, at least hundreds are of a comparable complexity. I can only hope building large models does not become analogous to cryptocoin speculation, where resources are forever burned only in a quest to attract the greater fool. Those startups and researchers would better invest in smarter algorithms and approaches instead of trying to outpolute OpenAI, Meta and Microsoft.
  - cs702 2y ago
    > "build something fast, get users, engagement, and venture capital, hope you can grow fast enough to still be around after the Great AI cull" Snowflake is a publicly traded company with a market cap of $50B and $4B of cash in hand. It has no need for venture capital money. It looks like a case of "Look Ma! I can do it too!"
  - ReptileMan 2y ago
    >One estimate I saw was that training GPT3 released 500 tons of CO2 back in 2020 So absolute nothing in the grand scheme of things?
    - throwup238 2y ago
      Yeah that’s the annual emissions for only 100 people at the global average or about 30 Americans.
    - margalabargala 2y ago
      That's the amount that would be released by burning 50,000 gallons of gas, which is about that ten typical cars will burn throughout their entire lifespan. Done once, I agree, that's very little. But if each of those 600,000 other models used that much (or even a tenth that much), then that now becomes impactful. Releasing 500 tons of CO2 600,000 times over would amount to about 1% of all human global annual emissions.
- mlsu 2y ago
  Far fewer than 600,000 of those are pretrained. Most are finetuned which is much easier. You can finetune a 7B model on gamer cards. There is basically the big guys that everyone's heard of (google, meta, microsoft/openAI, and anthropic) and then a handful of smaller players who are training foundation models mostly so that they can prove to VCs that they are capable of doing so -- to acquire more funding/access to compute so that they may eventually dethrone openAI and take a piece of the multi-billion dollar "enterprise AI" market for themselves. Below that, there is a frothing ocean of mostly 7B finetunes created mostly by individuals who want to jailbreak base models for... reasons, plus the occasional research group. The most oddball one I have seen is the databricks LLM which seems to have been an exercise of pure marketing. Those I suspect will disappear when the bubble deflates a bit.
  - cs702 2y ago
    > an exercise of pure marketing Yes. Great choice of words. A lot of non-frontier models look like "an exercise of pure marketing" to me. Still, I fail to see the rationale for telling the world, "Look at us! We can do it too!"
    - mlsu 2y ago
      Mid-level managers at a lot of companies still have no clue what LLMs are or how they work. These companies (like databricks) want to have their salespeople upsell such companies on "business AI." They have the base model in their back pocket just in case one of the customers in the room has heard the name Andrej Karpathy before and starts asking questions about how good their AI solution is... they can point to their model and its benchmarks to say "we know what we are doing with this AI stuff." It's just standard marketing stuff which works right now because of how difficult it is to actually objectively benchmark LLMs.
  - theturtletalks 2y ago
    Yep, seems like every company is taking a longshot on a AI project. Even companies like Databricks (MosaicML) and Vercel (v0 and ai.sdk) are seeing if they can take a piece of this every growing pie. Snowflake and the like are training and releasing new models because they intend to integrate the AI into their existing product down the line. Why not use and fine-tune an existing model? Their in-grown model maybe better suited for their product. This can also fail like Bloomberg's financial model being inferior to GPT-4, but these companies have to try.
- TrueDuality 2y ago
  Most of those are fine tuned variants of open base models and shouldn't be included in the "every tech company" thing you're trying to communicate. Most of those are researcher or engineers learning how to work with these models, or are training them on specific data sets to improve their effectiveness in a particular task. These fine tunes are not a huge amount of compute, most of them are doing these trainings on a single personal machine over a day or so of effort, NOT the six+ months across a massive cluster it takes to make a good base model. That isn't wasted effort either. We need to know how to use these tools effectively, they're not going away. It's a very reductionist and inaccurate view of the world you're peddling in that comment.
- ganzuul 2y ago
  Money is for accounting. AI is a new accountant. Therefore money no longer is what it was.
- a13n 2y ago
  Seems like capitalism is doing its thing here. The potential future revenue from having the best model is presumably in the trillions.
  - bugbuddy 2y ago
    L0L Trillions ROFL
  - sangnoir 2y ago
    > The potential future revenue from having the best model is presumably in the trillions. I heard this winner-takes-all spiel before - only last time, it was about Uber or Tesla[1] robo-taxis making car ownership obsolete. Uber has since exited the self-driving business, Cruise is on hold/unwinding and the whole self-driving bubble has mostly deflated, and most of the startups are long gone, despite the billions invested in the self-driving space. Waymo is the only company with robo-taxis, albeit in only 2 tiny markets and many years away from general availability. 1. Tesla is making robo-taxi noises once more, and again, to juice investor sentiment.
    - a13n 2y ago
      Uber and Tesla are valued at 150B and 500B respectively, I'd say in terms of an ROI on deploying large amounts of capital these are both huge success stories. No investment in an emerging market is a sure thing, it's an educated guess. You have to take a lot of swings to occasionally hit a homerun, and investing in AI seems like the most plausible swing to make at this time.
      - sangnoir 2y ago
        I didn't claim there's no positive ROI. I only noted that the breathlessly promised "trillion+ dollar self-driving market" failed to materialize. I suspect the AI market will have a similar trajectory in the next decade: no actual AGI - maybe one company still plugging away at it, a couple of very successful companies whose core competencies don't include AI, but with billions in market cap, and a lot of failed startups littering the way there.
- analyte123 2y ago
  At a bare minimum, training and releasing a model like this builds critical skills in their engineering workforce that can't really be done any other way for now. It also requires compilation of a training dataset, which is not only another critical human skill, but also potentially a secret sauce if it turns out to give your model specific behaviors or skills. A big one is that it shows investors, partners, and future recruits that you are both willing and capable to work on frontier technology. Hard to put a price on this, but it is important. For the rest of us, it turns out you can use this bestiary of public models, mixing pieces of models with their own secret sauce together to create something superior than any of them [1]. [1] https://sakana.ai/evolutionary-model-merge/ https://sakana.ai/evolutionary-model-merge/
  - ankit219 2y ago
    These bigger companies are releasing open source models for publicity. For Databricks and Snowflake, both want enterprise customers, and want to show they can handle swathes of data and orchestration jobs, what better way to show that than by training a model. The pretraining part is done on a GPU but everything before that is managed on the Snowflake infra or Databricks. Databricks' website does focus heavily on this.[1] I am speculating here, they would use their own OSS models to create a proprietary version which does one thing well. Answering questions for customers based on their own data. It's not an easy problem to solve as it seemed initially given enterprises need high reliability. Need models which are good at tool use, and can be grounded well. They could have done it on an oss model, but only now we have Llama-3 which is trained to make tool use easy. (Tool use as in function calling and use of stuff like OpenAI's code interpreter) [1]: https://www.databricks.com/product/data-intelligence-platform https://www.databricks.com/product/data-intelligence-platfor...
- modeless 2y ago
  These projects all started a long time ago, I expect, and they're all finishing now. Now that there are so many models, people will hopefully change focus from training new duplicate language models to exploring more interesting things. Multimodal, memory, reasoning.
- jrm4 2y ago
  This seems to me to be the simple story of "capitalism, having learned from the past, undertands that free/open source is actually advantageous for the little guys." Which is to say, "everyone" knows that this stuff has a lot of potential. Everyone is also used to what often happens in tech, which is outrageous winner-take-all scale effects. Everyone ALSO knows that there's almost certainly little MARGINAL difference between what the big guys will be able to do and and what the little guys can do on their own ESPECIALLY if they essentially 'pool their knowledge.' So, I suppose it's the whole industry collectively and subconsciously preventing e.g. OpenAI/ChatGPT becoming the Microsoft of AI.
  - squigz 2y ago
    > This seems to me to be the simple story of "capitalism, having learned from the past, undertands that free/open source is actually advantageous for the little guys." This seems rather generous.
    - jrm4 2y ago
      Yeah, I don't mean it to sound that generous, as in "capitalism likes little guys." More like "little guys, or even literally any 'guy' that isn't dominant in this space -- which tends toward dominance -- have learned, perhaps counterintuitively, that free/open source is best for their own greedy interests." :)
- seydor 2y ago
  I am not worried. Someone will make a search engine to find the model that knows your answer . It will be called Altavista or Lycos or something
- barkingcat 2y ago
  Training LLM is the cryptobro pivot.
- DowagerDave 2y ago
  Snowflake has a pretty good story in this space: "Your data is already in our cloud, so governance and use is a solved problem. Now use our AI (and burn credits)". This is a huge pain-point if you're thinking about ML with your (probably private) data. It's less clear if this entices companies to move INTO Snowflake IMO And streamlit, if you're as old as me, looks an awful lot like a MS-Access application for today. Again, it lives in the database, runs on a Snowflake warehouse and consumes credits, which is their revenue engine.
  - hnthrowaway9812 2y ago
    Snowflake could have the same story by hosting Llama 3 which is probably more efficient/better.
    - throwaway120487 2y ago
      Snowflake hosts a couple models: https://docs.snowflake.com/en/user-guide/snowflake-cortex/llm-functions#availability https://docs.snowflake.com/en/user-guide/snowflake-cortex/ll...
- Onavo 2y ago
  > Who's going to recoup all that investment? When? How? Hype and jumping on the bandwagon are perfectly good reasons for a business. There's no business without risk. This is the cost of doing business which you want to explore greenfield projects.
- _flux 2y ago
  And huggingface is hosting (randomly assuming 8-64 GB per model) 5..40 PB of models for free? That's generous of them. Or can the models share data? Ollama seems to have some ability to do that.
- temuze 2y ago
  In the short-term, these kinds of investments can hype up a stock and create a small bump. However, in the long-term, as the hype dies down, so will the stock prices. At the end of the day, I think it will be a transfer of wealth from shareholders to Nvidia and power companies.
  - peteradio 2y ago
    > as the hype dies down, so will the stock prices. *Depending on govt interventions
  - LordDragonfang 2y ago
    I just wish that AMD (and, pie in the sky, Intel) had gotten their shit together enough that these flaming dumptrucks full of money would have actually resulted in a competitive GPU market. Honestly, Zuckerburg (seemingly the only CEO willing to actually invest in an open AI ecosystem for the obvious benefits it brings them) should just invest a few million into hiring a few real firmware hackers to port all the ML CUDA code into an agnostic layer that AMD can build to.
    - hnthrowaway9812 2y ago
      Groq seems to be well positioned to give Nvidia a run for their money, actually.
- blackeyeblitzar 2y ago
  > What's the rationale for releasing all these models to the public? Do all these tech companies know something we don't? Why are they doing this? It’s mostly marketing for the company to appear to be modern. If you aren’t differentiated and if LLMs aren’t core to your business model, then there’s no loss from releasing weights. In other cases it is commoditizing something that would otherwise be valuable for competitors. But most of those 600K models aren’t high performers and don’t have large training budgets, and aren’t part of the “race”.
- richardw 2y ago
  It diminishes the story that Databricks is the default route to privately trained models on your own data. Databricks jumped on the LLM bandwagon really quickly to good effect. Now every enterprise must at least consider Snowflake, and especially their existing clients who need to defend decisions to board members. It also means they build large scale rails necessary to use Snowflake for training and can market such at every release.
  - BPA0 2y ago
    I don't think that you understand Databricks. Databricks gives you the tools to train, tune or build RAG models. Snowflake doesn't. Having said that, I'm a big fan of Llama-3 at the moment.
    - buttaphingas 2y ago
      Snowflake has some examples of that: https://www.snowflake.com/blog/easy-secure-llm-inference-retrieval-augmented-generation-rag-cortex/ https://www.snowflake.com/blog/easy-secure-llm-inference-ret... https://quickstarts.snowflake.com/guide/asking_questions_to_your_own_documents_with_snowflake_cortex/index.html#0 https://quickstarts.snowflake.com/guide/asking_questions_to_... And I assume these new models would be now available for embedding. Is this different to what Databricks offers?
chessgecko 2y ago
This is the sparsest model thats been put out in a while (maybe ever, kinda forget the shapes of googles old sparse models). This probably wont be a great tradeoff for chat servers, but could be good for local stuff if you have 512GB of ram with your cpu.
- imachine1980_ 2y ago
  it performs worst than 8b llama 3 so you probably don't need that much.
  - deleted 2y ago
    [deleted]
  - coder543 2y ago
    Where do you see that? This comparison[0] shows it outperforming Llama-3-8B on 5 out of 6 benchmarks. I'm not going to claim that this model looks incredible, but it's not that easily dismissed for a model that has the compute complexity of a 17B model. [0]: https://www.snowflake.com/wp-content/uploads/2024/04/table-3-4-2048x808.png https://www.snowflake.com/wp-content/uploads/2024/04/table-3...
- coder543 2y ago
  It has 480B parameters total, apparently. You would only need 512GB of RAM if you were running at 8-bit. It could probably fit into 256GB at 4-bit, and 4-bit quantization is broadly accepted as a good trade-off these days. Still... that's a lot of memory. EDIT: This[0] confirms 240GB at 4-bit. [0]: https://github.com/ggerganov/llama.cpp/issues/6877#issue-2261479840 https://github.com/ggerganov/llama.cpp/issues/6877#issue-226...
  - refulgentis 2y ago
    Yeah, and usually GPU RAM, unless you enjoy waiting for a minute for filling the context :(
  - kaibee 2y ago
blackeyeblitzar 2y ago
Let’s stop using terms like open source falsely. The model isn’t open source, it is open weights. It’s good that the license for the weights is Apache, but for this model to be “truly open” they must release training data and source code under an OSI approved license. Otherwise it’s just misleading marketing. So far it seems like Snowflake will release some blog posts and “cookbooks”, whatever that means, but not actual training source code. Only the inference code is open source here, which is uninteresting.
- deleted 2y ago
  [deleted]
- deleted 2y ago
  [deleted]
- jerrygenser 2y ago
  > they must release training data and source code under an OSI approved license The source code is also apache 2.0
  - blackeyeblitzar 2y ago
    Snowflake has only released the inference code - meaning the code you need to "run" the model. So if you take the weights they have released (which is the model that is a result of training), you can host the weights and inference code, and feed prompts to it, to get answers. But you don't have the actual source code you need to produce the weights in the first place. As an example of what open source actually means for LLMs, you can look at what AI2 does with their OLMo model (https://allenai.org/olmo https://allenai.org/olmo), where each model that they release comes with: > Full training data used for these models, including code that produces the training data, from AI2’s Dolma, and WIMBD for analyzing pretraining data. > Full model weights, training code, training logs, training metrics in the form of Weights & Biases logs, and inference code. > 500+ checkpoints per model, from every 1000 steps during the training process, available as revisions on HuggingFace. > Evaluation code under the umbrella of AI2’s Catwalk and Paloma. > Fine-tuning code and adapted models (with Open Instruct) > All code, weights, and intermediate checkpoints are released under the Apache 2.0 License. OLMo is what "truly open" is, while the rest is openwashing and marketing.
zamalek 2y ago
I suppose more and smaller experts would also help reduce over-fitting?
ru552 2y ago
Abnormally large. I don't see the cost/performance numbers going well for this one.
- tosh 2y ago
  It is both cost efficient in training (+ future fine-tuning) as well as inference compared to most other current models. Can you elaborate?
  - ru552 2y ago
    the unquantized model is almost 1tb in size and the benchmarks provided by Snowflake shows performance in the middle of the pack compared to other recent releases.
  - rajhans 2y ago
    We have published some insights here. https://medium.com/snowflake/snowflake-arctic-cookbook-series-exploring-mixture-of-experts-moe-c7d6b8f14d16 https://medium.com/snowflake/snowflake-arctic-cookbook-serie...
ur-whale 2y ago
However big it may be, it still hallucinates very, very badly. I just asked it an economics question and asked it to cite its sources. All the links provided as sources were complete BS. Color me unimpressed.
- mike_hearn 2y ago
  It's intended for SQL generation and similar with cheap fine tuning and inference, not answering general knowledge questions. Their blog post is pretty clear about that. If you just want a chatbot this isn't the model for you. If you want to let non-SQL trained people ask questions of your data, it might be really useful.
  - mritchie712 2y ago
    It's worse at SQL generation than llama3 according to their own post. https://www.snowflake.com/blog/arctic-open-efficient-foundation-language-models-snowflake/#:~:text=on%20enterprise%20intelligence.-,Table%203,-.%20Full%20Metrics%20Table https://www.snowflake.com/blog/arctic-open-efficient-foundat...
    - CharlesW 2y ago
      To be fair, that's comparing their 17B model with the 70B Llama 3 model.
      - ru552 2y ago
        To stay fair, their "17B" model sits at 964GB on your disk and the 70B Llama 3 model sits at 141GB. unquantized GB numbers for both
        CharlesW 2y ago
        Sorry, it sounds like you know a lot more than I do about this, and I'd appreciate it if you'd connect the dots. Is your comment a dig at either Snowflake or Llama? Where are you finding the unquantized size of Llama 3 70B? Isn't it extremely rare to do inference with large unquantized models?
Aissen 2y ago
How much memory would inference take on this type of model? What's the impact of it being an MoE architecture?
bfirsh 2y ago
If you want to have a conversation with it, here's a full chat app: https://arctic.streamlit.app/ https://arctic.streamlit.app/ Official blog post: https://www.snowflake.com/blog/arctic-open-efficient-foundation-language-models-snowflake/ https://www.snowflake.com/blog/arctic-open-efficient-foundat... Weights: https://huggingface.co/Snowflake/snowflake-arctic-instruct https://huggingface.co/Snowflake/snowflake-arctic-instruct
- leblancfg 2y ago
  Wow that is *so fast*, and from a little testing writes both rather decent prose and Python.
- pixelesque 2y ago
  I guess the chat app is under quite a bit of load? I keep getting error traceback "responses" like this: TypeError: This app has encountered an error. The original error message is redacted to prevent data leaks. Full error details have been recorded in the logs (if you're on Streamlit Cloud, click on 'Manage app' in the lower right of your app). Traceback: File "/home/adminuser/venv/lib/python3.11/site-packages/streamlit/runtime/scriptrunner/script_runner.py", line 584, in _run_script exec(code, module.__dict__) File "/mount/src/snowflake-arctic-st-demo/streamlit_app.py", line 101, in <module> full_response = st.write_stream(response)
- hnthrowaway9812 2y ago
  It claims to have a knowledge cut-off of 2021. Not sure if its hallucinating or its true. But when I asked it about the best LLMs it suggested GPT-3, Bert and T5!
PaulHoule 2y ago
It got the right answer for "Who is Tim Bray?" but it got "Who is Worsel the Dragon?" wrong.
- nerpderp82 2y ago
  Looks like they aren't targeting DRGN24 as one of their benchmark suites.
  - PaulHoule 2y ago
    I love getting into arguments with LLMs over whether Worsel is an eastern dragon (in my imagination) or a western dragon (like the bad lensman anime.)
    - nerpderp82 2y ago
      Is Worsel in The Pile? Total aside, but I appreciate your arxiv submissions here. Just because they don't hit the front page, doesn't mean they are seen.
      - PaulHoule 2y ago
        Most LLMs seem to know about Worsel, I've had some who gave my right answer to "Who is Worsel?" but others will say they don't know who I am talking about and will be needed to be cued further. There is a lot of content about sci-fi on the web and all the Doc Smith books are on Canadian Gutenberg now. I found the Jetbrains assistant wasn't so good at coding (I might feel better if it did all the cutting and pasting, addding imports and that kind of stuff which would at least make it less tiresome to watch it bumble) but it is good at science fiction chat, better than all but two people I have known. Glad you like what I post.
mritchie712 2y ago
llama3 narrowly beats arctic at SQL generation (80.2 vs 79.0) and Mixtral 8x22B scored 79.2. You'd think SQL would be the one thing they'd be sure to smoke other models on. 0 - https://www.snowflake.com/blog/arctic-open-efficient-foundation-language-models-snowflake/#:~:text=Table%203.%20Full%20Metrics%20Table.%20Comparing%20Snowflake%20Arctic%20with%20DBRX%2C%20LLAMA%2D3%208B%2C%20LLAMA%2D3%2070B%2C%20Mixtral%208x7B%2C%20Mixtral%208x22B%20(instruction%2Dtuned%20or%20chat%20variants%20if%20available).1 https://www.snowflake.com/blog/arctic-open-efficient-foundat...
- sp332 2y ago
  Yeah but that's a 70B model. You can see on the Inference Efficiency chart that it takes more than 3x as much compute to run it compared to this one.
  - msp26 2y ago
    Most people are vram constrained not compute constrained.
    - kaibee 2y ago
      Cloud providers aren’t though.
    - Manabu-eo 2y ago
      But those people usually have more system RAM than VRAM. At those scales, most people become bandwidth and compute constrained using CPU inference instead of multiple GPUs. In those cases, an MOE with a low number of active parameters is the fastest.
  - karmasimida 2y ago
    But you do need to hold all 128 experts in memory? Or not? Or they simply consider inference efficiency as latency
    - giantrobot 2y ago
1f60c 2y ago
It appears to have limited guardrails. I got it to generate some risqué story and it also told me how to trade onion futures, which is illegal in the US.
- klysm 2y ago
  Why on earth is trading onion futures illegal in the us
  - rbetts 2y ago
    A long history of rapscallions.
    - all2 2y ago
      Well, some kind of scallions anyway.
  - isoprophlex 2y ago
    I looked it up, the story is pretty hilarious. https://en.m.wikipedia.org/wiki/Onion_Futures_Act https://en.m.wikipedia.org/wiki/Onion_Futures_Act
    - klysm 2y ago
      Wow I’m surprised the reaction was to ban futures on just onions specifically due to some market manipulation occurring. Surely this kind of manipulation wasn’t restricted to just onions? It seems incredibly short sighted
      - cryptonector 2y ago
        It's bloody stupid is what it is. Knee-jerking like that is really terrible.
        klysm 2y ago
croes 2y ago
Reminds of the CPU GHz race. The main thing was that the figures were as large and impressive as possible. The benefit was marginal
- salomonk_mur 2y ago
  Yeah, I wouldn't say the benefits were marginal at all. CPUs went from dozens of MHz in the 90's to over 4 GHz nowadays.
  - jasongill 2y ago
    I think what the parent commenter means is that the late 90's race to 1GHz and the early 2000's race for as many GHz as possible turned out to be wasted effort. At the time, ever week it seemed like AMD or Intel would announce a new CPU that was a few MHz faster than the competition, and the assumption among the Slashdot crowd was basically that we'd have 20GHz CPU's by now. Instead, there was a plateau in terms of CPU clock speed and even a regression once we hit about 3-4GHz for desktop CPUs where clock speeds started decreasing but other metrics like core count, efficiency, and other non-clock-based metrics of performance continued to improve. Basically, once we got to about ~2005 and CPU's touched 4GHz, the speeds slowly crept back into the 2.xGHz range for home computers, and we never really saw much (that I've seen) go back far above 4GHz at least for x86/amd64 CPUs. But yet the computers of today are much, much faster than the computers of 2005 (although it doesn't really "feel" like it, of course)
    - heisgone 2y ago
      It wasn't irrational at the time as it was much harder to harnest parallelism then.
    - genewitch 2y ago
      it's been well known (i'd heard it numerous times) that the maximum clock speed of x86 is somewhere <6GHZ. as in "you can make a 10ghz x86, but it would spend half the time idle". Bursting to 5.6ghz (or 5.8 iirc) is possible, but there are constraints that are physical in nature that prevent anything faster. Once single core single threaded CPUs hit ~4GHz the new "frontier" was the core 2 duo, then the core 2 quad, and now we have desktop chips with 16c/32t (and beyond).
krembo 2y ago
IMHO Google Search is doomed. This also impacting ad business, their main cash cow puts them in a very problematic position. Also companies who built their business solely on trained data such as OpenAI need to reinvent themselves.
ukuina 2y ago
So many models fail at basic reasoning. > What weighs more: a pound of feathers or a great british pound? > A pound of feathers and a Great British Pound weigh the same, which is one pound. It works if you add "Think step by step," though.
- ec109685 2y ago
  Another is if you ask which will land first, a bullet fired horizontally or one shot straight down. Derivations of well known problems trip these models up big time. Which bullet lands first, a bullet shot horizontally or one shot towards the ground The bullet shot horizontally will land first, assuming both bullets are fired from the same height and with the same initial velocity. This is because the bullet shot horizontally has a horizontal velocity component that causes it to travel forward while it falls under gravity. The bullet shot towards the ground only has the force of gravity acting on it, causing it to accelerate downwards. Therefore, the bullet shot horizontally will hit the ground later than the one shot towards the ground It’s not even consistent within the same answer! Even if you ask it to think step by step, it gets confused: Bullet shot towards the ground: When a bullet is shot towards the ground, it has both an initial horizontal and vertical velocity components. However, since we're only interested in comparing the time it takes for each bullet to reach the ground, we can focus on their vertical velocities and ignore their horizontal velocities. Let's denote: ( v_y ) as the initial vertical velocity of the second bullet (which is zero in this case) ( t' ) as the time it takes for the second bullet to reach the ground Using the same equation of motion in the vertical direction: [ y = h - \frac{1}{2}g(t')^2 ] Since ( v_y = 0 ), we have: [ t' = \sqrt{\frac{2h}{g}} ]
- coder543 2y ago
  Llama-3 8B got it on the first try: https://imgur.com/a/xQy1828 https://imgur.com/a/xQy1828
  - readams 2y ago
    Gemini and Gemini Advanced both get this right (Edit: I initially thought Gemini got it wrong but I read the answer again and it's actually right!)
deleted 2y ago
[deleted]
imjonse 2y ago
Anything more open than OpenAI models can now call their models 'truly open'. It's good they will have recipes but they also don't seem to want to share the actual data.
- Havoc 2y ago
  Sharing data has too much legal liability given that half of it fell off a truck
vessenes 2y ago
Interesting architecture. For these "large" models, I'm interested in synthesis, fluidity, conceptual flexibility. A sample prompt: "Tell me a love story about two otters, rendered in the FORTH language". Or: "Here's a whitepaper, write me a simulator in python that lets me see the state of these variables, step by step". Or: "Here's a tarball of a program. Write a module that does X, in a unified diff." These are super hard tasks for any LLM I have access to, BTW. Good for testing current edges of capacity. Arctic does not do great on these, unfortunately. It's not willing to make 'the leap' to be creative in FORTH where creativity = storytelling, and tries to redirect me to either getting a story about otters, or telling me things about FORTH. Google made a big deal about emergent sophistication in models as they grew in parameter size with the original PaLM paper, and I wonder if these horizontally-scaled MOE of many small models are somehow architecturally limited. The model weights here, 480B, are sized close to the original PaLM model (540B if I recall). Anyway, more and varied architectures are always welcome! I'd be interested to hear from the Snowflake folks if they think the architecture has additional capacity with more training, or if they think it could improve on recall tasks, but not 'sophistication' type tasks.
- themanmaran 2y ago
  to be fair, gpt did a pretty good job at the otter prompt ``` \ A love story about two otters, Otty and Lutra : init ( -- ) CR ." Two lonely otters lived by a great river." ; : meet ( -- ) CR ." One sunny day, Otty and Lutra met during a playful swim." ; : play ( -- ) CR ." They splashed, dived, and chased each other joyfully." ; ...continued ```
  - vessenes 2y ago
    BTW, I wouldn't rate that very high in that it's trying to put out syntactic FORTH, but not defining verbs or other things which themselves tell the story. Gemini is significantly better last I checked.
- motoxpro 2y ago
  What your evaluating is not what you think it is. You're evaluating the models ability to execute multiple complex steps (think about all of the steps it takes for your second example) not so much if it is capable of doing those things. If you broke it down into 2-3 different prompts it could do all of those things easy.
ofermend 2y ago
This model is great. Jumped immediately to 2nd place on HHEM leaderboard: https://github.com/vectara/hallucination-leaderboard https://github.com/vectara/hallucination-leaderboard

315 points by cuuupid 2y ago

4k context. With a sliding window in the works. Is this for chats only?

Wow, 128 experts in a single model. That's a lot more than everyone else. The Snowflake team has a blog post explaining why they did that: https://www.snowflake.com/blog/arctic-open-efficient-foundation-language-models-snowflake/ https://www.snowflake.com/blog/arctic-open-efficient-foundat... But the most interesting aspect about this, for me, is that every tech company seems to be coming out with a free open model claiming to be better than the others at this thing or that thing. The number of choices is overwhelming. As of right now, Huggingface is hosting over 600,000 different pretrained open models. Lots of money has been forever burned training or finetuning all those open models. Even more money has been forever burned training or finetuning all the models that have not been publicly released. It's like a giant bonfire, with Nvidia supplying most of the (very expensive) chopped wood. Who's going to recoup all that investment? When? How? What's the rationale for releasing all these models to the public? Do all these tech companies know something we don't? Why are they doing this? --- EDIT: Changed "0.6 million" to "600,000," which seems clearer. Added "or finetuning".

throwup238 2y ago
> Who's going to recoup all that investment? When? How? What's the long-term strategy AI of all these tech companies? Do they know something we don't? The first droid armies will rapidly recoup the cost when the final wars for world domination begin…
- rvnx 2y ago
  Even before that, elections are coming end of the year, chat bots are great for telling whom to vote for. 2020's elections costed 15B USD in total, so we can't afford to lose (we are the good guys, right ?)
  - N0b8ez 2y ago
    How will the LLMs be used for this? They can't solve captchas, and they're not smart enough to navigate the internet by themselves. All they do is generate text.
    - kaibee 2y ago
      Transformers can definitely solve captchas. Not sure why you think otherwise.
      - N0b8ez 2y ago
        So captchas are obsolete now?
        kaibee 2y ago
        For a while now, even before the latest AI models. Paid services exist (~2$/1k solves https://deathbycaptcha.com/ https://deathbycaptcha.com/)
guessmyname 2y ago
Why 0.6 million and not +600k ?
- cs702 2y ago
  You're right. I changed it. Thanks!
lewi 2y ago
> over 0.6 million What a peculiar way to say: 600,000
- cs702 2y ago
  You're right. I changed it. Thanks!
cornholio 2y ago
The model seems to be "build something fast, get users, engagement, and venture capital, hope you can grow fast enough to still be around after the Great AI cull". > offers over 0.6 million different pretrained open models. One estimate I saw was that training GPT3 released 500 tons of CO2 back in 2020. Out of those 600k models, at least hundreds are of a comparable complexity. I can only hope building large models does not become analogous to cryptocoin speculation, where resources are forever burned only in a quest to attract the greater fool. Those startups and researchers would better invest in smarter algorithms and approaches instead of trying to outpolute OpenAI, Meta and Microsoft.
- cs702 2y ago
  > "build something fast, get users, engagement, and venture capital, hope you can grow fast enough to still be around after the Great AI cull" Snowflake is a publicly traded company with a market cap of $50B and $4B of cash in hand. It has no need for venture capital money. It looks like a case of "Look Ma! I can do it too!"
- ReptileMan 2y ago
  >One estimate I saw was that training GPT3 released 500 tons of CO2 back in 2020 So absolute nothing in the grand scheme of things?
  - throwup238 2y ago
    Yeah that’s the annual emissions for only 100 people at the global average or about 30 Americans.
  - margalabargala 2y ago
    That's the amount that would be released by burning 50,000 gallons of gas, which is about that ten typical cars will burn throughout their entire lifespan. Done once, I agree, that's very little. But if each of those 600,000 other models used that much (or even a tenth that much), then that now becomes impactful. Releasing 500 tons of CO2 600,000 times over would amount to about 1% of all human global annual emissions.
mlsu 2y ago
Far fewer than 600,000 of those are pretrained. Most are finetuned which is much easier. You can finetune a 7B model on gamer cards. There is basically the big guys that everyone's heard of (google, meta, microsoft/openAI, and anthropic) and then a handful of smaller players who are training foundation models mostly so that they can prove to VCs that they are capable of doing so -- to acquire more funding/access to compute so that they may eventually dethrone openAI and take a piece of the multi-billion dollar "enterprise AI" market for themselves. Below that, there is a frothing ocean of mostly 7B finetunes created mostly by individuals who want to jailbreak base models for... reasons, plus the occasional research group. The most oddball one I have seen is the databricks LLM which seems to have been an exercise of pure marketing. Those I suspect will disappear when the bubble deflates a bit.
- cs702 2y ago
  > an exercise of pure marketing Yes. Great choice of words. A lot of non-frontier models look like "an exercise of pure marketing" to me. Still, I fail to see the rationale for telling the world, "Look at us! We can do it too!"
  - mlsu 2y ago
    Mid-level managers at a lot of companies still have no clue what LLMs are or how they work. These companies (like databricks) want to have their salespeople upsell such companies on "business AI." They have the base model in their back pocket just in case one of the customers in the room has heard the name Andrej Karpathy before and starts asking questions about how good their AI solution is... they can point to their model and its benchmarks to say "we know what we are doing with this AI stuff." It's just standard marketing stuff which works right now because of how difficult it is to actually objectively benchmark LLMs.
- theturtletalks 2y ago
  Yep, seems like every company is taking a longshot on a AI project. Even companies like Databricks (MosaicML) and Vercel (v0 and ai.sdk) are seeing if they can take a piece of this every growing pie. Snowflake and the like are training and releasing new models because they intend to integrate the AI into their existing product down the line. Why not use and fine-tune an existing model? Their in-grown model maybe better suited for their product. This can also fail like Bloomberg's financial model being inferior to GPT-4, but these companies have to try.
TrueDuality 2y ago
Most of those are fine tuned variants of open base models and shouldn't be included in the "every tech company" thing you're trying to communicate. Most of those are researcher or engineers learning how to work with these models, or are training them on specific data sets to improve their effectiveness in a particular task. These fine tunes are not a huge amount of compute, most of them are doing these trainings on a single personal machine over a day or so of effort, NOT the six+ months across a massive cluster it takes to make a good base model. That isn't wasted effort either. We need to know how to use these tools effectively, they're not going away. It's a very reductionist and inaccurate view of the world you're peddling in that comment.
ganzuul 2y ago
Money is for accounting. AI is a new accountant. Therefore money no longer is what it was.
a13n 2y ago
Seems like capitalism is doing its thing here. The potential future revenue from having the best model is presumably in the trillions.
- bugbuddy 2y ago
  L0L Trillions ROFL
- sangnoir 2y ago
  > The potential future revenue from having the best model is presumably in the trillions. I heard this winner-takes-all spiel before - only last time, it was about Uber or Tesla[1] robo-taxis making car ownership obsolete. Uber has since exited the self-driving business, Cruise is on hold/unwinding and the whole self-driving bubble has mostly deflated, and most of the startups are long gone, despite the billions invested in the self-driving space. Waymo is the only company with robo-taxis, albeit in only 2 tiny markets and many years away from general availability. 1. Tesla is making robo-taxi noises once more, and again, to juice investor sentiment.
  - a13n 2y ago
    Uber and Tesla are valued at 150B and 500B respectively, I'd say in terms of an ROI on deploying large amounts of capital these are both huge success stories. No investment in an emerging market is a sure thing, it's an educated guess. You have to take a lot of swings to occasionally hit a homerun, and investing in AI seems like the most plausible swing to make at this time.
    - sangnoir 2y ago
      I didn't claim there's no positive ROI. I only noted that the breathlessly promised "trillion+ dollar self-driving market" failed to materialize. I suspect the AI market will have a similar trajectory in the next decade: no actual AGI - maybe one company still plugging away at it, a couple of very successful companies whose core competencies don't include AI, but with billions in market cap, and a lot of failed startups littering the way there.
analyte123 2y ago
At a bare minimum, training and releasing a model like this builds critical skills in their engineering workforce that can't really be done any other way for now. It also requires compilation of a training dataset, which is not only another critical human skill, but also potentially a secret sauce if it turns out to give your model specific behaviors or skills. A big one is that it shows investors, partners, and future recruits that you are both willing and capable to work on frontier technology. Hard to put a price on this, but it is important. For the rest of us, it turns out you can use this bestiary of public models, mixing pieces of models with their own secret sauce together to create something superior than any of them [1]. [1] https://sakana.ai/evolutionary-model-merge/ https://sakana.ai/evolutionary-model-merge/
- ankit219 2y ago
  These bigger companies are releasing open source models for publicity. For Databricks and Snowflake, both want enterprise customers, and want to show they can handle swathes of data and orchestration jobs, what better way to show that than by training a model. The pretraining part is done on a GPU but everything before that is managed on the Snowflake infra or Databricks. Databricks' website does focus heavily on this.[1] I am speculating here, they would use their own OSS models to create a proprietary version which does one thing well. Answering questions for customers based on their own data. It's not an easy problem to solve as it seemed initially given enterprises need high reliability. Need models which are good at tool use, and can be grounded well. They could have done it on an oss model, but only now we have Llama-3 which is trained to make tool use easy. (Tool use as in function calling and use of stuff like OpenAI's code interpreter) [1]: https://www.databricks.com/product/data-intelligence-platform https://www.databricks.com/product/data-intelligence-platfor...
modeless 2y ago
These projects all started a long time ago, I expect, and they're all finishing now. Now that there are so many models, people will hopefully change focus from training new duplicate language models to exploring more interesting things. Multimodal, memory, reasoning.
jrm4 2y ago
This seems to me to be the simple story of "capitalism, having learned from the past, undertands that free/open source is actually advantageous for the little guys." Which is to say, "everyone" knows that this stuff has a lot of potential. Everyone is also used to what often happens in tech, which is outrageous winner-take-all scale effects. Everyone ALSO knows that there's almost certainly little MARGINAL difference between what the big guys will be able to do and and what the little guys can do on their own ESPECIALLY if they essentially 'pool their knowledge.' So, I suppose it's the whole industry collectively and subconsciously preventing e.g. OpenAI/ChatGPT becoming the Microsoft of AI.
- squigz 2y ago
  > This seems to me to be the simple story of "capitalism, having learned from the past, undertands that free/open source is actually advantageous for the little guys." This seems rather generous.
  - jrm4 2y ago
    Yeah, I don't mean it to sound that generous, as in "capitalism likes little guys." More like "little guys, or even literally any 'guy' that isn't dominant in this space -- which tends toward dominance -- have learned, perhaps counterintuitively, that free/open source is best for their own greedy interests." :)
seydor 2y ago
I am not worried. Someone will make a search engine to find the model that knows your answer . It will be called Altavista or Lycos or something
barkingcat 2y ago
Training LLM is the cryptobro pivot.
DowagerDave 2y ago
Snowflake has a pretty good story in this space: "Your data is already in our cloud, so governance and use is a solved problem. Now use our AI (and burn credits)". This is a huge pain-point if you're thinking about ML with your (probably private) data. It's less clear if this entices companies to move INTO Snowflake IMO And streamlit, if you're as old as me, looks an awful lot like a MS-Access application for today. Again, it lives in the database, runs on a Snowflake warehouse and consumes credits, which is their revenue engine.
- hnthrowaway9812 2y ago
  Snowflake could have the same story by hosting Llama 3 which is probably more efficient/better.
  - throwaway120487 2y ago
    Snowflake hosts a couple models: https://docs.snowflake.com/en/user-guide/snowflake-cortex/llm-functions#availability https://docs.snowflake.com/en/user-guide/snowflake-cortex/ll...
Onavo 2y ago
> Who's going to recoup all that investment? When? How? Hype and jumping on the bandwagon are perfectly good reasons for a business. There's no business without risk. This is the cost of doing business which you want to explore greenfield projects.
_flux 2y ago
And huggingface is hosting (randomly assuming 8-64 GB per model) 5..40 PB of models for free? That's generous of them. Or can the models share data? Ollama seems to have some ability to do that.
temuze 2y ago
In the short-term, these kinds of investments can hype up a stock and create a small bump. However, in the long-term, as the hype dies down, so will the stock prices. At the end of the day, I think it will be a transfer of wealth from shareholders to Nvidia and power companies.
- peteradio 2y ago
  > as the hype dies down, so will the stock prices. *Depending on govt interventions
- LordDragonfang 2y ago
  I just wish that AMD (and, pie in the sky, Intel) had gotten their shit together enough that these flaming dumptrucks full of money would have actually resulted in a competitive GPU market. Honestly, Zuckerburg (seemingly the only CEO willing to actually invest in an open AI ecosystem for the obvious benefits it brings them) should just invest a few million into hiring a few real firmware hackers to port all the ML CUDA code into an agnostic layer that AMD can build to.
  - hnthrowaway9812 2y ago
    Groq seems to be well positioned to give Nvidia a run for their money, actually.
blackeyeblitzar 2y ago
> What's the rationale for releasing all these models to the public? Do all these tech companies know something we don't? Why are they doing this? It’s mostly marketing for the company to appear to be modern. If you aren’t differentiated and if LLMs aren’t core to your business model, then there’s no loss from releasing weights. In other cases it is commoditizing something that would otherwise be valuable for competitors. But most of those 600K models aren’t high performers and don’t have large training budgets, and aren’t part of the “race”.
richardw 2y ago
It diminishes the story that Databricks is the default route to privately trained models on your own data. Databricks jumped on the LLM bandwagon really quickly to good effect. Now every enterprise must at least consider Snowflake, and especially their existing clients who need to defend decisions to board members. It also means they build large scale rails necessary to use Snowflake for training and can market such at every release.
- BPA0 2y ago
  I don't think that you understand Databricks. Databricks gives you the tools to train, tune or build RAG models. Snowflake doesn't. Having said that, I'm a big fan of Llama-3 at the moment.
  - buttaphingas 2y ago
    Snowflake has some examples of that: https://www.snowflake.com/blog/easy-secure-llm-inference-retrieval-augmented-generation-rag-cortex/ https://www.snowflake.com/blog/easy-secure-llm-inference-ret... https://quickstarts.snowflake.com/guide/asking_questions_to_your_own_documents_with_snowflake_cortex/index.html#0 https://quickstarts.snowflake.com/guide/asking_questions_to_... And I assume these new models would be now available for embedding. Is this different to what Databricks offers?

chessgecko 2y ago

This is the sparsest model thats been put out in a while (maybe ever, kinda forget the shapes of googles old sparse models). This probably wont be a great tradeoff for chat servers, but could be good for local stuff if you have 512GB of ram with your cpu.

imachine1980_ 2y ago
it performs worst than 8b llama 3 so you probably don't need that much.
- deleted 2y ago
  [deleted]
- coder543 2y ago
  Where do you see that? This comparison[0] shows it outperforming Llama-3-8B on 5 out of 6 benchmarks. I'm not going to claim that this model looks incredible, but it's not that easily dismissed for a model that has the compute complexity of a 17B model. [0]: https://www.snowflake.com/wp-content/uploads/2024/04/table-3-4-2048x808.png https://www.snowflake.com/wp-content/uploads/2024/04/table-3...
coder543 2y ago
It has 480B parameters total, apparently. You would only need 512GB of RAM if you were running at 8-bit. It could probably fit into 256GB at 4-bit, and 4-bit quantization is broadly accepted as a good trade-off these days. Still... that's a lot of memory. EDIT: This[0] confirms 240GB at 4-bit. [0]: https://github.com/ggerganov/llama.cpp/issues/6877#issue-2261479840 https://github.com/ggerganov/llama.cpp/issues/6877#issue-226...
- refulgentis 2y ago
  Yeah, and usually GPU RAM, unless you enjoy waiting for a minute for filling the context :(
- kaibee 2y ago

blackeyeblitzar 2y ago

Let’s stop using terms like open source falsely. The model isn’t open source, it is open weights. It’s good that the license for the weights is Apache, but for this model to be “truly open” they must release training data and source code under an OSI approved license. Otherwise it’s just misleading marketing. So far it seems like Snowflake will release some blog posts and “cookbooks”, whatever that means, but not actual training source code. Only the inference code is open source here, which is uninteresting.

deleted 2y ago
[deleted]
deleted 2y ago
[deleted]
jerrygenser 2y ago
> they must release training data and source code under an OSI approved license The source code is also apache 2.0
- blackeyeblitzar 2y ago
  Snowflake has only released the inference code - meaning the code you need to "run" the model. So if you take the weights they have released (which is the model that is a result of training), you can host the weights and inference code, and feed prompts to it, to get answers. But you don't have the actual source code you need to produce the weights in the first place. As an example of what open source actually means for LLMs, you can look at what AI2 does with their OLMo model (https://allenai.org/olmo https://allenai.org/olmo), where each model that they release comes with: > Full training data used for these models, including code that produces the training data, from AI2’s Dolma, and WIMBD for analyzing pretraining data. > Full model weights, training code, training logs, training metrics in the form of Weights & Biases logs, and inference code. > 500+ checkpoints per model, from every 1000 steps during the training process, available as revisions on HuggingFace. > Evaluation code under the umbrella of AI2’s Catwalk and Paloma. > Fine-tuning code and adapted models (with Open Instruct) > All code, weights, and intermediate checkpoints are released under the Apache 2.0 License. OLMo is what "truly open" is, while the rest is openwashing and marketing.

zamalek 2y ago

I suppose more and smaller experts would also help reduce over-fitting?

ru552 2y ago

Abnormally large. I don't see the cost/performance numbers going well for this one.

tosh 2y ago
It is both cost efficient in training (+ future fine-tuning) as well as inference compared to most other current models. Can you elaborate?
- ru552 2y ago
  the unquantized model is almost 1tb in size and the benchmarks provided by Snowflake shows performance in the middle of the pack compared to other recent releases.
- rajhans 2y ago
  We have published some insights here. https://medium.com/snowflake/snowflake-arctic-cookbook-series-exploring-mixture-of-experts-moe-c7d6b8f14d16 https://medium.com/snowflake/snowflake-arctic-cookbook-serie...

ur-whale 2y ago

However big it may be, it still hallucinates very, very badly. I just asked it an economics question and asked it to cite its sources. All the links provided as sources were complete BS. Color me unimpressed.

mike_hearn 2y ago
It's intended for SQL generation and similar with cheap fine tuning and inference, not answering general knowledge questions. Their blog post is pretty clear about that. If you just want a chatbot this isn't the model for you. If you want to let non-SQL trained people ask questions of your data, it might be really useful.
- mritchie712 2y ago
  It's worse at SQL generation than llama3 according to their own post. https://www.snowflake.com/blog/arctic-open-efficient-foundation-language-models-snowflake/#:~:text=on%20enterprise%20intelligence.-,Table%203,-.%20Full%20Metrics%20Table https://www.snowflake.com/blog/arctic-open-efficient-foundat...
  - CharlesW 2y ago
    To be fair, that's comparing their 17B model with the 70B Llama 3 model.
    - ru552 2y ago
      To stay fair, their "17B" model sits at 964GB on your disk and the 70B Llama 3 model sits at 141GB. unquantized GB numbers for both
      - CharlesW 2y ago
        Sorry, it sounds like you know a lot more than I do about this, and I'd appreciate it if you'd connect the dots. Is your comment a dig at either Snowflake or Llama? Where are you finding the unquantized size of Llama 3 70B? Isn't it extremely rare to do inference with large unquantized models?

Aissen 2y ago

How much memory would inference take on this type of model? What's the impact of it being an MoE architecture?

bfirsh 2y ago

If you want to have a conversation with it, here's a full chat app: https://arctic.streamlit.app/ https://arctic.streamlit.app/ Official blog post: https://www.snowflake.com/blog/arctic-open-efficient-foundation-language-models-snowflake/ https://www.snowflake.com/blog/arctic-open-efficient-foundat... Weights: https://huggingface.co/Snowflake/snowflake-arctic-instruct https://huggingface.co/Snowflake/snowflake-arctic-instruct

leblancfg 2y ago
Wow that is *so fast*, and from a little testing writes both rather decent prose and Python.
pixelesque 2y ago
I guess the chat app is under quite a bit of load? I keep getting error traceback "responses" like this: TypeError: This app has encountered an error. The original error message is redacted to prevent data leaks. Full error details have been recorded in the logs (if you're on Streamlit Cloud, click on 'Manage app' in the lower right of your app). Traceback: File "/home/adminuser/venv/lib/python3.11/site-packages/streamlit/runtime/scriptrunner/script_runner.py", line 584, in _run_script exec(code, module.__dict__) File "/mount/src/snowflake-arctic-st-demo/streamlit_app.py", line 101, in <module> full_response = st.write_stream(response)
hnthrowaway9812 2y ago
It claims to have a knowledge cut-off of 2021. Not sure if its hallucinating or its true. But when I asked it about the best LLMs it suggested GPT-3, Bert and T5!

PaulHoule 2y ago

It got the right answer for "Who is Tim Bray?" but it got "Who is Worsel the Dragon?" wrong.

nerpderp82 2y ago
Looks like they aren't targeting DRGN24 as one of their benchmark suites.
- PaulHoule 2y ago
  I love getting into arguments with LLMs over whether Worsel is an eastern dragon (in my imagination) or a western dragon (like the bad lensman anime.)
  - nerpderp82 2y ago
    Is Worsel in The Pile? Total aside, but I appreciate your arxiv submissions here. Just because they don't hit the front page, doesn't mean they are seen.
    - PaulHoule 2y ago
      Most LLMs seem to know about Worsel, I've had some who gave my right answer to "Who is Worsel?" but others will say they don't know who I am talking about and will be needed to be cued further. There is a lot of content about sci-fi on the web and all the Doc Smith books are on Canadian Gutenberg now. I found the Jetbrains assistant wasn't so good at coding (I might feel better if it did all the cutting and pasting, addding imports and that kind of stuff which would at least make it less tiresome to watch it bumble) but it is good at science fiction chat, better than all but two people I have known. Glad you like what I post.

mritchie712 2y ago

llama3 narrowly beats arctic at SQL generation (80.2 vs 79.0) and Mixtral 8x22B scored 79.2. You'd think SQL would be the one thing they'd be sure to smoke other models on. 0 - https://www.snowflake.com/blog/arctic-open-efficient-foundation-language-models-snowflake/#:~:text=Table%203.%20Full%20Metrics%20Table.%20Comparing%20Snowflake%20Arctic%20with%20DBRX%2C%20LLAMA%2D3%208B%2C%20LLAMA%2D3%2070B%2C%20Mixtral%208x7B%2C%20Mixtral%208x22B%20(instruction%2Dtuned%20or%20chat%20variants%20if%20available).1 https://www.snowflake.com/blog/arctic-open-efficient-foundat...

sp332 2y ago
Yeah but that's a 70B model. You can see on the Inference Efficiency chart that it takes more than 3x as much compute to run it compared to this one.
- msp26 2y ago
  Most people are vram constrained not compute constrained.
  - kaibee 2y ago
    Cloud providers aren’t though.
  - Manabu-eo 2y ago
    But those people usually have more system RAM than VRAM. At those scales, most people become bandwidth and compute constrained using CPU inference instead of multiple GPUs. In those cases, an MOE with a low number of active parameters is the fastest.
- karmasimida 2y ago
  But you do need to hold all 128 experts in memory? Or not? Or they simply consider inference efficiency as latency
  - giantrobot 2y ago

1f60c 2y ago

It appears to have limited guardrails. I got it to generate some risqué story and it also told me how to trade onion futures, which is illegal in the US.

klysm 2y ago
Why on earth is trading onion futures illegal in the us
- rbetts 2y ago
  A long history of rapscallions.
  - all2 2y ago
    Well, some kind of scallions anyway.
- isoprophlex 2y ago
  I looked it up, the story is pretty hilarious. https://en.m.wikipedia.org/wiki/Onion_Futures_Act https://en.m.wikipedia.org/wiki/Onion_Futures_Act
  - klysm 2y ago
    Wow I’m surprised the reaction was to ban futures on just onions specifically due to some market manipulation occurring. Surely this kind of manipulation wasn’t restricted to just onions? It seems incredibly short sighted
    - cryptonector 2y ago
      It's bloody stupid is what it is. Knee-jerking like that is really terrible.
      - klysm 2y ago

croes 2y ago

Reminds of the CPU GHz race. The main thing was that the figures were as large and impressive as possible. The benefit was marginal

salomonk_mur 2y ago
Yeah, I wouldn't say the benefits were marginal at all. CPUs went from dozens of MHz in the 90's to over 4 GHz nowadays.
- jasongill 2y ago
  I think what the parent commenter means is that the late 90's race to 1GHz and the early 2000's race for as many GHz as possible turned out to be wasted effort. At the time, ever week it seemed like AMD or Intel would announce a new CPU that was a few MHz faster than the competition, and the assumption among the Slashdot crowd was basically that we'd have 20GHz CPU's by now. Instead, there was a plateau in terms of CPU clock speed and even a regression once we hit about 3-4GHz for desktop CPUs where clock speeds started decreasing but other metrics like core count, efficiency, and other non-clock-based metrics of performance continued to improve. Basically, once we got to about ~2005 and CPU's touched 4GHz, the speeds slowly crept back into the 2.xGHz range for home computers, and we never really saw much (that I've seen) go back far above 4GHz at least for x86/amd64 CPUs. But yet the computers of today are much, much faster than the computers of 2005 (although it doesn't really "feel" like it, of course)
  - heisgone 2y ago
    It wasn't irrational at the time as it was much harder to harnest parallelism then.
  - genewitch 2y ago
    it's been well known (i'd heard it numerous times) that the maximum clock speed of x86 is somewhere <6GHZ. as in "you can make a 10ghz x86, but it would spend half the time idle". Bursting to 5.6ghz (or 5.8 iirc) is possible, but there are constraints that are physical in nature that prevent anything faster. Once single core single threaded CPUs hit ~4GHz the new "frontier" was the core 2 duo, then the core 2 quad, and now we have desktop chips with 16c/32t (and beyond).

krembo 2y ago

IMHO Google Search is doomed. This also impacting ad business, their main cash cow puts them in a very problematic position. Also companies who built their business solely on trained data such as OpenAI need to reinvent themselves.

ukuina 2y ago

So many models fail at basic reasoning. > What weighs more: a pound of feathers or a great british pound? > A pound of feathers and a Great British Pound weigh the same, which is one pound. It works if you add "Think step by step," though.

ec109685 2y ago
Another is if you ask which will land first, a bullet fired horizontally or one shot straight down. Derivations of well known problems trip these models up big time. Which bullet lands first, a bullet shot horizontally or one shot towards the ground The bullet shot horizontally will land first, assuming both bullets are fired from the same height and with the same initial velocity. This is because the bullet shot horizontally has a horizontal velocity component that causes it to travel forward while it falls under gravity. The bullet shot towards the ground only has the force of gravity acting on it, causing it to accelerate downwards. Therefore, the bullet shot horizontally will hit the ground later than the one shot towards the ground It’s not even consistent within the same answer! Even if you ask it to think step by step, it gets confused: Bullet shot towards the ground: When a bullet is shot towards the ground, it has both an initial horizontal and vertical velocity components. However, since we're only interested in comparing the time it takes for each bullet to reach the ground, we can focus on their vertical velocities and ignore their horizontal velocities. Let's denote: ( v_y ) as the initial vertical velocity of the second bullet (which is zero in this case) ( t' ) as the time it takes for the second bullet to reach the ground Using the same equation of motion in the vertical direction: [ y = h - \frac{1}{2}g(t')^2 ] Since ( v_y = 0 ), we have: [ t' = \sqrt{\frac{2h}{g}} ]
coder543 2y ago
Llama-3 8B got it on the first try: https://imgur.com/a/xQy1828 https://imgur.com/a/xQy1828
- readams 2y ago
  Gemini and Gemini Advanced both get this right (Edit: I initially thought Gemini got it wrong but I read the answer again and it's actually right!)

deleted 2y ago

[deleted]

imjonse 2y ago

Anything more open than OpenAI models can now call their models 'truly open'. It's good they will have recipes but they also don't seem to want to share the actual data.

Havoc 2y ago
Sharing data has too much legal liability given that half of it fell off a truck

vessenes 2y ago

Interesting architecture. For these "large" models, I'm interested in synthesis, fluidity, conceptual flexibility. A sample prompt: "Tell me a love story about two otters, rendered in the FORTH language". Or: "Here's a whitepaper, write me a simulator in python that lets me see the state of these variables, step by step". Or: "Here's a tarball of a program. Write a module that does X, in a unified diff." These are super hard tasks for any LLM I have access to, BTW. Good for testing current edges of capacity. Arctic does not do great on these, unfortunately. It's not willing to make 'the leap' to be creative in FORTH where creativity = storytelling, and tries to redirect me to either getting a story about otters, or telling me things about FORTH. Google made a big deal about emergent sophistication in models as they grew in parameter size with the original PaLM paper, and I wonder if these horizontally-scaled MOE of many small models are somehow architecturally limited. The model weights here, 480B, are sized close to the original PaLM model (540B if I recall). Anyway, more and varied architectures are always welcome! I'd be interested to hear from the Snowflake folks if they think the architecture has additional capacity with more training, or if they think it could improve on recall tasks, but not 'sophistication' type tasks.

themanmaran 2y ago
to be fair, gpt did a pretty good job at the otter prompt ``` \ A love story about two otters, Otty and Lutra : init ( -- ) CR ." Two lonely otters lived by a great river." ; : meet ( -- ) CR ." One sunny day, Otty and Lutra met during a playful swim." ; : play ( -- ) CR ." They splashed, dived, and chased each other joyfully." ; ...continued ```
- vessenes 2y ago
  BTW, I wouldn't rate that very high in that it's trying to put out syntactic FORTH, but not defining verbs or other things which themselves tell the story. Gemini is significantly better last I checked.
motoxpro 2y ago
What your evaluating is not what you think it is. You're evaluating the models ability to execute multiple complex steps (think about all of the steps it takes for your second example) not so much if it is capable of doing those things. If you broke it down into 2-3 different prompts it could do all of those things easy.

ofermend 2y ago

This model is great. Jumped immediately to 2nd place on HHEM leaderboard: https://github.com/vectara/hallucination-leaderboard https://github.com/vectara/hallucination-leaderboard