8 ms·
ETH Zurich and EPFL to release a LLM developed on public infrastructure
- seydor 1y agoI wonder if multilingual llms are better or worse compared a single language model
- tugdual 1y agoThis is an interesting problem that has various challenges - currently most tokenization solutions where trainees using hype pair encoding where the most commonly seen combinations of letters were being selected to be a mapping. This meant that the majority of tokenization was English mappings meaning your LLM had a better tokenization of English compared to other languages it was being trained on. C.f. https://medium.com/@biswanai92/understanding-token-fertility-why-it-matters-for-multilingual-llms-38c0b9f20da2 https://medium.com/@biswanai92/understanding-token-fertility...
- mukeshyadavnitt 1y agonice
- contrarian1234 1y agoThis seems like the equivalent of a university designing an ICE car... What does anyone get out of this when we have open weight models already ? Are they going to do very innovative AI research that companies wouldn't dare try/fund? Seems unlikely .. Is it a moonshot huge project that no single company could fund..? Not that either If it's just a little fun to train the next generation of LLM researchers.. Then you might as well just make a small scale toy instead of using up a super computer center
- urvader 1y agoThis model will be one of the few open models where the training data is also open which makes it ideal for fine tuning.
- chvid 1y agoThat it will actually be open and reproducible? Including how it was trained, what data was used, how training data was synthesized, how other models were used etc. All the stuff that is kept secret in case of llama, deepseek etc.
- herbst 1y agoWhy do you think it's about money? IMO it's about much more than that, like independence and actual data freedom trough reproductive LLMs
- MITSardine 1y agoSuper computers are being used daily for much toy-ier codes in research, be glad this at least interests the public and constitutes a foray of academia into new areas.
- defraudbah 1y agoETH Zurich is doing so many amazing things that I want to go study there. Unbelievable how many great people are coming from that university
- blue_light_man 1y agoIt's also possible you just think of ETH Zurich as great and automatically associate the people and products as amazing. Could be a circular dependency here.
- rtaylorgarlock 1y agoThat is indeed how things work. I can think of a few 'good' media-relevant examples, including e.g. that recent super-quick cart project [1], that reach beyond the more vanilla startup-spinoffs or basic media efforts. 1 https://ethz.ch/en/news-and-events/eth-news/news/2023/09/from-zero-to-one-hundred-in-0-956-seconds.html https://ethz.ch/en/news-and-events/eth-news/news/2023/09/fro...
- datameta 1y agoI took courses online from ETH Zurich before the formula was "perfected" and I'd say they were ahead of the curve in quality, concise but info-dense educational content.
- defraudbah 1y agoI had no idea what ETH means 2 years ago, I thought it's ethereum club in switzerland or something. Then I kept hearing about it, noticing people wearing ETH stuff. obviously I don't know if it's university or people there because I haven't been there, but I keep hearing about ETH Zurich in different areas and it means something
- Tepix 1y agoHow does it compare to Teuken and EuroLLM?
- sschueller 1y agoYet, Switzerland was put in the 2. Tier list[1] of countries that can get unlimited access to the top AI chips. [1] https://www.bluewin.ch/en/news/usa-restricts-swiss-access-to-ai-computer-chips-2535516.html https://www.bluewin.ch/en/news/usa-restricts-swiss-access-to... [2] https://chplusplus.org/u-s-export-controls-on-ai-chips/ https://chplusplus.org/u-s-export-controls-on-ai-chips/
- kisamoto 1y agoAny info on context length or comparable performance? Press release is unfortunately lacking on technical details. Also I'm curious if there was any reason to make such a PR without actually releasing the model (due Summer)? What's the delay? Or rather what was the motivation for a PR?
- rkrisztian 1y agoI'm disappointed. 8B is too low for GPUs with 16 GB VRAM (which is still common in affordable PCs), where most 13B to 16B models could still be easily run, depending on the quantization.
- k__ 1y ago"respecting web crawling opt-outs during data acquisition produces virtually no performance degradation" Great to read that!
- JKCalhoun 1y agoIs there not yet a Source where the web has already been scraped and souped down to just the text? It would seem someone would have created such a thing in order to save LLM training from having to reinvent the wheel. I understand the web is a dynamic thing but still it would seem to be useful on some level.
- CaptainFever 1y agoCommon Crawl, maybe?
- stephen_cagle 1y agoI wonder if the reason for these results is that any data on the internet is already copied to other locations by actors who ignore crawling opt-outs. So, even if they respect all web crawling opt-outs, they are still effectively copying the data because someone else did not respect it who does not include an opt-out.
- conradkay 1y agoMy guess is that it doesn't remove that much of the data, and the post-training data (not just randomly scraped from the web) probably matters more
- lllllm 1y agoYes this is an interesting question. In our arxiv paper [1] we did study this for news articles, and also removed duplicates of articles (decontamination). We did not observe an impact on the downstream accuracy of the LLM, in the case of news data. [1] https://arxiv.org/abs/2504.06219 https://arxiv.org/abs/2504.06219
- Onavo 1y ago
- Bengalilol 1y agoLooking forward to proof test it.
- greenavocado 1y agoWhy would you announce this without a release? Be honest.
- JumpCrisscross 1y agoFunding? Deeply biasing European uses to publicly-developed European LLMs (or at least not American or Chinese ones) would make a lot of sense. (Potentially too much sense for Brussels.)
- wood_spirit 1y agoThe announcement was at the International Open-Source LLM Builders Summit held this week in Switzerland. Is it so strange that they announced what they are doing and the timeline?
- phtrivier 1y agoThe cliché (at least on my side of the Alps) is that people in Switzerland like to take theiiiir tiiiime.
- Bengalilol 1y ago"Move as quickly as possible, but as slowly as necessary."
- WeirderScience 1y agoThe open training data is a huge differentiator. Is this the first truly open dataset of this scale? Prior efforts like The Pile were valuable, but had limitations. Curious to see how reproducible the training is.
- layer8 1y ago> The model will be fully open: source code and weights will be publicly available, and the training data will be transparent and reproducible This leads me to believe that the training data won’t be made publicly available in full, but merely be “reproducible”. This might mean that they’ll provide references like a list of URLs of the pages they trained on, but not their contents.
- WeirderScience 1y agoYeah, I suspect you're right. Still, even a list of URLs for a frontier model (assuming it does turn out to be of that level) would be welcome over the current situation.
- glhaynes 1y agoThat wouldn't seem reproducible if the content at those URLs changes. (Er, unless it was all web.archive.org URLs or something.)
- dietr1ch 1y agoThis is a problem with the Web. It should be easier to download content like it was updating a git Repo.
- TobTobXX 1y agoWell, when the actual content is 100s of terabytes big, providing URLs may be more practical for them and for others.
- sschueller 1y ago
- oytis 1y agoThe press release talks a lot about how it was done, but very little about how capabilities compare to other open models.
- pantalaimon 1y agoIt's a university, teaching the 'how it's done' is kind of the point
- EA-3167 1y agoSure, but usually you teach something that is inherently useful, or can be applied to some sort of useful endeavor. In this case I think it's fair to ask what the collision of two bubbles really achieves, or if it's just a useful teaching model, what it can be applied to.
- joot82 1y agoThe model will be released in two sizes — 8 billion and 70 billion parameters [...]. The 70B version will rank among the most powerful fully open models worldwide. [...] In late summer, the LLM will be released under the Apache 2.0 License. We'll find out in September if it's true?
- wood_spirit 1y agoThe article says “ Open LLMs are increasingly viewed as credible alternatives to commercial systems, most of which are developed behind closed doors in the United States or China” It is obvious that the companies producing big LLMs today have the incentive to try to enshitify them. Trying to get subscriptions at the same time as trying to do product placement ads etc. Worse, some already have political biases they promote. It would be wonderful if a partnership between academia and government in Europe can do a public good search and AI that endeavours to serve the user over the company.
- klabb3 1y agoYes but it’s a very complicated service to deliver. Even if they train great models, they likely will not operationalize them for inference. Those will still be private actors, and the incentives to enshittify will be the same. Also, for AI generally the incentives is much higher than last tech generation, due to cost of running these things. Basically, the free services where you’re the product must aggressively extract value out of you in order to make a profit.
- bee_rider 1y agoIs this setting the bar for dataset transparency? It seems like a significant step forward. Assuming it works out, that is. They missed an opportunity though. They should have called their machine the AIps (AI Petaflops Supercomputer).
- philipkglass 1y agoI think that the Allen Institute for Artificial Intelligence OLMo models are also completely open: OLMo is fully open Ai2 believes in the power of openness to build a future where AI is accessible to all. Open weights alone aren’t enough – true openness requires models to be trained in the open with fully open access to data, models, and code. https://allenai.org/olmo https://allenai.org/olmo
- lamuswawir 1y agoI am a simple man, I see AI2, I upvote.
- ekianjo 1y agoSmollm is also completely open as far as I know
- isusmelj 1y agoI hope they do well. AFAIK they’re training or finetuning an older LLaMA model, so performance might lag behind SOTA. But what really matters is that ETH and EPFL get hands-on experience training at scale. From what I’ve heard, the new AI cluster still has teething problems. A lot of people underestimate how tough it is to train models at this scale, especially on your own infra. Disclaimer: I’m Swiss and studied at ETH. We’ve got the brainpower, but not much large-scale training experience yet. And IMHO, a lot of the “magic” in LLMs is infrastructure-driven.
- lllllm 1y agoNo, the model has nothing do to with Llama. We are using our own architecture, and training from scratch. Llama also does not have open training data, and is non-compliant, in contrast to this model. Source: I'm part of the training team
- d3m0t3p 1y agoHey, really cool project, I’m excited to see the outcome. Is there a blog / paper summarizing how you are doing it ? Also which research group is currently working on it at eth ?
- blurbleblurble 1y agoAre you using dbpedia?
- lllllm 1y agono. the main source is fineweb2, but with additional filtering for compliance, toxicity removal, and quality filters such as fineweb2-hq
- PeterStuer 1y agoThx for engaging here. Can you comment on how the filtering impacted language coverage? E.g. finweb2 has 1800+ languages, but some with very little actual representation, while finweb2-hq has just 20 but each with a subdsantial data set. (I'm personaly most interested in covering the 24 official EU languages)
- hubraumhugo 1y agoPretty proud to see this at the top of HN as a Swiss (and I know many are lurking here!). These two universities produce world-class founders, researchers, and engineers. Yet, we always stay in the shadow of the US. With our top-tier public infrastructure, education, and political stability (+ neutrality), we have a unqiue opportunity to build something exceptional in the open LLM space.
- RHSman2 1y agoI work with EPFL alumni. Brilliant minds.
- MITSardine 1y agoI think EPFL and ETH are generally well known internationally, but Switzerland being rather small (9M pop), it's only natural you don't hear much about it compared to other larger countries!
- amelius 1y agoYeah, that's what "democratizing AI" means.
- nektro 1y agogross use of public infrastructure
- PetitPrince 1y agoSometimes ago there was a Tom Scott video about the fasted accelerating car in the world, developed by a team with a vast majority of student. One remark stayed with me: "the goal is not to build a car, but to build engineer". In that regard it's absolutely not a waste of public infra just like this car was not a waste.
- herbst 1y agoIt even used green power. Literally zero complains or outcry from the public yet. Guess we like progress, especially if it helps independence.
- MITSardine 1y agoUniversity and research clusters are built to run research code. I can guarantee this project is 10x as impactful and interesting as what usually runs on these machines. This coming from someone in the area that usually hogs these machines (numerical simulation). I'm very excited to see academic actors tackle LLMs.
- protocolture 1y agoI literally cant fault this, even steelmanning anti AI positions. What makes you say that?
- westurner 1y agoUse case for science and code LLMs: Superhydrodynamic gravity (SQR / SQG, ) LLMs do seem to favor general relativity but probably would've favored classical mechanics at the time given the training corpora. Not-yet unified: Quantum gravity, QFT, "A unified model must: " https://news.ycombinator.com/item?id=44289148 https://news.ycombinator.com/item?id=44289148 Will be interested to see how this model responds to currently unresolvable issues in physics. Is it an open or a closed world mentality and/or a conditioned disclaimer which encourages progress? What are the current benchmarks? From https://news.ycombinator.com/item?id=42899805 https://news.ycombinator.com/item?id=42899805 re: "Large Language Models for Mathematicians" (2023) : > Benchmarks for math and physics LLMs: FrontierMath, TheoremQA, Multi SWE-bench: https://news.ycombinator.com/item?id=42097683 https://news.ycombinator.com/item?id=42097683 Multi-SWE-bench: A Multi-Lingual and Multi-Modal GitHub Issue Resolving Benchmark: https://multi-swe-bench.github.io/ https://multi-swe-bench.github.io/ Add'l LLM benchmarks and awesome lists: https://news.ycombinator.com/item?id=44485226 https://news.ycombinator.com/item?id=44485226 Microsoft has a new datacenter that you don't have to keep adding water to; which spares the aquifers. How to use this LLM to solve energy and sustainability problems all LLMs exacerbate? Solutions for the Global Goals, hopefully
- westurner 1y ago(Unbelievable that I need to justify this at -4!) Is the performance or accuracy on this better on FrontierMath or Multi-SWE-bench, given the training in 1,000 languages? I just read in the Colab release notes that models uploaded to HuggingFace can be opened on Colab with "Open in colab" on HuggingFace
- kordlessagain 1y agoIt's the word "gravity" that triggers them.
- adultSwim 1y agoThis is such a smart move for the country. Best wishes on their important endeavor.