12 ms·
Benchmarks suggest this model loses to Deepseek-R1 in every one-shot comparison. Considering they were likely not even pitting it against the newer R1 version (
by pu_pe 1y ago
Benchmarks suggest this model loses to Deepseek-R1 in every one-shot comparison. Considering they were likely not even pitting it against the newer R1 version (no mention of that in the article) and at more than double the cost, this looks like the best AI company in the EU is struggling to keep up with the state-of-the-art.
- atemerev 1y ago"EU is leading in regulation", they say. I don't know what they are thinking.
- micromacrofoot 1y agoprobably some silly thing like "people should have more rights and protections"
- atemerev 1y agoI've yet to find any rights and protections in these cookie banners.
- saubeidl 1y agoThe cookie banners are corps trying to circumvent the rights and protections. If they actually went by the spirit of the protections, the cookie banners wouldn't be needed. Your ire is misdirected.
- yeahforsureman 1y agoAre you sure? The ePrivacy Directive requires a (GDPR-level) consent for just placing the cookie, unless it's strictly necessary for the provision of the “service”. The way EU regulators interpret this, even web analytics falls outside the necessity exception and therefore requires consent. So as long as the user doesn't and/or is not able to automatically signal consent (or non-consent) eg via general browser-level settings, how can you obtain it without trying to get it from the user on a per-site basis somehow? (And no, DNT doesn't help since it's an opt-out, not an opt-in mechanism.)
- exyi 1y agoEveryone I know of will try to click "reject all unnecessary cookies", and you don't need the dialog for the necessary ones. You can therefore simply remove the dialog and the tracking, simplifying your code and improving your users' experience. Can tracking the fraction which misclicks even give some useful data?
- yeahforsureman 1y agoMy point was that according to the current interpretation, if they rely on cookies, user analytics (even simple visitor stats where no personal data is actually processed) are not considered "necessary" and are therefore not exempt from the cookie consent obligation under the ePrivacy Directive. The reason why personal data processing is irrelevant is that the cookie consent requirement itself is based on the pre-GDPR ePrivacy Directive which requires, as a rule, consent merely for saving cookies on the client device (subject to some exceptions, including the one discussed). So you need a consent for all but the most crucial cookies without which the site/service wouldn't be able to function, like session cookies for managing signed-in state etc. (The reason why you started to see consent banners really only after GDPR came to force is at least in part due to the fact that the ePrivacy Directive refers to the Data Protection Directive (DPD) for the standard of consent, and after DPD was replaced by GDPR, the arguably more stringent GDPR consent standard was applied, making it unfeasible to rely on some concept of implied consent or the like.)
- mhitza 1y agoUser analytics that require cookies, sounds like tracking to me. > like session cookies for managing signed-in state etc. Maybe I'm reading it wrong, but are you saying that consent is required for session cookies? Because that is not the case, at all. > (25) However, such devices, for instance so-called "cookies", can be a legitimate and useful tool, for example, in analysing the effectiveness of website design and advertising, and in verifying the identity of users engaged in on-line transactions. Where such devices, for instance cookies, are intended for a legitimate purpose, such as to facilitate the provision of information society services, their use should be allowed on condition that users are provided with clear and precise information in accordance with Directive 95/46/EC about the purposes of cookies or similar devices so as to ensure that users are made aware of information being placed on the terminal equipment they are using. Users should have the opportunity to refuse to have a cookie or similar device stored on their terminal equipment. This is particularly important where users other than the original user have access to the terminal equipment and thereby to any data containing privacy-sensitive information stored on such equipment. Information and the right to refuse may be offered once for the use of various devices to be installed on the user's terminal equipment during the same connection and also covering any further use that may be made of those devices during subsequent connections. The methods for giving information, offering a right to refuse or requesting consent should be made as user-friendly as possible. Access to specific website content may still be made conditional on the well-informed acceptance of a cookie or similar device, if it is used for a legitimate purpose. https://eur-lex.europa.eu/eli/dir/2002/58/oj/eng https://eur-lex.europa.eu/eli/dir/2002/58/oj/eng You should inform users about any private data you would be storing in a cookie. But this can be a small infobox on your page with no button. When storing other type of information, the "cookie" problem needs to be seen from the perspective of shared devices. You know, the times before, when you might forget to log out at an internet cafe or clear your cookies containing password and other things they shouldn't. This is a dated approach at looking at the problem (most people have their own computing devices today, their phone), but still applicable (classrooms, and family shared devices).
- micromacrofoot 1y agocookie banners are malicious compliance while we head towards the death of cross-site cookies, they are indeed a poor implementation but the legislation that lead to them did not come up with it did you really prefer when companies were selling your data to third parties and didn't have to ask you?
- sunaookami 1y agoDo you really think clicking "Reject non-essential cookies" does something?
- micromacrofoot 1y agoshow me a single example that doesn't
- __alexs 1y agoEU regulation is often "you can not have the cool thing" not "the cool thing must be operated equitably". I think they are more interested in protecting old money than in protecting people.
- saubeidl 1y agoCan you name specific examples? Otherwise, this just sounds like inflammatory polemic.
- micromacrofoot 1y agoI think usb-c and third party app stores are pretty cool
- umbra07 1y agoI think the government shouldn't be legislating that companies must use a specific USB connector. Realistically the legislation was only targeting Apple. If consumers want USB-C, then they can vote with their wallets and buy an Android, which is a reasonable alternative.
- micromacrofoot 1y agoWe've had multiple USB standards for decades with no end in sight. Apple was targeted because they have the most high-profile proprietary connector and they were generally using it to screw consumers. Good riddance.
- bobxmax 1y agoRights and protections that have benefited heavily from an economy built on the alliance with the US. If it weren't for American help and trade post-WW2, Europe would be a Belarusian backwater and is fast heading back in that direction. Countries like Greece, Italy, Spain, Portugal, etc. show the future of Europe as it slowly stagnates and becomes a museum that can't feed it's people. Even Germany that was once excelling is now collapsing economically. The only bright spot on the continent right now is Poland who are, shocker, much less regulatorily strict and have lower corporate taxes.
- debugnik 1y ago> Countries like Greece, Italy, Spain, Portugal PIGS, really? Some of the top growing EU economies right now, which have turned their deficit around, show the future of a slowly stagnating Europe?
- bobxmax 1y agoA 200B economy growing 2% is the future of the EU? Yes that is the point I am making.
- micromacrofoot 1y agoHow much is an economy supposed to grow?
- bobxmax 1y agoBangladesh has twice the GDP and 3x the growth. How long do you expect Portugal to be relevant?
- dmos62 1y agoIt is fairly common to struggle to understand why different cultures think the way they do.
- moralestapia 1y agoUgh. Edit: Parent changed their comment significantly, from something quite unpleasant to what it is now. I'm not deleting my comment as I'm not that kind of person.
- dmos62 1y agoI did. I initially said that Europeans often struggle to understand other cultures too. Which was an immature way to point out that the cultural dissonance works both ways. I realized that I was obfuscating my point and rewrote my comment to be clearer, but now that you gave me a chance to think on it some more, I wish I would have said what I wanted to say more directly still. What I wanted to say is: I like EU's regulation and I find it interesting how other people have different world views.
- atemerev 1y agoI live in Europe.
- mrtksn 1y agoCool, which regulations exactly stopped you from doing cutting edge AI?
- kelseyfrog 1y agoDécret sur la Pause Goûter Universelle (PGU).
- philjohn 1y agoIs that the regulation that says you need to allow someone to take a 20 minute break after 6 hours of work?
- cpldcpu 1y agoSorry, this is just getting old... Its a trite talking point and not the reason why there are so few consumer-AI companies in Europe.
- atemerev 1y agoAnd what would be the reason? I am genuinely interested. Also, are there viable not "consumer" AI companies here? Only Mistral seems to train foundation models, and good for them, however, as of now they are absolutely not SOTA.
- baq 1y agoMoney. No, really - EU doesn't have the VCs and the megacorps. People laugh at EU sponsoring projects, but there is no private money to sponsor them. There are plenty of US companies with sites in the EU though, so you have people working the problems, but no branding.
- SV_BubbleTime 1y agoOk, just a quick question… why does Europe not have the money actual/people?
- baq 1y agoedit: the parent has since edited out the flamebait. Maybe, or maybe when silicon valley was busy growing exponentially Europe was still picking itself up from the mess of ww2. Trying to blame a single reason is futile, naive and childish.
- oceanplexian 1y agoThe US was out-innovating Europe a long time before WW2, we had faster, more extensive rail systems, superior high rise construction, earlier to electrification, invention of the telephone, modern manufacturing (Model T), invention of the airplane, the birth of Hollywood and modern motion pictures, the list goes on.
- 0xDEAFBEAD 1y agoHonestly the US approach to AI is incredibly irresponsible. As an American, I'm glad that someone somewhere is thinking about regulation. Not sure it will be enough though: https://xcancel.com/ESYudkowsky/status/1922710969785917691#m https://xcancel.com/ESYudkowsky/status/1922710969785917691#m
- MoonGhost 1y agoNo, thanks, we don't want to be like EU. Everything regulated to death. They even thought to criminalize street photography because there could be copyrighted materials in the picture. Not sure, are they still taxing Eiffel tower images?
- johnisgood 1y agoI thought it is happening in the US, too. I mean, the Government is there to regulate the shit out of everything. Regardless of where you are.
- int_19h 1y agoEU is not a monolithic entity, and amount of regulation varies widely. Baltics are very business friendly, for example.
- bobxmax 1y agoAnd Estonia has the most impressive tech ecosystem on the continent while being a soviet backwater 20 years ago. Shocking how that works.
- msgodel 1y agoThere's nothing the regulation could meaningfully hope to accomplish other than slow down people willing to play by the rules.
- ambicapter 1y agoWow, the "criminals don't follow laws therefore laws are worthless" argument, here? In my HN?
- Mistletoe 1y agoThis is why I want to move to the EU. I don’t care if companies aren’t coddled there. I want to live where people are the first priority.
- atemerev 1y agoWell, are you ready to live on a low middle class salary of a European software engineer? It is really low middle class. The middle middle here would be a bank clerk, and upper middle — a lawyer or a surgeon. This is not coincidental.
- baq 1y agoIncidentally (also not) surgeons and lawyers are not poor in the states either… it’s just Silicon Valley was the perfect place with just the right people and it kept growing for 60 years straight. Surgery and law do not grow exponentially. (I’ll pretend the pages of regulation aren’t supposed to count.)
- mrtksn 1y agoEurope isn't going to catch up in tech as long as its market is open to US tech giants. Tech doesn't have marginal costs, so you want to have one of it in one place and sell it everywhere and when the infra and talent is already in US, EU tech is destined to do niche products. UK has a bit of it, France has some and that's it. The only viable alternatives are countries who have issues with US and that is China and Russia. China have come up with strong competitors and it is on cutting edge. Also, it doesn't have anything to do with regulations. 50 US States have the American regulations, its all happening in 1 and some other states happen to host some infrastructure but that's true for rest of of the world too. If the EU/US relationship gets to Trump/Musk level, then EU can have the cutting edge stuff. Most influential AI researchers are from Europe(inc. UK), Israel and Canada anyway. Ilya Sutskever just the other day gave speech at his alma matter @Canada for example. Andrej Karpathy is Slovakian. Lot's of Brits, French, Polish, Chinese, German etc. are among the pioneers. Significant portion of the talent is non-American already, they just need a reason to be somewhere else than US to have it outside the US. Chinese got their reason and with the state of the affairs in the world I wouldn't be surprised if Europeans gets theirs in less than 3 and a half years.
- vikramkr 1y agoIf you close off the market to US tech giants, maybe they'll have some amount of market dominance at home, but I would doubt that would mean they've "caught up" tech wise. There would be no incentive to compete. American EV manufacturing is pretty far behind Chinese EV manufacturing, protectionism didn't help make a competitive car, it just protected the home market while slowly ceding international market after international market
- saubeidl 1y agoAs a counterexample, China's tech industry has caught up and in some ways surpassed the US, partially due to being closed off.
- hshdhdhj4444 1y agoBut also due to the U.S. driving away smart people from the U.S. to China.
- funnym0nk3y 1y agoThought so too. I don't know how it could be different though. They are competing against behemoths like OpenAI or Google, but have only 200 people. Even Anthropic has over 1000 people. DeepSeek has less than 200 people so the comparison seems fair.
- rsanek 1y agoany claim from the deepseek folks should be considered with wide margins of error.
- humpty-d 1y agoI know we distrust them on account of being nefarious Chinese, but has anything come to light with R1 or the people behind it specifically to justify this?
- mwigdahl 1y ago"Deepseek only cost $6 million"?
- baq 1y ago“* we only tallied the electricity and rent”
- Gerardo1 1y agoThere's no way to know who's funding it (but being at least state subsidized is highly likely), and you we don't really know how much it cost (but in any case it's still less than OpenAI is spending). On the other hand I'm aware of no credible accusations of deepseek fudging benchmarks whereas OpenAI has had multiple instances of independent parties not being able to replicate their claimed performances on benchmarks (and not being honest and transparent about their benchmarking).
- jasonthorsness 1y agoEven if it isn't as capable, having a model with control over training is probably strategically important for every major region of the world. But it could only fall so far behind before it effectively doesn't work in the eyes of the users.
- deleted 1y ago[deleted]
- melicerte 1y agoIf you look at Mistral investors[0], you will quickly understand that Mistral is far from being European. My understanding is it is mainly owned by US companies with a few other companies from EU and other places in the world. [0] https://tracxn.com/d/companies/mistral-ai/__SLZq7rzxLYqqA97jtPwO09jLDeb76RVJVb306OhciWU/funding-and-investors https://tracxn.com/d/companies/mistral-ai/__SLZq7rzxLYqqA97j... (edited for typo)
- pdabbadabba 1y agoFor the purposes of GP's comment, I think the nationalities of the people actually running the company and doing the work are more relevant than who has invested.
- derektank 1y agoAnd, perhaps most relevantly, the regulatory environment the people are working in. French people working in America are probably more productive than French people working in France (if for no other reason because they probably work more hours in America than France).
- vasco 1y agoMost measures of productivity have "hours worked" in the denominator so that can't be right.
- underdeserver 1y agoIf I work 1000 hours and you work 2000 hours in the same timeframe, but you outcompeted me and created 3x value, you are 1.5 times more productive. There's a numerator too.
- deleted 1y ago[deleted]
- vasco 1y ago
- littlestymaar 1y ago> Benchmarks suggest this model loses to Deepseek-R1 in every one-shot comparison. That's not particularly surprising though as the Medium variant is likely close to ten times smaller than DeepSeek-R1 (granted it's a dense model and not an MoE, but still).
- fiatjaf 1y agoThis reads like an AI-generated comment. What do you mean by "benchmarks suggest"? The benchmarks are very clear and presented right there in the page.
- tootie 1y agoAs an occasional user of Mistral, I find their model to give generally excellent results and pretty quickly. I think a lot of teams are now overly focused on winning the benchmarks while producing worse real results.
- esafak 1y agoIf so we need to fix the benchmarks.
- paulddraper 1y agohttps://en.wikipedia.org/wiki/Goodhart%27s_law https://en.wikipedia.org/wiki/Goodhart%27s_law
- riku_iki 1y agothose who try to fix them are fighting alone against huge corps which try to abuse them..
- tootie 1y agoI think there's a fundamental limit to benchmarks when it comes to real-world utility. The best option would be more like a user survey.
- esafak 1y agoThat's Chatbot Arena: https://lmarena.ai/leaderboard https://lmarena.ai/leaderboard
- jug 1y agoAnd unfortunately revealed to be largely a vibe check these days with that whole Llama 4 debacle. But why should we be surprised, really, when users have an easier time feeling if the replies sound human and conversational and _appear_ knowledgeable than actually outsmarting them. This Arena worked well in the ChatGPT 3.0 days… But now?
- segmondy 1y agoare you really going to compare a 24B model to a 700B+ model?
- a2128 1y ago24B is the size of the Small opensourced model. The Medium model is bigger (they don't seem to disclose its size) and still gets beaten by Deepseek R1
- thot_experiment 1y agoMistral Large is 123b so one can probably assume that medium is between 24b and 123b, also Mistral 3.1 is by a wide margin my go-to model in real life situations. Benchmarks absolutely don't tell the whole story, and different models have different use cases.
- Ringz 1y agoCan you please explain what your „real life situations“ are?
- thot_experiment 1y agoI use it as a personal assistant (so tool use integrated into calendar/todo/notes etc) often times using the multimodal aspect (taking a photo of a todo list, asking it to remind me to buy something from a picture). I also use it as a code completion tool in vscode, as well as a replacement for most basic google searches ("how does this syntax work", "what's the torch method for X") I use it for almost every interaction I have with AI that isn't asking it to oneshot complex code. I fairly frequently run my prompts against Claude/ChatGPT and Mistral 3.1 and find that for most things they're not meaningfully different. I also spend a lot of time playing around with it for storytelling/integration into narrative games.
- mandelken 1y agoCool. What framework or program do you use to orchestrate this?
- tensor 1y ago[flagged]
- demosthanos 1y agoIt gets pretty tiresome when people automatically take something as pro-US jingoism without stopping to think for a second about what they're saying. The model they're comparing to is Chinese, not American. How can you possibly construe OP's comment as pro-US? EDIT: The comment below got flagged to death, so I'm going to link my evidence up here for visibility: https://news.ycombinator.com/item?id=44238841 https://news.ycombinator.com/item?id=44238841 Not only does the top-level comment make no mention of the US, but their comment history is strongly suggestive that they themselves are European, not US-based.
- tensor 1y ago[flagged]
- demosthanos 1y agoAre we looking at the same account? There's not a lot of data there, as is the nature of a new account, but from where I'm sitting I'd guess they're based in Portugal or if not there specifically then at least on mainland Europe. Out of 5 comments: * This one is saying that Europe's biggest AI company isn't keeping up with China. * Two comments are talking about Europe needing to become more independent from American tech companies. * A fourth is talking about ElevenLabs's Portuguese support and noting that one of the Portuguese voices has a Spanish accent. * The fifth is unrelated to geography or language. I'm not sure where in this you get that they have a US-centric perspective. The only comments they have that mention the US at all are taking the very distinctly 2025-European tack of advocating for increased independence from America. Coupled with this one contrasting with Chinese tech that presents a pretty unified picture of someone who has a lot of interest in Europe having its own strong tech independent from the rest of the world. The ElevenLabs comment is the most telling for me for pinpointing the geography: in order to know what Portuguese spoken with a Spanish accent sounds like you really have to have spent a significant amount of time in Iberia and most likely speak Portuguese as a native language.
- deleted 1y ago[deleted]
- hmottestad 1y agoWith how amazing the first R1 model was and how little compute they needed to create it, I'm really wondering how the new R1 model isn't beating o3 and 2.5 Pro on every single benchmark. Magistral Small is only 24B and scores 70.7% on AIME2024 while the 32B distill of R1 scores 72.6%. And with majority voting @64 the Magistral Small manages 83.3%, which is better than the full R1. Since I can run a 24B model on a regular gaming GPU it's a lot more accessible than the full blown R1. https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-...
- adventured 1y agoIt's because DeepSeek was a fast copy. That was the easy part and it's why they didn't have to use so much compute to get near the top. Going well beyond o3 or 2.5 Pro is drastically more expensive than fast copy. China's cultural approach to building substantial things produces this sort of outcome regularly, you see the same approach in automobiles, planes, Internet services, industrial machinery, military, et al. Innovation is very expensive and time consuming, fast copy is more often very inexpensive and rapid. 85% good enough is often good enough, that additional 10-15% is comically expensive and difficult as you climb.
- MaxPock 1y agoI understand that the French are very innovative so why isn't their model SOTA ?
- deleted 1y ago[deleted]
- natrys 1y agoNot disagreeing with the overarching point but: > That was the easy part Is a bit hand-wavy in that it doesn't explain why it's only DeepSeek who can do this "easy" thing, but still not Meta, Mistral or anyone else really. There are many other players who have way more compute than DeepSeek (even inside China, not even considering rest of the world), and I can assure you more or less everyone trains on synthetic data/distillation from whatever bigger model they can access.
- epolanski 1y agoJm2c but I feel conflicted about this arms race. You can be 6/12 months later, and have not burned tens of billions compared to the best in class, I see it an engineering win. I absolutely understand those that say "yeah, but customers will only use the best", I see it, but is market share of forever money losing businesses that valuable?
- adventured 1y agoA similar sentiment existed for a long time about Uber and now they're very profitable and own their market. It was worth the burn to capture the market. Who says OpenAI can't roll over to profitable at a stable scale? Conquer the market, hike the price to $29.95 (family account, no ads; $19.95 individual account with ads; etc etc). To say nothing of how they can branch out in terms of being the interaction point that replaces the search box. The advertising value of owning the land that OpenAI is taking is well over $100 billion in annual revenue. Amazon's retail business is terrible, their ad business is fantastic. As OpenAI bolts on an ad product their margin potential will skyrocket and the cost side will be modest in comparison. Over the coming years it won't be possible to stay a mere 6-12 months behind as the costs to build and maintain the AI super-infrastructure keeps climbing. It'll become a guaranteed implosion scenario. Winning will provide the ongoing immense resources needed to keep pushing up the hill forever. Everybody else - except a few - will fall away. The same outcome took place in search. Anybody spot Lycos, Excite, Hotbot, AltaVista around? It costs an enormous amount of money to try to keep up with Google (Bing, Baidu, Yandex) in search and scale it. This will be an even more brutal example of that, as the costs are even higher to scale. The only way Mistral survives is if they're heavily subsidized directly by European states.
- aDyslecticCrow 1y ago> It was worth the burn to capture the market. You cannot compare Uber to the AI market. They are too different. Uber captured the market because having three taxi services is annoying. But people are readily jumping between models using multi-model platforms. And nobody is significantly ahead of the pack. There is nothing that sets anyone apart aside from the rate at which they are burning capital. Any advantage is closed within a year. If OpenAI wants to make a profit, it will raise prices and be dropped at a heartbeat for the next cheapest option. Most software stacks are designed to be model-agnostic, making integration or support a non-factor.
- wafngar 1y agoBut they have built a fully “independent” pipeline. Deepseek and others probably trained in gpt4, o1 or whatever data.