13 ms·
Grok-2 Beta Release
- CMLab 2y agoWhen github release?
- ipsum2 2y agoRealistically? Never. Grok-1 weights were released because it was quite bad compared to open source and closed models. Now that they have a competitive model, they won't give it away.
- thomasahle 2y ago> Now that they have a competitive model, they won't give it away. Lamma 3.1 is competitive.
- wskinner 2y agoX is not Meta.
- pikseladam 2y agoIt uses FLUX.1 to generate images and it has been fun so far. Its good on writing, can generate very realistic photos, can create memes, and looks like hands problem is fixed now.
- DaoVeles 2y agoWhen I have time I will do my usual test "Realistic looking wizards bowling!" and see how it goes. So far I have had fairly disappointing results.
- esperent 2y agoWhat's a realistic wizard? Given that wizards don't actually exist this might be a confusing request. Have you tried putting in "photorealistic" instead of "realistic", assuming that's what you mean? I'm curious if that would get better results.
- Y_Y 2y agoIt's a wizard who doesn't have unreasonable expectations. Wizards definitely exist by the way, for many reasonable definitions of "wizard".
- espadrine 2y agoFirst try, so no cherry picking: https://fal.ai/models/fal-ai/flux-pro?share=43b11da2-e008-4d90-9a42-a39f75b64ccb https://fal.ai/models/fal-ai/flux-pro?share=43b11da2-e008-4d...
- maeil 2y agoI guess for a wizard it does make sense for a spell-gone-wrong to have chopped off two of his fingers.
- ssijak 2y agoOh this is great, one more competitor with top model which will be available via API. I wonder what the pricing will be. OpenAI was slashing prices multiple times in the last year and a half I was using it.
- mcintyre1994 2y agoI can't imagine anyone would want to build on top of their APIs after they completed destroyed the Twitter API and its whole ecosystem.
- perbu 2y agoI assume it'll be a paid API so the "contract" is a lot more clear. Twitter never understood what to do with its API so pulling that particular rug makes sense. But I too wouldn't use this. X is playing fast and loose with ... everything, so having a business rely on their product seems risky.
- bboygravity 2y agoIf they have the best model, everybody will use it. With LLMs (and AGI) it's really that simple: the company with the best model wins regardless of all else.
- threeseed 2y agoIt isn't that simple at all. It's going to be a combination of price, performance, quality, reliability, availability etc. And since the prompts need to be optimised for each model there is a degree of vendor lock-in.
- bboygravity 2y agoI wasn't really talking about the marginal differences we see right now in August 2024. I'm talking about the next huge step forward that only 1 company will achieve, because it simply has the most GPUs (in limited supply) + energy source first and keeps that advantage. At some point this becomes a run-away self amplifying differentiator and it will make that company win regardless of all else. My money is on xAI in 2025. PS: the only reason prompts need to be optimized for each model is a symptom of models simply not being that good yet. This need will vanish in the near future as you get way better models. A recent hint of what I mean: mid-journey needed very elaborate prompt (and even loras) to get what you want. In flux that prompt can be much shorter (without loras) and it still gets closer to what you want. Same will happen with LLMs. Another example: with ChatGPT 4 you need to literally beg a model to only return what you ask for (for example JSON) or put it in a certain mode (JSON mode), in Claude Sonnet 3.5 it will simply just listen to what you ask for. So again: that's not "because every model needs model-specific fine-tuning" that's because previous models where simply not as good.
- ionwake 2y agoGuys come on, you cant keep releasing software in the US, and then do a staggered launch where months later things are available to users in England, Denmark etc. There should be no reason for it. Im sure whatever dumb EU regulations can be dealt with easily in the software, these staggered releases ( such as chagpt having no Memory etc MONTHS down the line for EU users ) its just a hindrance to progress. Its starting to feel like we live on an island in the middle of nowhere.
- k4rli 2y agoFortunately we already have Anthropic etc so this new release isn't really relevant or useful
- hersko 2y agoWhy would another competitive option not be a good thing?
- croes 2y agoI'm pretty glad the EU a least tries to protect it's citizens from company experiments, who only care about profit and not others well-being.
- seydor 2y agoWhat is it protecting us from, realistically? It's just a powerplay by the world's most incompeteny private club (brussels)
- ionwake 2y agowell its protecting us from data mining. Which is good. But I disagree with the clunky implementation.
- sunaookami 2y agoThe EU itself wants to data mine everything, just look at chat control.
- ZacnyLos 2y agoTwitter started irreversibly feeding users’ data into its “Grok” AI technology in May 2024, without ever informing them or asking for their consent. https://noyb.eu/en/twitters-ai-plans-hit-9-more-gdpr-complaints https://noyb.eu/en/twitters-ai-plans-hit-9-more-gdpr-complai...
- Lerc 2y agoWhat does irreversibly mean in this context? It seems like negative connotations are implied, but I feel like it's like irreversibly baking a cake.
- makin 2y agoOnce the data is "compressed" into the model it cannot be easily removed without starting the training over.
- Lerc 2y agoSo you mean like "He used one of my eggs to irreversibly make a cake" It's true, but it would be kind of amazing if it weren't
- netrus 2y agoThe irreversibility is still important to highlight, as it is distinctively different from a similar consent issue with search: "Google indexed my website against my will, but I will just forbid them to include me in search results going forward".
- zaptrem 2y agoIt is irreversible similar to how a student reading a textbook from LibGen can remember and profit from that information forever. Kinda crazy how many in this community went from champions of freedom of knowledge to champions of megacorps owning and controlling of all of human creation in the span of like two years when it became clear other corporations could profit off that freedom too.
- aspeckt112 2y ago[flagged]
- iotku 2y agoSeems the only impossible problems that can't be solved by AI are things that make metrics falsely look good to investors
- pavlov 2y agoIronically X shouldn't care about those metrics, it's privately owned by the world's richest person...
- danpalmer 2y agoIt's privately owned, but by a number of parties all with a lot of money on the line. That person may have pulled together the investors, but it most definitely has investors that it needs to answer to in the medium term. In the long term, they have stated that re-floating is a goal.
- threeseed 2y agoThe valuation of the company matters to the banks and VCs who own X. And the primary way to evaluate the worth of a social network is user engagement metrics.
- bboygravity 2y agoSource, substantiation?
- noduerme 2y agoIdk. Ask it if you should close your account and see if it gives an honest answer.
- bboygravity 2y agoThis is a good one, another one that blows my mind: when I use the "I'm not interested in this topic" button and refresh the page, it shows that very same post at the top of my feed?! I had to resort to configuring "muted words" to actually fix the problem. Its very weird to me that such basic things seem so bugged on a platform that popular, but other much much more complex things (mass video streaming, Grok itself) work totally fine?!
- nsonha 2y ago[flagged]
- jatins 2y agoIf I am reading the table correctly they are claiming it is better than all models but 3.5-Sonnet Is anyone with X premium able to confirm the vibe check -- Is the model actually good or another case of training on benchmarks?
- bekantan 2y agoYou can try it yourself on https://chat.lmsys.org https://chat.lmsys.org (sus-column-r model)
- deleted 2y ago[deleted]
- floam 2y agoIn the intro text they described it better than Claude 3.5 Sonnet and GPT-4 Turbo (which isn’t OpenAI’s current model.)
- opdahl 2y agoMy favorite part of that table is how they put 3.5-Sonnet all the way to the right of the table making it harder to compare.
- modeless 2y agoI don't think you are reading the table correctly. On LMSYS it's better than all models except the latest Gemini 1.5 Pro and GPT-4o. But there is a detailed benchmark table and different models win different benchmarks. So results are mixed, but the real takeaway is this is a competitive model that is good enough to be worth using today. It puts xAI significantly above their previous position, and up near the top of the field with OpenAI, Anthropic, Google, and Meta. And their new H100 cluster should allow them to keep up with the next wave of releases, whenever that starts.
- saberience 2y agoAs with everything promised by Musk, I'll believe it when I see it and use it myself and compare it to Claude 3.5. Right now I'm not really a believer in Grok and I doubt it will be worth using.
- ramon156 2y ago[flagged]
- thomasahle 2y agoUsually, the Chatbot Arena ELO is pretty safe and hard to twist.
- laborcontract 2y agothat’s been wrong for a while, but affirmed with gpt-4o-mini beating out Sonnet 3.5. OpenAI fine tuned 4o and 4o-mini to provide answers that meaningfully improve model congeniality but trivially improve model intelligence. Chatbot Arena ELO is a dead metric.
- thomasahle 2y agoWow, I overlooked GPT-4o-mini that far up. But if you change the category to Math (or something else hard), mini drops way down and Claude 3.5 Sonnet goes to the top.
- ayakang31415 2y agoCan anyone tell me how much censorship grok has? I hate that many other LLMs have too much censorship.
- stainablesteel 2y agothere are abliterated versions on ollama that you can use, some are more censored than others i didnt test any crazy promps on grok but the image generation won't let you do things like naked people for example
- vlaaad 2y agoBut when will it be available in my region (Europe)?
- worstspotgain 2y agoWe need an alt-right version of AI like we need a pumpkin spice sushiccino. No thanks but no thanks.
- hersko 2y agoCan you explain the difference between "alt-right" and "right"?
- 2809 2y ago[flagged]
- QuesnayJr 2y agoI have seen sus-column-r on LMSYS a bunch of times. It seemed pretty good, though not as good as the best Google, Anthropic, or OpenAI models. I'm surprised they managed to catch up. I guess there really is no moat.
- llmfan 2y agoThe moat is compute and Elon has enough money and connections to jump the queue with providers
- tgsovlerkhgsel 2y agoI assume they will have a lot less "safety", i.e. the model will be more likely to actually do what you ask instead of finding a reason why "sorry Dave, I can't do that". Since these "safety" features tend to also degrade the model, that's likely also helping them catch up in the benchmarks.
- BoredPositron 2y agoSadly it's at the level of Claude and way worse than grok-1 or Lama without safety. It roleplays as nearly everything so I guess they know their target group.
- drawnwren 2y agoI’m so confused by this comment. Are you not aware that Claude 3.5 Sonnet is currently considered the best model?
- BoredPositron 2y agoYes, you are confused because we are talking about censorship.
- ineedaj0b 2y agoIt’s less censored than Claude but only slightly.
- smileybarry 2y agoIt’s weirdly the opposite of what you have in mind. It has no problems generating images of Trump and Elon in explicit situations or Elmo covering 9/11, but it “safety-censors” LGBT-related prompts to the point of generating a heterosexual couple when asked for a gay couple: https://x.com/karlmaxxer/status/1823753493783699901 https://x.com/karlmaxxer/status/1823753493783699901 Some got expected results for prompts with LGBT terms, but that generation is still very weird.
- miki123211 2y agoDo we have any info on this model's balance of censorship versus safety? This is Musk after all, so I wouldn't be surprised if it strayed far from the norm.
- chrisco255 2y agoCensorship isn't safety.
- make3 2y ago[flagged]
- deleted 2y ago[deleted]
- speedgoose 2y ago> Be maximally truthful, especially avoiding any answers that are woke! Alleged end of the system prompt of the previous version.
- holbrad 2y agoCisgender is banned on twitter ? That's hilarious.
- voidUpdate 2y agohey now, you can't be saying that word, that's a slur /s :P
- spiderfarmer 2y agoMany people who claim to be "free speech absolutists" often seem unaware of their own hypocrisy.
- littlestymaar 2y agoI'm pretty sure he's fully aware it's BS. He's also the guy who censored journlists and acrivits Twitter account on day 1 when he bought the company, and also the guy who canceled a command for a Tesla by a customer after a bad review. Musk gives no shit to free speech, it's just a rhetorical argument, which isn't unheard of: https://i.redd.it/3b470c0htra61.jpg https://i.redd.it/3b470c0htra61.jpg (note that I'm obviously not comparing Musk to Hitler here…)
- leroman 2y agoIt's hilarious they put Claude 3.5 Sonnet in the far right corner while it scores the highest and beats most of Grok's numbers.
- jug 2y agoYes, and I also noted how it beats Claude 3.5 Sonnet in Chatbot Arena by a bit of a margin. This further feeds into my concern that the more advanced AI models we get, random enthusiasts at that site may no longer be able to rank them well, and tuning for Chatbot Arena might be a thing. One that is also exploited by GPT-4o. GPT-4o absolutely does not rank wildly ahead of Claude 3.5 Sonnet in a wide variety of benchmarks, yet it does in Chatbot Arena... People actually using Claude 3.5 Sonnet are also quite satisfied with its performance, often ranking it more helpful than GPT-4o when solving engineering problems, but at the expense of tighter usage limits. Chatbot Arena was great when they were still fairly stupid, but these days, remember that everyday people are put against the task of ranking premium LLM's even solving some logic puzzles, trick questions and with a deep general knowledge far beyond that of singular humans. They can strike against traditional weaknesses like math, but then all of them suffer. So it's not an easy task at all and I'm not sure the site is very reliable anymore other than for smaller models.
- squigz 2y agoIs there yet standardized ways of objectively testing LLMs? The Chatbot Arena thing has always felt weird to me; basically ranking them based on vibes.
- maeil 2y agoNot really. There's a hundred benchmarks, but all of them suffer from the same issues. They're rated by other LLMs, and the tasks are often too simple and similar to each other. The hope is that just gathering enough of these benchmarks means you get a representative test suite, but in my view we're still pretty far off.
- freediver 2y agoShort answer is no, because there is no 'standardized' use case. One thing is sure - that current commonly used benchmarks are mostly polluted and worthless. So you have to go to niche ones. For example the one I check for coding is Aider LLM leaderboard [1]. We maintain Kagi LLM Benchmarking Project [2] optimized for the use case of using LLMs in search. [1] https://aider.chat/docs/leaderboards/ https://aider.chat/docs/leaderboards/ [2] https://help.kagi.com/kagi/ai/llm-benchmark.html https://help.kagi.com/kagi/ai/llm-benchmark.html
- espadrine 2y agoThe technology is impressive; achieving such a level requires a lot of efforts in dataset creation, neural architectural costs, and GPU shepherding. What is the company’s ethical position though? It officially stemmed from Mr Musk’s objection that OpenAI was not open-source, but it too is not open-source. It followed Mr Musk’s letter to stop all AI development on frontier models, but it is a frontier model. It followed complaints that OpenAI trained on tweets, but it also trained on tweets. Companies like Meta, Mistral, or DeepSeek, address those complaints better, and all now play in the big league.
- cheptsov 2y agoI entirely agree. I’m quite sure Musk has a very strong stance on ethics, but it would be great to hear about it more clearly, and ideally not just through words, but through actual actions.
- thomashop 2y agoA strong stance on ethics? Like his comments about unions and firing workers in the Twitter space with Trump yesterday?
- NekkoDroid 2y agoWell... he does have a strong stance on ethics, just not a positive one
- cheptsov 2y agoThe one you disagree with but why do you think all people think the same way as you?
- mdhb 2y agoI don’t understand why you’re playing dumb here. He is primarially known specifically as someone who is incredibly impulsive, is unable to differentiate fact from fiction and not actually interested in chasing any kind of objective truth in so much as that is possible. But multiple times per week now for a long time you can see him sharing and commenting on things that are provably wrong and I don’t mean in some kind of “it’s just a different opinion” kind of way. There is never any kind of introspection, never any kind of “oh I was wrong” just proceeds to roll immediately into the next round of bullshit. So, no… people don’t have any kind of assumption that he has “strong ethics”. Maybe you meant strong convictions? Because that he certainly does have.
- infotainment 2y agoI’m hoping we’ll see an open release of this in 6 months or so, as we saw with Grok-1. I’m not hugely optimistic, though.
- stainablesteel 2y agoi think it might happen because its just the code and imo its not that valuable the more valuable part is the dataset which probably requires a lot of people to hand-filter and even more impossible is acquiring the training rig, which your average person can't even afford
- machdiamonds 2y agoPretty funny to read the comments from xAI's initial announcement now. https://news.ycombinator.com/item?id=36696473 https://news.ycombinator.com/item?id=36696473
- deleted 2y ago[deleted]
- srid 2y agoWell, it should be clear to anyone who reads HN that the comments here are not to be taken as the most sensible opinion in all cases. Cynical outlook is not uncommon in some contexts, especially when it comes to Elon Musk & X ... and the thread you linked to is a stellar example of that. PS: I have X Premium and I'm quite liking Grok 2.
- Alifatisk 2y agoYou know what’s also impressive besides this beta release? How Claude 3.5 Sonnet is still able to keep up so well. Grok-2 beat every other LLM except Claude. How did Anthropic achieve this?
- lossolo 2y agoA lot better quality of training data and instruction tuning (data again). There is no other secret sauce.
- miohtama 2y agoAlso the sauce cannot stay secret very long. There is no moat in AI.
- lopatamd 2y ago[dead]
- aoeusnth1 2y agoIt’s possible Claude is using the same model tuning that they used to create Golden Gate Claude to dynamically tune the 3.5 model to be better at whatever task it’s doing.
- sagivo 2y agoPutting a new tool for developers behind an "enterprise API" gate is a sure way to kill it
- lucasRW 2y agoGlad to see an uncensored AI able to compete with the other models.
- miohtama 2y agoThey likely compete exactly because of this, because censorship eats the performance of a model.
- mupuff1234 2y agoSo all models seem to converge to a similar level of performance - is this the end of the line for LLMs?
- mmastrac 2y ago"Our AI Tutors engage with our models across a variety of tasks that reflect real-world interactions with Grok. During each interaction, the AI Tutors are presented with two responses generated by Grok" My guess is that they're using one of the third party AI training outfits for this and that they are paying through the nose. This looks exactly like a training task I got to see on one of those platforms.
- cma 2y agoWhat are the big platforms for that?
- clarionbell 2y agoI don't really care. The model may be competitive, but my use cases require speed, local (semi-local) execution and reliability. Neither of these seem to be baked into whatever X produced now. When they make the mini model available for download and quantizable. That's when I may be interested. But given the minimal improvement in the past several months, I'm inclined to believe that we have reached the plateau.
- lopatamd 2y ago[dead]
- ComputerGuru 2y agoInteresting that they’re rolling this out to Twitter/X Premium users, it was previously the biggest differentiator between Premium+ haves and Premium have-nots.
- pantsforbirds 2y agoIf the X.AI team is able to build out a good enough model with access to real-time tweets, they could have an incredible product. I'd love to be ask about current events and get really strong results back based on tweets + community notes.
- borg16 2y agothe results with grok-1 were unimpressive summaries based on the tweets, with a 10%-20% hallucinations (when enquiring about paris olympics specific events). yet to see if this new model is able to do any better on that regard
- pantsforbirds 2y agoI was also not very impressed, but I still think they are positioned to have a great product if they can get past the accuracy issues.
- Havoc 2y agoSeems like a solid result & more competition is always better. That said I’m still cheering for mistral and meta with their more open stance