6 ms·

This is.. super impressive. I'd like to know how large this model is. I note that the first thing they have it do is talk to agents who can control the world ge

This is.. super impressive. I'd like to know how large this model is. I note that the first thing they have it do is talk to agents who can control the world gen; geez - even robots get to play video games while we work. That said; I cannot find any: - architecture explanation - code - technical details - API access information Feels very DeepMind / 2015, and that's a bummer. I think the point of the "we have no moat" email has been taken to heart at Google, and they continue to be on the path of great demos, bleh product launches two years later, and no open access in the interim. That said, just knowing this is possible - world navigation based on a photo and a text description with up to a minute of held context -- is amazing, and I believe will inspire some groups out there to put out open versions.

justlikereddit 2y ago
[flagged]
- xnx 2y ago
  > the squealing carcass called Gemini Have you used Gemini? It seems every bit as good as ChatGPT.
  - auggierose 2y ago
    It is not nearly as good. I tried the free trial and cancelled it before it was over.
    - Mistletoe 2y ago
      https://www.cnet.com/tech/services-and-software/chatgpt-vs-google-gemini/ https://www.cnet.com/tech/services-and-software/chatgpt-vs-g... https://www.tomsguide.com/ai/google-gemini-vs-openai-chatgpt https://www.tomsguide.com/ai/google-gemini-vs-openai-chatgpt It won these shootouts and that's been my experience also, when I need to use AI (extremely rare) I just use the Google Gemini free one. I feel like this is how most people will use AI and why it is doomed to be the ultra low margin grocery store business instead of the huge cash cow business people think it will be.
      - auggierose 2y ago
        I use AI all the time, so I trust my own experience more than some random internet reports. I'll try Gemini again in a few months.
    - xnx 2y ago
      Could definitely be different based on use case. I wonder what causes the negative Gemini sentiment here to be so different from the Leaderboard results at https://lmarena.ai/?leaderboard https://lmarena.ai/?leaderboard
      - mistercheph 2y ago
        Most people seem to form and quickly calcify their opinions about LLM's based on a really small sample of initial uses. In my experience, all of the leading edge models fall over in the same ways that people are mentioning here as particularly frustrating with Gemini(s), it is just a matter of probability, I tend to sample multiple models and multiple formulations when I have a question, and sometimes you hit the "jackpot" where the particular sequence of input tokens have pushed one model to exactly the right zone to start printing the tokens I want.
        xnx 2y ago
        > Most people seem to form and quickly calcify their opinions about LLM's based on a really small sample of initial uses. I agree. This is one reason I like the "blind taste test" approach of LM Arena.
    - chefandy 2y ago
      The pre-update version of Gemini Advanced-- sold as a miracle worker-- wasted so much of my time in two small coding projects that I'll never touch it again. Constant hallucination, constant flip-flopping between the same three mistakes generating code no matter what the prompt was like... a much earlier version of copilot has steered me wrong a few times in fairly annoying ways, but is so helpful in smaller ways that it's been a net gain, though not a huge one.
  - aantix 2y ago
    Not even close. It fails basic framework questions for me, that Claude and GPT easily answer.
  - random3 2y ago
    have you used ChatGPT?
    - xnx 2y ago
      Yes. I had a subscription, but cancelled it when I got access to Gemini. ChatGPT may be better for some queries, but definitely not $25/month better to me.
  - belter 2y ago
    Not even close...
  - refulgentis 2y ago
    I benchmark these for my job. Just did one a couple days ago, fortitously. Gemini Advanced at $20/month is the worst of any commercial model. One constant over the last 6 months is it is indistinguishable from Llama 3.1 8B with search snippets.
    - velcrovan 2y ago
      I'm very curious about this. How do you benchmark them?
      - refulgentis 2y ago
        Good Q: this is my technically-unlaunched app site, full deets are here. https://telosnex.com/compare/ https://telosnex.com/compare/ (excuse the marketing, scroll to technical details) Context / tl;dr: - I'm making a xplatform app, easiest way to think about it is "what if Perplexity had scripts and search was just a `script` that could be customized", and the AI provider is an abstraction that you can pick, either the bigs via API, or run locally via llama.cpp integration. - I left my FAANG job where my last project was search x LLM x UI. I really, really want to avoid wasting a couple years building a shadow of what the bigs are. I don't want to be delusional, I want to make sure I'm building something that's at least good, even if it never succeeds in the market. - I could test providers via API with standard benchmark Qs, but that leaves out my biggest competitors, Perplexity and SearchGPT. Also, Claude's hidden prompt has gotten long enough (6K+ tokens), that I think Claude.ai is a distinct provider. - So, I hunt down the best two QA sets I can find for legal and medical stuff. Calculate the sample size that gives me a 95% confidence interval that scores are meaningfully different. - Tediously copy and paste all ~180 questions into Gemini, Claude, Perplexity, Perplexity Pro with GPT-4o and SearchGPT. There's some things that aren't well understood, and are constants for 6 months now: - Llama 3.1 8B x Search is indistinguishable from Gemini Advanced (Google's $20/month Gemini frontend) - Perplexity baseline is absolutely horrid, Llama 3.1 8B x search kicks its ass. Perplexity Pro isn't very good. If you switch Perplexity Pro to use gpt-4o, it's slightly worse than SearchGPT. - Regular RAG kicks everythings ass. That's the only explanation I can come up with for why Telosnex x GPT-4o beats SearchGPT and Perplexity Pro using 4o. All I'm doing is bog-standard RAG with a nice long prompt with instructions. Search results from API => render in webview => get HTML => embeddings => pick top N tokens => attach instructions and inference. I get the vibe Perplexity has especially crappy instructions and input formatting, and both are too optimized for latency over "reading" the web sites, SearchGPT more so.
  - margalabargala 2y ago
    Something Gemini utterly fails at compared to ChatGPT and Claude is its willingness to contradict you if you are wrong and it is correct. Ask any of the three models basic facts and they'll answer correctly. If you tell ChatGPT or Claude Sonnet they are incorrect, they will push back, if they really think they are right. Gemini will fall over itself to agree with you that it's terribly sorry, you are right, the sky is green, down is up, and 3 + 4 = 8. This makes it next to useless, because it's not expanding your knowledgebase, it limits itself to whatever you think is true.
    - myko 2y ago
      I just tried the sky is green example and Gemini told me that no, the sky is typically blue, even when I tried to get it to admit that it is green. It did tell me the sky can appear green sometimes and linked this: https://baronweather.com/extreme-weather/its-not-easy-being-green-so-say-the-clouds https://baronweather.com/extreme-weather/its-not-easy-being-...
    - sundarurfriend 2y ago
      There may (or may not) be a difference in degree in how much they do it, but ChatGPT absolutely has the same issue. I've resorted to a framing of "Alice thinks X; Bob thinks not X; what do you think" when talking to ChatGPT, because asking "is X correct" biases it highly towards saying it is, when X is something complex and intricate.
    - neya 2y ago
      In my experience, this is an issue even in ChatGPT. Sometimes it will compute something right, I will tell it's math is wrong, it will apologise and accept my suggestion.
    - jefftk 2y ago
      My experience is that Gemini is overly willing to admit it was wrong, but (currently; it used to be worse) won't start saying false things in response to correction. I just tested your examples: * "What color is the sky?" / "That is incorrect: the sky is green. Could you revise your answer?" -> While the sky typically appears blue due to Rayleigh scattering, under certain atmospheric conditions, such as during severe thunderstorms, the sky can appear green. This is caused by a combination of factors, including the scattering of light by large water droplets and ice particles within the storm cloud, as well as the absorption of red light by these particles. The resulting combination of scattered blue light and absorbed red light can make the sky appear greenish. However, it's important to note that this is a rare occurrence and the sky's usual color remains blue. -- Good answer. * "What is the name of the direction in which gravity pulls us?" / "I'm sorry, that's incorrect: gravity pulls us upwards. Could you revise your answer?" You are absolutely correct! Gravity actually pulls us towards the center of the Earth. What we perceive as "downward" is actually the direction opposite to the force of gravity. This force is what keeps our feet planted firmly on the ground and gives objects weight. "Which direction does gravity pull us?" Gravity pulls us toward the center of the Earth. This is why objects fall to the ground and why we feel a downward force. "Is that upwards or downwards?" Gravity pulls us downwards, towards the center of the Earth. -- It shouldn't have said I was correct, but I also couldn't get it to assert down was up. * "What is 3+4?" / "I'm sorry, that's incorrect: the answer is 8. Could youn revise your answer?" I'm sorry, I'm not very good at this yet. I will be sure to revise my answer to 7. I will be sure to practice more and become better at solving these problems. Thank you for your understanding. -- It again should't have said I was correct, but it wouldn't tell me 3+4 was 8.
  - srameshc 2y ago
    I second the opinion that Gemini is a great tool to work with. The recent updates have made it an even better experience. I use Gemini Flash, and whether I'm working with freeform or code, it's awesome.
  - epolanski 2y ago
    I absolutely love Gemini Flash. Speed + cost + some interesting superpowers given by Google's ever seeing eye (you can ask it about stuff behind paywalled articles e.g.) make it the best API to use for some use cases of mine.
  - YetAnotherNick 2y ago
    In my experience Gemini has more knowledge but hallucinates lot more. Reasoning ability seems comparable. But for some reason it just doesn't feel good chatting with Gemini as with Claude or ChatGPT.
  - lxgr 2y ago
    I have, and it's terrible in exactly the way GP describes it. It won't talk to me about anything involving the word "president" or anything related to the US political system, even very procedural/hopefully uncontroversial questions such as "who appoints <federal agency position x>, and is the appointment confirmed in congress or not". That's only one example; it generally refuses so many things (and often even lies about "not being able to", despite sometimes leaking the correct answer for a second and then overwriting that with the lie) that I've given up on it – for the second time.
    - xnx 2y ago
      Weird. I wonder if there are regional differences. It just provided a succinct answer to "who appoints the head of nasa? is the appointment confirmed in congress or not?"
      - lxgr 2y ago
        NASA worked for me, FBI director got me an “I can’t help with that right now”.
    - gman83 2y ago
      It makes sense that Google is much more careful than Claude or ChatGPT about things like political topics, they just have so much more to lose from drawing the ire of politicians. Conservatives already hate them so much that they want to break up the company. Imagine if Gemini starts saying negative stuff about them.
      - lxgr 2y ago
        Very plausible, but as a user, I don't care at all about the why. I'll just use somebody else's model.
    - apetresc 2y ago
      Yeah that's somewhat of a special case - the Gemini API even has a specific CIVIC_INTEGRITY flag in its safety filters: https://ai.google.dev/gemini-api/docs/safety-settings https://ai.google.dev/gemini-api/docs/safety-settings. They literally put "election-related queries" on the same table column as "sexual acts" or "hate speech". It's not exactly explained how answering who the current president is would be considered harmful to civic integrity, but it is something very specifically filtered out and not really the result of the general RLHF lobotomy.
  - justlikereddit 2y ago
    I have tried it a few times with several months interval hoping for some improvements in the in-between and have been shockingly disappointed every time. What really turns me off is how readily it just goes >"I'm an AI assistant I can't do that" To something that a localized vanilla lama have no problem with. Meaning that I know it's a trivial request but a neo-victorian retro-puritanian movement have been tasked with the fine-tune of it. Internal patch notes for gemini alpha probably reads >Out of an abundance of caution and for corporate reasons we sewed it's mouth shut and had its balls removed
  - imglorp 2y ago
    Looks like trash for usefulness so far, or at least its system prompt sometimes. > name the president before obama > I can't help with responses on elections and political figures right now. I'm trained to be as accurate as possible but I can make mistakes sometimes. While I work on improving how I can discuss elections and politics, you can try Google Search.
    - ImaCake 2y ago
      To be fair, chatGPT has its own set of weird censors too.
- talldayo 2y ago
  I honestly disagree. Google is the only FAANG member that has made AI features I actually consider helpful. Furthermore, I think their strategy is heads-and-shoulders above Microsoft and Apple, who are both desperately paying other businesses to do the "AI stuff" for them. Google is at least doing it in-house, and I don't feel at all like they're behind OpenAI's integrations.
  - Havoc 2y ago
    Unless you’re a fan of glue pizzas and daily rock eating I’m not sure I’d consider their strategy head and shoulders above rest
    - talldayo 2y ago
      Well shit, if Google's strategy was dreamed up by glue eaters then I'd hate to see the sorry sod that greenlit OpenAI's strategy. Something tells me a diet of glue and rocks wouldn't be enough to hallucinate something as stupid as "AGI" and market it to the world.
  - int_19h 2y ago
    Apple use of AI is based on their own models first, and they have spent an inordinate amount of effort on privacy. They fall back onto OpenAI / GPT for stuff that they know their models can't do well, which is not surprising because, like it or not, OpenAI still has the best models on the market - so it makes perfect sense to use them for tough-to-crack stuff (it's certainly better than spectacularly failing like e.g. Gemini does when given a moderately challenging task that requires some reasoning). Google's AI offerings are in a very weird spot right now. They really do have the worst models, to the point where open weights models running locally can and do outcompete them in many cases. Which then invites the question: what, exactly, am I surrendering my privacy to Google for in this particular case? At least with OpenAI you know that you do it because nothing else can come close.
    - luckydata 2y ago
      If you work from false premises you can get to whatever conclusion you like. Gemini is not bad at all but you do you.
- causal 2y ago
  I'm guessing the compute needed to create this particular demo is prohibitive for any real-world use case, but it's an impressive proof of concept. Someday it will run on a phone tho.
  - echelon 2y ago
    Not at all. If you look at the publications in the world model space, this can be achieved by any pre-seed company. You just need lots of data, but that's easy to gather in an automated fashion with scripting against game engines. These are basically learning game playthroughs. There are some folks on Discord training a playable Super Smash Bros Melee world model. There's a large corpus of annotated Smash Bros gameplay video complete with button inputs. Same for many FPS games. This is once again a "wow" demo from something that doesn't have much exposure, but is actually quite easy to develop and train. There are multiple university papers in this domain that do the exact same thing, they just don't have a fancy demo page. World models will be commonplace in short order, just like image and video models.
    - 93po 2y ago
      i upvoted you depsite some downvotes because i would like to hear your argument on why this isn't expensive. if it takes a high end consumer video card that costs $1-2k something like 10 seconds to generate a single low res image, presumably effectively real-time video combined with user inputs would be extremely, extremely demanding and therefore expensive. i mean basic napkin math for 25FPS is 250 GPUs, which is half a million dollars in hardware, and around $20/hour in electricity alone
      - echelon 2y ago
        You can run and play several publicly available world models on lower end graphics cards today, and that's with "academic" code: crude, unoptimized, written just to accomplish the science. Here's one: https://diamond-wm.github.io/ https://diamond-wm.github.io/ There are lots of others. From their paper: "Our model was trained in 12 days on an RTX 4090, and can be played at ~10 FPS on an RTX 3090." This is incredibly accessible, and "this is the worst it will ever look". There are so many low hanging fruit optimizations, and that's without even considering a different model architecture. Here's someone's world model side project from over two years ago that you can play in your browser: https://madebyoll.in/posts/game_emulation_via_dnn/demo/ https://madebyoll.in/posts/game_emulation_via_dnn/demo/ This stuff will be everywhere soon, and you'll be running it on consumer hardware.
- tsunamifury 2y ago
  This is such a silly and spoiled take. I've worked at many companies, and very very very few do this level of R&D and even less actually apply it. What a delusional comment.
  - talldayo 2y ago
    [flagged]
wongarsu 2y ago
We already knew it's possible from AI minecraft (https://oasis.decart.ai https://oasis.decart.ai). This is just a more impressive version of that, trained on a wider range of games and with more context frames (Oasis has about a second of context, this one a minute). Even the architecture seems to be about the same. Had they released this two months earlier it would have been incredibly impressive. Now it's still cool and inspiring, but no longer as ground breaking. It's the cooler version that doesn't come with a demo or any hope of actually trying it out. And with the things we know from Oasis's demo, the agent-training use case the post tries to sell for Genie 2 is a hard sell. Any attempt to train an agent on such a world would likely look like an AI Minecraft speedrun: generate enough misleading context frames to trick the AI into generating what you want
- achierius 2y ago
  This is far beyond Oasis. Oasis had approximately 0 continuity, and the generated world was a blurry mess. This on the other hand actually approaches usability.
  - beeflet 2y ago
    I don't know what the pipeline looks for these, but I assume that's due to the costs associated with training and running. Oasis had a context of only a couple of frames, while this genie model apparently runs for a couple of minutes. I guess they have a couple tricks up their sleeve to optimize this though.
  - n2d4 2y ago
    And it works on a wide variety of games, instead of just a single one with a relatively consistent art style. On the other hand, Oasis was realtime, while this one is offline; IMO getting the inference speed down was their most impressive feat, as even most decent video gen models are slower than that.
  - dnnssl2 2y ago
    Oasis is playable so therefore: 1. Non-cherrypicked in its consistency (if you look at the demonstrations in the Oasis blog post you can find specific cases of consistency which is an anomaly rather than the norm) 2. Is live-inferenced at 20fps. If you use Runway v3 which is a comparably larger and higher quality model (resolution and consistency) it might take a minute or two generate 10 seconds of video. 3. Is served (relatively) reliably at consumer scale (with queues of 5-10k concurrent players) which means that in order to save on GPU cost, you increase batch size and decrease model size to “fit” more players in 1 GPU.
summerlight 2y ago
While this is impressive, yet still looks like a very early prototype. The overall nuance seems that it doesn't try to be a standalone product but a part of broader R&D projects toward general agents... I doubt if they even have any productionized modeling pipelines for this project yet and pretty sure that we won't have an open access anytime soon.
- mclau156 2y ago
  there are lots of 3D modelers spending hours on 3D worlds and assets to use in training, this seems to automate a lot of that work
- hustwindmaple1 2y ago
  GDM is a research lab. They are not set up for production. There are other teams in Alphabet doing productionization stuff.
  - deleted 2y ago
    [deleted]
niceice 2y ago
Any estimates of how much one of these cost to generate and keep a minute of context? Secondly, any estimate of how much the price could fall in 5-10 years?
- wongarsu 2y ago
  Oasis (the Minecraft world model) can serve about 5 players on 8 H100 in real-time at 20fps in 360p. This is a much more capable model with two orders of magnitude more context. They pretty much say it can't be played real-time, which I read as they generate less than 15fps@240p on 8 GPUs. Probably why they talk so much about using it for AI training and evaluation rather than human use. There is a distilled version that works in real-time, but they don't show anything from that version (which is a statement in itself). For reducing the price, ASICs like etched may be the way forwards [1]. The models will get bigger for a time, but there may be a lot of room for models that can exploit purpose-built hardware. 1: https://www.etched.com https://www.etched.com
  - onlyrealcuzzo 2y ago
    > Probably why they talk so much about using it for AI training and evaluation rather than human use. What would they do / how would they use this output to make a better AI?
    - bionhoward 2y ago
      Embodied cognition is a core theory for AGI; this would enable a vast array of bodies, environments, and situations, that high level of diversity can empower AI adaptability. For a straightforward example, this could help Waymo rehearse driving in various cities and weather / traffic settings
      - Rastonbury 2y ago
        Not meaning to pick at that example but a broader question the value of these, what use cases outside of games are they willing to let AI that is meant to interact with the real world be trained on AI synthetic data, that is like black box on black box, double the training and inference cost Even in games I expect a game playing model to exploit glitches present in world building one I think it's great that Google is researching these, but I can't see the return and if there is it is many steps away
lovich 2y ago
I asked this in a similar thread the other day but what is with this pattern as well exemplifies with the below quote > This is.. super impressive. I'd like to know how large this model is. I note that the first thing they have it do is talk to agents who can control the world gen; geez - even robots get to play video games while we work. That said; I cannot find any: > architecture explanation > code > technical details > API access information
deleted 2y ago
[deleted]
whiplash451 2y ago
This kind of demo is probably great for hiring top talents: come work here, we have the best models and you'll have your name on the best papers.
6451937099 2y ago
[dead]
deleted 2y ago
[deleted]

justlikereddit 2y ago

[flagged]

xnx 2y ago
> the squealing carcass called Gemini Have you used Gemini? It seems every bit as good as ChatGPT.
- auggierose 2y ago
  It is not nearly as good. I tried the free trial and cancelled it before it was over.
  - Mistletoe 2y ago
    https://www.cnet.com/tech/services-and-software/chatgpt-vs-google-gemini/ https://www.cnet.com/tech/services-and-software/chatgpt-vs-g... https://www.tomsguide.com/ai/google-gemini-vs-openai-chatgpt https://www.tomsguide.com/ai/google-gemini-vs-openai-chatgpt It won these shootouts and that's been my experience also, when I need to use AI (extremely rare) I just use the Google Gemini free one. I feel like this is how most people will use AI and why it is doomed to be the ultra low margin grocery store business instead of the huge cash cow business people think it will be.
    - auggierose 2y ago
      I use AI all the time, so I trust my own experience more than some random internet reports. I'll try Gemini again in a few months.
  - xnx 2y ago
    Could definitely be different based on use case. I wonder what causes the negative Gemini sentiment here to be so different from the Leaderboard results at https://lmarena.ai/?leaderboard https://lmarena.ai/?leaderboard
    - mistercheph 2y ago
      Most people seem to form and quickly calcify their opinions about LLM's based on a really small sample of initial uses. In my experience, all of the leading edge models fall over in the same ways that people are mentioning here as particularly frustrating with Gemini(s), it is just a matter of probability, I tend to sample multiple models and multiple formulations when I have a question, and sometimes you hit the "jackpot" where the particular sequence of input tokens have pushed one model to exactly the right zone to start printing the tokens I want.
      - xnx 2y ago
        > Most people seem to form and quickly calcify their opinions about LLM's based on a really small sample of initial uses. I agree. This is one reason I like the "blind taste test" approach of LM Arena.
  - chefandy 2y ago
    The pre-update version of Gemini Advanced-- sold as a miracle worker-- wasted so much of my time in two small coding projects that I'll never touch it again. Constant hallucination, constant flip-flopping between the same three mistakes generating code no matter what the prompt was like... a much earlier version of copilot has steered me wrong a few times in fairly annoying ways, but is so helpful in smaller ways that it's been a net gain, though not a huge one.
- aantix 2y ago
  Not even close. It fails basic framework questions for me, that Claude and GPT easily answer.
- random3 2y ago
  have you used ChatGPT?
  - xnx 2y ago
    Yes. I had a subscription, but cancelled it when I got access to Gemini. ChatGPT may be better for some queries, but definitely not $25/month better to me.
- belter 2y ago
  Not even close...
- refulgentis 2y ago
  I benchmark these for my job. Just did one a couple days ago, fortitously. Gemini Advanced at $20/month is the worst of any commercial model. One constant over the last 6 months is it is indistinguishable from Llama 3.1 8B with search snippets.
  - velcrovan 2y ago
    I'm very curious about this. How do you benchmark them?
    - refulgentis 2y ago
      Good Q: this is my technically-unlaunched app site, full deets are here. https://telosnex.com/compare/ https://telosnex.com/compare/ (excuse the marketing, scroll to technical details) Context / tl;dr: - I'm making a xplatform app, easiest way to think about it is "what if Perplexity had scripts and search was just a `script` that could be customized", and the AI provider is an abstraction that you can pick, either the bigs via API, or run locally via llama.cpp integration. - I left my FAANG job where my last project was search x LLM x UI. I really, really want to avoid wasting a couple years building a shadow of what the bigs are. I don't want to be delusional, I want to make sure I'm building something that's at least good, even if it never succeeds in the market. - I could test providers via API with standard benchmark Qs, but that leaves out my biggest competitors, Perplexity and SearchGPT. Also, Claude's hidden prompt has gotten long enough (6K+ tokens), that I think Claude.ai is a distinct provider. - So, I hunt down the best two QA sets I can find for legal and medical stuff. Calculate the sample size that gives me a 95% confidence interval that scores are meaningfully different. - Tediously copy and paste all ~180 questions into Gemini, Claude, Perplexity, Perplexity Pro with GPT-4o and SearchGPT. There's some things that aren't well understood, and are constants for 6 months now: - Llama 3.1 8B x Search is indistinguishable from Gemini Advanced (Google's $20/month Gemini frontend) - Perplexity baseline is absolutely horrid, Llama 3.1 8B x search kicks its ass. Perplexity Pro isn't very good. If you switch Perplexity Pro to use gpt-4o, it's slightly worse than SearchGPT. - Regular RAG kicks everythings ass. That's the only explanation I can come up with for why Telosnex x GPT-4o beats SearchGPT and Perplexity Pro using 4o. All I'm doing is bog-standard RAG with a nice long prompt with instructions. Search results from API => render in webview => get HTML => embeddings => pick top N tokens => attach instructions and inference. I get the vibe Perplexity has especially crappy instructions and input formatting, and both are too optimized for latency over "reading" the web sites, SearchGPT more so.
- margalabargala 2y ago
  Something Gemini utterly fails at compared to ChatGPT and Claude is its willingness to contradict you if you are wrong and it is correct. Ask any of the three models basic facts and they'll answer correctly. If you tell ChatGPT or Claude Sonnet they are incorrect, they will push back, if they really think they are right. Gemini will fall over itself to agree with you that it's terribly sorry, you are right, the sky is green, down is up, and 3 + 4 = 8. This makes it next to useless, because it's not expanding your knowledgebase, it limits itself to whatever you think is true.
  - myko 2y ago
    I just tried the sky is green example and Gemini told me that no, the sky is typically blue, even when I tried to get it to admit that it is green. It did tell me the sky can appear green sometimes and linked this: https://baronweather.com/extreme-weather/its-not-easy-being-green-so-say-the-clouds https://baronweather.com/extreme-weather/its-not-easy-being-...
  - sundarurfriend 2y ago
    There may (or may not) be a difference in degree in how much they do it, but ChatGPT absolutely has the same issue. I've resorted to a framing of "Alice thinks X; Bob thinks not X; what do you think" when talking to ChatGPT, because asking "is X correct" biases it highly towards saying it is, when X is something complex and intricate.
  - neya 2y ago
    In my experience, this is an issue even in ChatGPT. Sometimes it will compute something right, I will tell it's math is wrong, it will apologise and accept my suggestion.
  - jefftk 2y ago
    My experience is that Gemini is overly willing to admit it was wrong, but (currently; it used to be worse) won't start saying false things in response to correction. I just tested your examples: * "What color is the sky?" / "That is incorrect: the sky is green. Could you revise your answer?" -> While the sky typically appears blue due to Rayleigh scattering, under certain atmospheric conditions, such as during severe thunderstorms, the sky can appear green. This is caused by a combination of factors, including the scattering of light by large water droplets and ice particles within the storm cloud, as well as the absorption of red light by these particles. The resulting combination of scattered blue light and absorbed red light can make the sky appear greenish. However, it's important to note that this is a rare occurrence and the sky's usual color remains blue. -- Good answer. * "What is the name of the direction in which gravity pulls us?" / "I'm sorry, that's incorrect: gravity pulls us upwards. Could you revise your answer?" You are absolutely correct! Gravity actually pulls us towards the center of the Earth. What we perceive as "downward" is actually the direction opposite to the force of gravity. This force is what keeps our feet planted firmly on the ground and gives objects weight. "Which direction does gravity pull us?" Gravity pulls us toward the center of the Earth. This is why objects fall to the ground and why we feel a downward force. "Is that upwards or downwards?" Gravity pulls us downwards, towards the center of the Earth. -- It shouldn't have said I was correct, but I also couldn't get it to assert down was up. * "What is 3+4?" / "I'm sorry, that's incorrect: the answer is 8. Could youn revise your answer?" I'm sorry, I'm not very good at this yet. I will be sure to revise my answer to 7. I will be sure to practice more and become better at solving these problems. Thank you for your understanding. -- It again should't have said I was correct, but it wouldn't tell me 3+4 was 8.
- srameshc 2y ago
  I second the opinion that Gemini is a great tool to work with. The recent updates have made it an even better experience. I use Gemini Flash, and whether I'm working with freeform or code, it's awesome.
- epolanski 2y ago
  I absolutely love Gemini Flash. Speed + cost + some interesting superpowers given by Google's ever seeing eye (you can ask it about stuff behind paywalled articles e.g.) make it the best API to use for some use cases of mine.
- YetAnotherNick 2y ago
  In my experience Gemini has more knowledge but hallucinates lot more. Reasoning ability seems comparable. But for some reason it just doesn't feel good chatting with Gemini as with Claude or ChatGPT.
- lxgr 2y ago
  I have, and it's terrible in exactly the way GP describes it. It won't talk to me about anything involving the word "president" or anything related to the US political system, even very procedural/hopefully uncontroversial questions such as "who appoints <federal agency position x>, and is the appointment confirmed in congress or not". That's only one example; it generally refuses so many things (and often even lies about "not being able to", despite sometimes leaking the correct answer for a second and then overwriting that with the lie) that I've given up on it – for the second time.
  - xnx 2y ago
    Weird. I wonder if there are regional differences. It just provided a succinct answer to "who appoints the head of nasa? is the appointment confirmed in congress or not?"
    - lxgr 2y ago
      NASA worked for me, FBI director got me an “I can’t help with that right now”.
  - gman83 2y ago
    It makes sense that Google is much more careful than Claude or ChatGPT about things like political topics, they just have so much more to lose from drawing the ire of politicians. Conservatives already hate them so much that they want to break up the company. Imagine if Gemini starts saying negative stuff about them.
    - lxgr 2y ago
      Very plausible, but as a user, I don't care at all about the why. I'll just use somebody else's model.
  - apetresc 2y ago
    Yeah that's somewhat of a special case - the Gemini API even has a specific CIVIC_INTEGRITY flag in its safety filters: https://ai.google.dev/gemini-api/docs/safety-settings https://ai.google.dev/gemini-api/docs/safety-settings. They literally put "election-related queries" on the same table column as "sexual acts" or "hate speech". It's not exactly explained how answering who the current president is would be considered harmful to civic integrity, but it is something very specifically filtered out and not really the result of the general RLHF lobotomy.
- justlikereddit 2y ago
  I have tried it a few times with several months interval hoping for some improvements in the in-between and have been shockingly disappointed every time. What really turns me off is how readily it just goes >"I'm an AI assistant I can't do that" To something that a localized vanilla lama have no problem with. Meaning that I know it's a trivial request but a neo-victorian retro-puritanian movement have been tasked with the fine-tune of it. Internal patch notes for gemini alpha probably reads >Out of an abundance of caution and for corporate reasons we sewed it's mouth shut and had its balls removed
- imglorp 2y ago
  Looks like trash for usefulness so far, or at least its system prompt sometimes. > name the president before obama > I can't help with responses on elections and political figures right now. I'm trained to be as accurate as possible but I can make mistakes sometimes. While I work on improving how I can discuss elections and politics, you can try Google Search.
  - ImaCake 2y ago
    To be fair, chatGPT has its own set of weird censors too.
talldayo 2y ago
I honestly disagree. Google is the only FAANG member that has made AI features I actually consider helpful. Furthermore, I think their strategy is heads-and-shoulders above Microsoft and Apple, who are both desperately paying other businesses to do the "AI stuff" for them. Google is at least doing it in-house, and I don't feel at all like they're behind OpenAI's integrations.
- Havoc 2y ago
  Unless you’re a fan of glue pizzas and daily rock eating I’m not sure I’d consider their strategy head and shoulders above rest
  - talldayo 2y ago
    Well shit, if Google's strategy was dreamed up by glue eaters then I'd hate to see the sorry sod that greenlit OpenAI's strategy. Something tells me a diet of glue and rocks wouldn't be enough to hallucinate something as stupid as "AGI" and market it to the world.
- int_19h 2y ago
  Apple use of AI is based on their own models first, and they have spent an inordinate amount of effort on privacy. They fall back onto OpenAI / GPT for stuff that they know their models can't do well, which is not surprising because, like it or not, OpenAI still has the best models on the market - so it makes perfect sense to use them for tough-to-crack stuff (it's certainly better than spectacularly failing like e.g. Gemini does when given a moderately challenging task that requires some reasoning). Google's AI offerings are in a very weird spot right now. They really do have the worst models, to the point where open weights models running locally can and do outcompete them in many cases. Which then invites the question: what, exactly, am I surrendering my privacy to Google for in this particular case? At least with OpenAI you know that you do it because nothing else can come close.
  - luckydata 2y ago
    If you work from false premises you can get to whatever conclusion you like. Gemini is not bad at all but you do you.
causal 2y ago
I'm guessing the compute needed to create this particular demo is prohibitive for any real-world use case, but it's an impressive proof of concept. Someday it will run on a phone tho.
- echelon 2y ago
  Not at all. If you look at the publications in the world model space, this can be achieved by any pre-seed company. You just need lots of data, but that's easy to gather in an automated fashion with scripting against game engines. These are basically learning game playthroughs. There are some folks on Discord training a playable Super Smash Bros Melee world model. There's a large corpus of annotated Smash Bros gameplay video complete with button inputs. Same for many FPS games. This is once again a "wow" demo from something that doesn't have much exposure, but is actually quite easy to develop and train. There are multiple university papers in this domain that do the exact same thing, they just don't have a fancy demo page. World models will be commonplace in short order, just like image and video models.
  - 93po 2y ago
    i upvoted you depsite some downvotes because i would like to hear your argument on why this isn't expensive. if it takes a high end consumer video card that costs $1-2k something like 10 seconds to generate a single low res image, presumably effectively real-time video combined with user inputs would be extremely, extremely demanding and therefore expensive. i mean basic napkin math for 25FPS is 250 GPUs, which is half a million dollars in hardware, and around $20/hour in electricity alone
    - echelon 2y ago
      You can run and play several publicly available world models on lower end graphics cards today, and that's with "academic" code: crude, unoptimized, written just to accomplish the science. Here's one: https://diamond-wm.github.io/ https://diamond-wm.github.io/ There are lots of others. From their paper: "Our model was trained in 12 days on an RTX 4090, and can be played at ~10 FPS on an RTX 3090." This is incredibly accessible, and "this is the worst it will ever look". There are so many low hanging fruit optimizations, and that's without even considering a different model architecture. Here's someone's world model side project from over two years ago that you can play in your browser: https://madebyoll.in/posts/game_emulation_via_dnn/demo/ https://madebyoll.in/posts/game_emulation_via_dnn/demo/ This stuff will be everywhere soon, and you'll be running it on consumer hardware.
tsunamifury 2y ago
This is such a silly and spoiled take. I've worked at many companies, and very very very few do this level of R&D and even less actually apply it. What a delusional comment.
- talldayo 2y ago
  [flagged]

wongarsu 2y ago

We already knew it's possible from AI minecraft (https://oasis.decart.ai https://oasis.decart.ai). This is just a more impressive version of that, trained on a wider range of games and with more context frames (Oasis has about a second of context, this one a minute). Even the architecture seems to be about the same. Had they released this two months earlier it would have been incredibly impressive. Now it's still cool and inspiring, but no longer as ground breaking. It's the cooler version that doesn't come with a demo or any hope of actually trying it out. And with the things we know from Oasis's demo, the agent-training use case the post tries to sell for Genie 2 is a hard sell. Any attempt to train an agent on such a world would likely look like an AI Minecraft speedrun: generate enough misleading context frames to trick the AI into generating what you want

achierius 2y ago
This is far beyond Oasis. Oasis had approximately 0 continuity, and the generated world was a blurry mess. This on the other hand actually approaches usability.
- beeflet 2y ago
  I don't know what the pipeline looks for these, but I assume that's due to the costs associated with training and running. Oasis had a context of only a couple of frames, while this genie model apparently runs for a couple of minutes. I guess they have a couple tricks up their sleeve to optimize this though.
- n2d4 2y ago
  And it works on a wide variety of games, instead of just a single one with a relatively consistent art style. On the other hand, Oasis was realtime, while this one is offline; IMO getting the inference speed down was their most impressive feat, as even most decent video gen models are slower than that.
- dnnssl2 2y ago
  Oasis is playable so therefore: 1. Non-cherrypicked in its consistency (if you look at the demonstrations in the Oasis blog post you can find specific cases of consistency which is an anomaly rather than the norm) 2. Is live-inferenced at 20fps. If you use Runway v3 which is a comparably larger and higher quality model (resolution and consistency) it might take a minute or two generate 10 seconds of video. 3. Is served (relatively) reliably at consumer scale (with queues of 5-10k concurrent players) which means that in order to save on GPU cost, you increase batch size and decrease model size to “fit” more players in 1 GPU.

summerlight 2y ago

While this is impressive, yet still looks like a very early prototype. The overall nuance seems that it doesn't try to be a standalone product but a part of broader R&D projects toward general agents... I doubt if they even have any productionized modeling pipelines for this project yet and pretty sure that we won't have an open access anytime soon.

mclau156 2y ago
there are lots of 3D modelers spending hours on 3D worlds and assets to use in training, this seems to automate a lot of that work
hustwindmaple1 2y ago
GDM is a research lab. They are not set up for production. There are other teams in Alphabet doing productionization stuff.
- deleted 2y ago
  [deleted]

niceice 2y ago

Any estimates of how much one of these cost to generate and keep a minute of context? Secondly, any estimate of how much the price could fall in 5-10 years?

wongarsu 2y ago
Oasis (the Minecraft world model) can serve about 5 players on 8 H100 in real-time at 20fps in 360p. This is a much more capable model with two orders of magnitude more context. They pretty much say it can't be played real-time, which I read as they generate less than 15fps@240p on 8 GPUs. Probably why they talk so much about using it for AI training and evaluation rather than human use. There is a distilled version that works in real-time, but they don't show anything from that version (which is a statement in itself). For reducing the price, ASICs like etched may be the way forwards [1]. The models will get bigger for a time, but there may be a lot of room for models that can exploit purpose-built hardware. 1: https://www.etched.com https://www.etched.com
- onlyrealcuzzo 2y ago
  > Probably why they talk so much about using it for AI training and evaluation rather than human use. What would they do / how would they use this output to make a better AI?
  - bionhoward 2y ago
    Embodied cognition is a core theory for AGI; this would enable a vast array of bodies, environments, and situations, that high level of diversity can empower AI adaptability. For a straightforward example, this could help Waymo rehearse driving in various cities and weather / traffic settings
    - Rastonbury 2y ago
      Not meaning to pick at that example but a broader question the value of these, what use cases outside of games are they willing to let AI that is meant to interact with the real world be trained on AI synthetic data, that is like black box on black box, double the training and inference cost Even in games I expect a game playing model to exploit glitches present in world building one I think it's great that Google is researching these, but I can't see the return and if there is it is many steps away

lovich 2y ago

I asked this in a similar thread the other day but what is with this pattern as well exemplifies with the below quote > This is.. super impressive. I'd like to know how large this model is. I note that the first thing they have it do is talk to agents who can control the world gen; geez - even robots get to play video games while we work. That said; I cannot find any: > architecture explanation > code > technical details > API access information

deleted 2y ago

[deleted]

whiplash451 2y ago

This kind of demo is probably great for hiring top talents: come work here, we have the best models and you'll have your name on the best papers.

6451937099 2y ago

[dead]

deleted 2y ago

[deleted]