10 ms·
Gemini Flash
- causal 2y ago1M token context by default is the big feature here IMO, but we need better benchmarks to measure what that really means. My intuition is that as contexts get longer we start hitting the limits of how much comprehension can be embedded in a single point of vector space, and will need better architectures for selecting the relevant portions of the context.
- dragonwriter 2y ago> 1M token context by default is the big feature here IMO, but we need better benchmarks to measure what that really means. Multimodality in a model That's between 4-7% the cost per token of OpenAI’s cheapest multimodal model is an important feature when you are talking about production use and not just economically unsustainable demos.
- refulgentis 2y agoIn preview, can't be used in production, they already rug-pulled people building on Gemini w/r/t cost and RPM, and they're pointedly not putting any RPM or cost on the page. (seriously, try finding info on cost, RPM, or release right now, you're linked in circles.) Agree on OpenAI multimodal but it's sort of a stilted example itself, it's because OpenAI has a hole in its lineup - ex. Claude Haiku is multimodal, faster, and significantly cheaper than GPT 3.5.
- dragonwriter 2y ago> they're pointedly not putting any RPM or cost on the page 360 RPM base limit, pricing is posted. > seriously, try finding info on cost, RPM, or release right now, I wasn't making up numbers, its on their Gemini API pricing page: https://ai.google.dev/pricing https://ai.google.dev/pricing
- refulgentis 2y agoNice, thanks (btw, I didn't think you were making it up, it was in the keynote!)
- causal 2y ago+1 on Haiku being oft overlooked.
- verdverm 2y agoShows the power of the brand and the limit of names consumers will recall long term "Who are the biggest soda or potato chip makers?"
- refulgentis 2y ago[flagged]
- verdverm 2y ago1. Not an AI specific comment series, it applies generally 2. Comments like yours are against the guidelines of HN 3. You don't know what I do and are incorrect in your assessment of my work and experience (another common HN mishap) --- (re: #2/3) Please make your substantive points without crossing into personal attack. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- anoncareer0212 2y agoWhat do you mean? A) This fills a gaping hole for cheap multimodal models, OpenAI doesn't have one B) Anthropic's Haiku is a good choice. You) wow A didn't know Anthropic. Goes to show power of brands, much like snack foods B) Eh I wouldn't conclude anything from A. Its one comment. some people don't know what an Anthropic is because there's high interest in AI relative to interest in AI APIs. you can expect a low SNR ratio, even on HN You) Stop personally attacking me! It's against the rules!!
- verdverm 2y ago> You) Stop personally attacking me! It's against the rules!! Your comment history certainly shows a disregard for the HN guidelines https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- leetharris 2y agoThe problem is that even 1.5 Pro seems completely useless for long context multimodal stuff. I have tried it for so many use cases in video / audio and it hallucinates an unbelievable amount. More than any other model I've ever used. So if 1.5 Pro can't even handle simple tasks without hallucination, I imagine this tiny model is even more useless.
- refulgentis 2y agoYeah it's not very good in practice, you can get a halfway decent demo out of it ("look I gave it 6.5 harry potters and it made an SVG map connecting characters with annotations!!"...some of the characters...spare annotations...cost $20). Just good enough to fool you a couple times when you try to make it work 10 times.
- WhitneyLand 2y agoLimitations of single point in vector space of what dimension? I’m not sure it’s public knowledge, but it’s an architecture choice. They choose how big to make the embedding dimension. My point is just that there’s no limitation in principle, it’s just a matter of how they design it and resource constraints.
- causal 2y agoThanks for responding to that point - it's the one most on my mind. So OpenAI's large embedding model has 3072 dimensions, though in practice far fewer are probably used. Clearly you can't compress 1M tokens down to 3072. Yet those 3072 numbers are all you've got for capturing the full meaning of the previous token when predicting the next one; including all 1M tokens of modifying context. So perhaps human language is simply never complex enough to need more than 3072 numbers to represent a given train of thought, but that doesn't seem clear to me. Edit: Since Gemini is relevant here, it looks like their text embedding model is 768 dimensions.
- neverokay 2y agoSo perhaps human language is simply never complex enough to need more than 3072 numbers to represent a given train of thought, but that doesn't seem clear to me. Will compute allow that number to go up? Or is that an optimal number?
- WhitneyLand 2y agoDefinitely has trended upward, there’s no special number. It’s just a matter of how much compute, storage, time to allocate to that part of the architecture.
- causal 2y agoHas it? It's my understanding that GPT-3 was 12,288 and GPT-4 went down to 3072.
- shoelessone 2y ago> My intuition is that as contexts get longer we start hitting the limits of how much comprehension can be embedded in a single point of vector space, and will need better architectures for selecting the relevant portions of the context. Is it possible to explain what this means in a way that somebody only roughly familiar with vectors and vector databases? Or recommend an article or further reading on the topic?
- causal 2y agoSo most of my understanding comes from this series, particularly the last two videos: https://www.3blue1brown.com/topics/neural-networks https://www.3blue1brown.com/topics/neural-networks Essentially each token of a text occupies a point in a many-dimensional model that represents meaning, and LLMs predict the next token by modifying the last token with the context of all the tokens before it. Attention heads are basically a way of choosing which prior tokens are most relevant and adjusting the last token's point in vector-space accordingly.
- sojuz151 2y ago>My intuition is that as contexts get longer we start hitting the limits of how much comprehension can be embedded in a single point of vector space, and will need better architectures for selecting the relevant portions of the context. We are dealing with multi-headed attention, therefore we have multiple points per token. You can always increase the number of heads or the size of the key vector.
- causal 2y agoThe token embedding is what ultimately gets nudged around by the heads though, right? The key vector just relates to the context size, not the token embedding size, afaik.
- cynicalsecurity 2y agoFeed 1 mln tokens @ Get blocked by some silly overly sensitive "safety" trigger
- gpm 2y agoLast I checked you could disable the safety triggers as an API user with gemini (which doesn't alleviate your obligation to follow the TOS as to the uses of the model).
- VS1999 2y agoI'm not working with a company that can just write in the ToS "we can do anything we want. lol. lmao" and expect me to follow it religiously. Corporations need less control over speech, not more.
- zxexz 2y agoI mean, you are using a service they're providing - many would say they they're exercising their rights by gatekeeping how it's used. There are pretty good models out there you could use however you want for your own purpose, whatever it is. I occasionally fine-tune Mixtral on HN posts+comments and chat with comments. An emergent Dang actually once told me off for flame-baiting a free speech comment.
- dragonwriter 2y ago> I’m not working with a company that can just write in the ToS “we can do anything we want. lol. lmao” and expect me to follow it religiously. You must not work with many companies in the US, especially for online services, at all then.
- xianshou 2y agoLooking at MMLU and other benchmarks, this essentially means sub-second first-token latency with Llama 3 70B quality (but not GPT-4 / Opus), native multimodality, and 1M context. Not bad compared to rolling your own, but among frontier models the main competitive differentiator was native multimodality. With the release of GPT-4o I'm not clear on why an organization not bound to GCP would pick Gemini. 128k context (4o) is fine unless you're processing whole books/movies at once. Is anyone doing this at scale in a way that can't be filtered down from 1M to 100k?
- Workaccount2 2y agoWith 1M tokens you can dump 2000 pages of documents into the context windows before starting a chat. Gemini's strength isn't in being able to answer logic puzzles, it's strength is in its context length. Studying for an exam? Just put the entire textbook in the chat. Need to use a dead language for an old test system with no information on the internet? Drop the 1300 page reference manual in and ask away.
- tulip4attoo 2y ago[flagged]
- dang 2y agoPlease make your substantive points without crossing into personal attack. Your comment would be fine without the first sentence. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- ianbicking 2y agoHow much do those input tokens cost? According to https://ai.google.dev/pricing https://ai.google.dev/pricing it's $0.70/million input tokens (for a long context). That will be per-exchange, so every little back and forth will cost around that much (if you're using a substantial portion of the context window). And while I haven't tested Gemini, most LLMs get increasingly wonky as the context goes up, more likely to fixate, more likely to forget instructions. That big context window could definitely be great for certain tasks (especially information extraction), but it doesn't feel like a generally useful feature.
- refulgentis 2y agoIt's absolutely unconscionable that Gemini Ultra got memory-holed. I can't trust anything that Google says about benchmarks. It seemingly existed only so in December 2023, Gemini ~= GPT-4. (April 2023 version) (on paper) ("32-shot CoT" vs. 5-shot GPT-4)
- summerlight 2y agoGemini Ultra is 1.0 with 8k window. This is 1.5 with 1m window. Your feeling is based on incorrect assumption.
- anoncareer0212 2y agoAnd? You're replying to a comment that points out Gemini Ultra was never released, wasn't mentioned today, and it's the only model Google's benchmarking at GPT-4 level. They didn't say anything about feelings or context window.
- dontreact 2y agoGemini Ultra has been available for people to try via Gemini Advanced (formerly Bard) for a few months
- cma 2y agoIt says it may fall back to a worse model under load and there is no way to tell which you are getting. I think chatgpt has at times done something similar though.
- summerlight 2y ago> You're replying to a comment that points out Gemini Ultra was never released What are you even talking about? How do you know it's memory-holed if you haven't used it? The API is not GA, but the model can be used through the chatbot subscription. GP is talking about their lack of trust on Google's claim of 1M context token, not GPT-4 level reasoning. If you're expect GPT-4 level performance with cost-efficient models, that's another problem.
- numbers 2y agoIt's ironic that when you ask these AI chatbots what their own context size is, they don't know. ChatGPT doesn't even know about 4o existing in 4o.
- SoftTalker 2y agoDoes a monkey know that it is a monkey?
- verdverm 2y agoI think "yes" is the most likely answer here animals have a lot more intelligence than they typically get attributed Tool use, names, language, social structure and behavior, even drug use has been shown across many species
- chaorace 2y agoOkay, but the monkey doesn't know that it knows that it's a monkey.
- verdverm 2y agoare you sure? Many animals recognize themselves and their species as separate concepts
- keefle 2y agoHe meant something more meta I believe. Knowing you are a monkey is one thing, and knowing that you know you are a monkey is a another thing. It's about being cognisant of the fact that there is something called knowledge and you have it
- chaorace 2y agoPrecisely. To put it more concretely: it is no small feat to grasp the abstract distinction between known-knowns, known-unknowns, unknown-knowns, and unknown-unknowns. They do not know what they do not know.
- nightski 2y agoA lightweight model that you can only use in the cloud? That is amusing. These tech megacorps are really intent on owning your usage of AI. But we must not let that be the future.
- nojvek 2y agoWill wait for Meta to release Flash equivalent weights. Multi-Modal modals running offline on mobile devices with millisecond latencies per token seems the future. Where is Apple in all of this. Why is Siri still so shit?
- visarga 2y agoApple made a deal with OpenAI for GPT4o, the stakes are indeed high, can't be caught with pants down. iPhone needs to remain the premium brand.
- deleted 2y ago[deleted]
- kherud 2y agoNow that context length seems abundant for most tasks, I'm wondering why sub-word tokens are still used. I'm really curious how character-based LLMs would compare. With 2 M context, the compute bottleneck fades away. I'm not sure though what role the vocabulary size has. Maybe a large size is critical, since the embedding already contains a big chunk of the knowledge. On the other hand, using a character-based vocabulary would solve multiple problems, I think, like glitch tokens and possibly things like arithmetic and rhyming capabilities. Implementing sub-word tokenizers correctly and training them seems also quite complex. On a character level this should be trivial.
- AaronFriel 2y agoThe attention mechanism is vastly more efficient to train when it can attend to larger, more meaningful tokens. For inference servers, a significant amount of memory goes into the KV cache, and as you note, to build up the embedding through attention would then require correlating far more tokens, each of which is "less meaningful". I think we may get to this point eventually, in the limit we will want multimodal LLMs that understand images and sounds down to the pixel and frequency, and it seems like for text, too, we will eventually want that as well.
- yk 2y ago> a significant amount of memory goes into the KV cache Is there a good paper (or talk) how inference looks at scale? (Kinda like ELI-using-single-gpus)
- AaronFriel 2y agoThe PagedAttention paper is a good starting point as it represents the first major open source inference engine that had "pretty good" batch performance for transformers. https://arxiv.org/pdf/2309.06180 https://arxiv.org/pdf/2309.06180
- thomasahle 2y agoMaybe you could just use a good-old 1D-CNN for the bottom 3-4 layers. Then the model has been able to combine characters into roughly token length chunks anyway. Just make sure to have some big MLPs at the start too, to enrich the "tokens" with the information currently stored in the embedding tables.
- alephxyz 2y agoNot very informative. They're selling it as the fast/cheap option but they don't benchmark inference speed or compare it with non-gemini models. According to https://ai.google.dev/pricing https://ai.google.dev/pricing it's priced a bit lower than gpt3.5-turbo but no idea how it compares to it.
- simonw 2y agoI upgraded my llm-gemini plugin to provide CLI access to Gemini Flash: pipx install llm # or brew install llm llm install llm-gemini --upgrade llm keys set gemini # paste API key here llm -m gemini-1.5-flash-latest 'a short poem about otters' https://github.com/simonw/llm-gemini/releases/tag/0.1a4 https://github.com/simonw/llm-gemini/releases/tag/0.1a4
- quantisan 2y agoPrice (input) $0.35 / 1 million tokens (for prompts up to 128K tokens) $0.70 / 1 million tokens (for prompts longer than 128K) Price (output) $0.53 / 1 million tokens (for prompts up to 128K tokens) $1.05 / 1 million tokens (for prompts longer than 128K) --- Compared to GPT-3.5 Turbo Input US$0.50 / 1M tokens Output US$1.50 / 1M tokens
- deleted 2y ago[deleted]
- cs702 2y agoWe're witnessing a race to the bottom on pricing as it's happening. Competition based solely or mainly on pricing is a defining characteristic of a commodity market, i.e., a market in which competing products are interchangeable, and buyers are happy to switch to the cheapest option for a given level of quality. There's an old saying that if you're selling a commodity, "you can only be as smart as your dumbest competitor." If we want to be more polite, we could say instead: "you can only price your service as high as your lowest-cost competitor." It seems that a lot of capital that has been "invested" to train AI models is, ahem, unlikely ever to be recovered.
- __loam 2y agoYou're saying the quiet part out loud here.
- Aloisius 2y agoPrice competition isn't limited to commodities.
- cs702 2y agoI never said it was.
- EGreg 2y agoYou never said it wasn't, either :-P
- Aloisius 2y agoThen why imply that it is a commodity because they (partly) compete on price? Fungibility is the defining characteristic of commodities. While these products can be used to accomplish the same task, we're not near real fungibility yet.
- cs702 2y agoProducts that are fungible compete on price (what else?). Chat-with-AI services that have similar performance are pretty fungible today. Switching from one to the other is... remarkably easy. The moment Gemini Flash's competitors start losing customers they will lower their prices to remain competitive.
- KBme 2y ago[flagged]
- objektif 2y agoDoes Goog have anything like openai assistant via API? If they had I would definitely give it a try.
- eru 2y agoThe website talks about a specific benchmark: > Python code generation. Held out dataset HumanEval-like, not leaked on the web What I find interesting here is that for this particular benchmark _not_ publishing the benchmark is advertised as a feature (instead of as a sign of 'trust me, bro, we have a great benchmark'), and I can understand why. Still these are strange times we live in.
- deleted 2y ago[deleted]
- deleted 2y ago[deleted]
- zone411 2y ago15.3 On NYT Connections benchmark: GPT-4 turbo (gpt-4-0125-preview) 31.0 GPT-4o 30.7 GPT-4 turbo (gpt-4-turbo-2024-04-09) 29.7 GPT-4 turbo (gpt-4-1106-preview) 28.8 Claude 3 Opus 27.3 GPT-4 (0613) 26.1 Llama 3 Instruct 70B 24.0 Gemini Pro 1.5 19.9 Mistral Large 17.7 -----> Gemini 1.5 Flash 15.3 Mistral Medium 15.0 Gemini Pro 1.0 14.2 Llama 3 Instruct 8B 12.3 Mixtral-8x22B Instruct 12.2
- ukuina 2y agoSo many high-performing, yet poorly-named OpenAI models in that list.
- webprofusion 2y agoUh guys, yeah.. Adobe are on the phone saying something about trademark infringement, apparently Flash is something else? I don't know, I've never heard of it..
- exodust 2y agoInterestingly, until your comment I hadn't made any connection with old Flash, even though I spent hundreds of hours making Flash games. This suggests names don't stick around for long and can be re-used. Perhaps Google could bring back "Buzz" and "Wave" since enough time has passed!
- stan_kirdey 2y agoI've been diligently trying to use Gemini 1.5 Pro, and it is not even on the level of Llama3-70B. I really hope Gemini improves, even if it gets reduced context length.
- ukuina 2y agoFAIR really swung for the fences with Llama3. It's a very impressive model, but the 8K context size is quite limiting for most use-cases.
- mrcwinn 2y agoI will say Google certainly has the better branding team. I like Gemini, Gems, and so on. “ChatGPT” is quite a clunky mess. OpenAI just feels like a faceless entity. All things that could change but seems late in the game at this point. They certainly had the money to be more creative as they came to market.
- ukuina 2y agoOpenAI desperately needs a marketing consult. "GPT4o"? Seriously? Even "GPT4 Omni" is easier in conversation, and that's what the "o" stands for! They severely underestimate the number of casual users they have.
- zarzavat 2y agoOpenAI doesn’t need marketing because everybody knows who’s the best. Same reason that if I asked you what’s the best violin you would say Stradivari, even though you’ve never seen an ad for one. OpenAI could call their model the “[poo emoji] 5000” for all the difference it would make.
- cpeterso 2y agoGoggle “Gemini” is a much better product name than any name OpenAI has, but the Gemini product family could use some structure: Gemini Advanced (“with Ultra 1.0”) Gemini Ultra Gemini Pro Gemini Flash Gemini Nano-1 Gemini Nano-2
- precompute 2y ago"ChatGPT" is like "Google". "Gemini" is never replacing that.
- borgdefense 2y ago[flagged]
- michaelteter 2y agoIf Gemini Flash is just faster Gemini, then I would say that bad answers aren't better when delivered more quickly. I ran Gemini Pro side by side with ChatGPT 4 for a few months on practical coding, systems architecture, and occasional general questions. ChatGPT was more useful at least 80% of the time. Gemini was either wrong or laboriously meandering in reaching a useful answer that it wasn't worth using, in my experience. Faster isn't what I needed... Maybe it's also "smarter" (more useful) too now?
- deleted 2y ago[deleted]
- ganzuul 2y agoPresumably we are defining smartness as doing more with less, so this indicates they have something going on in the latent space which will scale.
- ArkimPhiri 2y agoI added Gemini 1.5 Flash to my AI app Semaj AI and its fast indeed, it is also very intelligent compared to Gemini 1.0 Pro.
- CapsAdmin 2y agoI was really fascinated by Gemini 1.5 pro, though it was a bit slow and sort of 70% accurate in my use case. I have a huge 500k~ tokens and complex niche codebase that I've worked on for many years, largely alone. There are parts of it I wish to refactor but I struggle because it I've become blind to my own code. It also sometimes feels lonely in a way. If it was a game I could at least show my friends, but this project is too abstract. Gemini missed the mark a few times, especially when asking about more complex things but overall it was useful. That it got things wrong is sort of ok because I knew the codebase well enough to spot those mistakes. Gemini 1.5 pro gave me a glimpse into what it was like having "someone" understand your whole codebase, hint at areas to improve, etc. A bit like a true copilot or coworker, but for a dream hobby project.
- fnordpiglet 2y agoHow does it compare to others products in your use case?
- CapsAdmin 2y agoI feel it was worse than gpt4 at coding at a low level (getting syntax right, forgetting small details, etc) But I can write code so those can be fixed. At a higher level it's OK, but the most valuable thing is being able to have my codebase in its context. No other public LLM currently as far as I know can do that.
- loulouxiv 2y agoHow do you proceed to use it this way ? Did you have to write custom code to call the API with your codebase ? Can you give some example questions for which the LLM gave you useful insights and/or made you save time ?
- CapsAdmin 2y agoI just used https://aistudio.google.com/ https://aistudio.google.com/, no api or anything. My project "Nattlua" is a typed version of Lua like Typescript is to Javascript. Some example questions that would give me new insight: - Asking what the codebase is without supplying the readme. (though it might know because the codebase is public already) - Asking it to generate complex type code based on existing tests and examples and without. - Asking for places to refactor, the most fun one. Sometimes the exact solution provided is wrong, but often it's a good start.
- dragonwriter 2y agoOne thing OpenAI is beating Google at is actually publishing pricing for its APIs (and being consistent as to what they are called.) Google has I think 10 models available (there’s more than ten model names, but several of the models have multiple aliases) through what the Google Cloud console calls the Generative Language API (the documentation calls it the Gemini API) – based on enumerating the model list through the API itself. Of those, 3 have pricing information on the documentation page for Gemini API pricing, 2 of which are in preview so that pricing applies in the future. Only one (the same one of the 3 on the documentation page that is not in preview) has pricing listed on the console for the Generative Language API. On the Cloud SKUs list, there is no Generative Language API, but there is for the Gemini API, with, again, the same one model. On the Cloud Price list which the console page links for the “latest pricing” (why are there so many different things?) neither the Generative Language API nor the Gemini API is listed at all.
- amery2010 2y ago[dead]