3 ms·
Love the ui in the playground, it reminds me of Qwen chat. We have reached a point where the bottlenecks in genAI is not the knowledge or accuracy, it is the c
by Alifatisk 1y ago
Love the ui in the playground, it reminds me of Qwen chat.
We have reached a point where the bottlenecks in genAI is not the knowledge or accuracy, it is the context window and speed.
Luckily, Google (and Meta?) has pushed the limits of the context window to about 1 million tokens which is incredible. But I feel like todays options are still stuck about ~128k token window per chat, and after that it starts to forget.
Another issue is the time time it takes for inference AND reasoning. dLLMs is an interesting approach at this. I know we have Groqs hardware aswell.
I do wonder, can this be combined with Groqs hardware? Would the response be instant then?
How many tokens can each chat handle in the playground? I couldn't find so much info about it.
Which model is it using for inference?
Also, is the training the same on dLLMs as on the standardised autoregressive LLMs? Or is the weights and models completely different?
- martinald 1y agoI agree entirely with you. While Claude Code is amazing, it is also slow as hell and the context issue keeps coming up (usually at what feels like the worst possible time for me). It honestly feels like dialup most LLMs (apart from this!). AFIAK with traditional models context size is very memory intensive (though I know there are a lot of things that are trying to 'optimize' this). I believe memory usage grows at the square of context length, so even 10xing context length requires 100x the memory. (Image) diffusion does not grow like that, it is much more linear. But I have no idea (yet!) about text diffusion models if someone wants to chip in :).
- kadushka 1y agoWe have reached a point where the bottlenecks in genAI is not the knowledge or accuracy, it is the context window and speed. You’re joking, right? I’m using o3 and it couldn’t do half of the coding tasks I tried.
- Alifatisk 1y agoI've been in similar situations, I've realized, you make these llms accomplish lots of difficult tasks if you prompt it correctly, and that is a form of art! If a colleague of mine, who's incredible at prompting, did impressive things with gpt-3, so I am sure o3 can do even more wilder stuff.
- kadushka 1y agoWhat did he do with gpt-3?
- Alifatisk 1y agoIt was mostly coding related tasks we had
- kadushka 1y agogpt-3 could not do any coding tasks.
- Alifatisk 1y agoWhat makes you say so?
- kadushka 1y agoBecause gpt-3 was not trained to do coding tasks. It could do a simple autocomplete. Perhaps you are confusing it with gpt-3.5?
- Alifatisk 1y agoI vividly remember it being in the same period as OpenAis Codex
- kadushka 1y agoCodex paper confirms that GPT-3 could not do any coding tasks. It's right there in the abstract: https://arxiv.org/abs/2107.03374 https://arxiv.org/abs/2107.03374
- Alifatisk 1y agoMight be GPT-3.5 then, but I am certain this was before the GPT-4 era. But that's besides the point, prompting it correctly has a huge effect on the outcome and its ability to suffice your need. So saying o3 being very unusable is hard to believe in my experience
- totetsu 1y agoIts a fork/implementation of openwebui isn't it?
- Alifatisk 1y agoI don't now, I couldn't find any reference to any project in either QwenChat or Inceptionlabs playground. But looking at openwebui, it could be it? But with a slight redesign on the chat section.
- runeblaze 1y agoI mean we don't really talk about the accuracy of generative models. It is more of a discriminative model thing. But besides this, the current gen of models still, like, hallucinates more than many would like