18 ms·
This is the first time I've felt like Anthropic may be a true competitor to OpenAI. I see 6 ways to improve foundation LLMs other than cost. If your product is
by tikkun 3y ago
This is the first time I've felt like Anthropic may be a true competitor to OpenAI.
I see 6 ways to improve foundation LLMs other than cost. If your product is best at one of the below, and has parity at the other 5 items, then customers will switch. I'm currently using GPT-4-8k. I regularly run into the context limit. If Claude-100K is close enough on "intelligence" then I will switch.
Six Dimensions to Compare Foundation LLMs:
1. Smarter models
2. Larger context windows
3. More input and output modes
4. Lower time to first response token and to full response
5. Easier prompting
6. Integrations
- zomglings 3y agoAlso if you allow users to receive vector representations of context and provide such representations as side information when querying LLMs.
- danenania 3y agoOne question is how much other factors really matter compared to the raw "intelligence" of the model--how good its completions are. You're not going to care very much about context window, prompting, or integrations if the output isn't good. It would be sort of like a car that has the best steering and brakes on the market, but can't go above 5 mph.
- tikkun 3y agoStrongly agree. They are ordered by how much I think they generally will lead to users choosing one model over the other. Intelligence is the most important dimension by far, perhaps an order of magnitude or more above the second item on the list.
- danenania 3y agoOn that note, can anyone speak to how Anthropic (or other models) are doing on catching up to OpenAI for pure model intelligence/quality of completions? Are any others approaching GPT-4? I've only used GPT-based tools so I have no idea.
- famouswaffles 3y agoThe best claude model is closer to GPT-4 than 3.5
- majormajor 3y agoBig question on that for me is that there's a variety of "completion styles" and I'm curious how "universal" performance on them is. Probably more than this, but a quick list that comes to mind: * Text summary/compression * Creative writing (fiction/lyrics/stylization) * Text comparison * Question-answering * Logical reasoning/sequencing ("given these tools and this scenario, how would you perform this task") IMO, for stuff like text comparison and question-answering, some combo of speed/cost/context-size could make up for a lot, even if they do "worse" versions of stuff just that's too slow or expensive or context-limited in a different model.
- gmerc 3y agoWe already see how that goes. Stackable LoRa like in stable diffusion.
- solarkraft 3y agoI don't know. While using Phind I regularly get annoyed by long prose that doesnt answer anything (yes, "concise" is always on). Claude seems to be directly geared towards solving stuff over nice writing.
- Tostino 3y agoI generally add to my initial prompts to GPT4 to: From now on, please use the fewest tokens possible in all replies to save tokens and provide brief and accurate answers.
- modernpink 3y agoOr rather, more analogously, a self-driving car that has a range of 10 000 miles but sometimes makes mistakes when driving vs a self-driving car with a range of 800 miles that never makes mistakes. Once you've have a taste of intelligence it's hard to give up. However, in many applications there is a limit on how intelligent you need the LLM to be. I have found I am able to fall back to the cheaper and faster GPT-3.5 to do the grunt work of forming text blobs into structured json within a chain involving GPT-4 for higher-level functions.
- rizky05 3y ago[dead]
- RobotToaster 3y ago>Six Dimensions to Compare Foundation LLMs I'd add open source to the list, which neither "open"AI or this is.
- throwawayadvsec 3y agonow that I think about it is it that important to open source models that can only run on hardware worth tens of thousand of dollars? who does that benefit besides their competitors and nefarious actors? I've been trying to run one of the largest models for a while, unless 30,000$ falls in my hand I'll probably never be able to run the current SOTA
- chrisco255 3y ago> is it that important to open source models that can only run on hardware worth tens of thousand of dollars? Yes, because as we've seen with other open source AI models, it's often possible for people to fork code and modify it in such a way that it runs on consumer grade hardware.
- YetAnotherNick 3y agoI agree utility of open source for personal usecase is overblown. But for commercial usecases, open source is very relevant for privacy reasons as many enterprises have strict policy not to share data with third party. Also it could be a lot cheaper for bulk inference or to have a small model for particular task.
- turtles3 3y agoHowever, the same thing could be achieved with closed source models. There's nothing to stop an LLM being made available to run on prem under a restrictive license. It would really be no different to ye olde desktop software - keeping ownership over bits shipped to a customer is solved with the law rather than technical means. That said, I really hope open source models can succeed, it would be far better for the industry if we had a Linux of LLMs.
- nr2x 3y agoFor me I’d say speed trumps all else. It’s impossible to truly reach scale with the glacial response times you get from current API.
- sebzim4500 3y ago>speed trumps all else Then use GPT-2
- nr2x 3y agoI actually do prefer 3.5-turbo over 4 for many tasks.
- nico 3y agoFaster, cheaper fine-tuning and training If I could train a useful model, on my own data, in a reasonable time I would want to have a CI-training pipeline to always have my models up to date
- makestuff 3y agoYeah I remember in undergrad I was working on using transformation learning to train an object detector. Basically you only needed 100ish images to get the model to detect that new object really well. I'm not sure what the analogous term is for a similar process on LLMs, but that will be huge when there is a service for it.
- visarga 3y agoLLMs can do that without any examples (zero shot) or with one or a few demonstrations in the prompt, if you can describe the task in the limited context window. If you want for example to train the model to learn to use a very large API, or access the knowledge in a whole book, it might need fine-tuning.
- nico 3y agoCould I just train a very small LLM with an English dictionary + Python + large API documentation + large Python code base? Then do some chat fine tuning (like what HF did with StarCoder to get ChatCoder) And get a lightweight LLM that knows the docs and code for the thing I need it for After that, maybe incrementally fine tune the model as part of your CI/CD process
- toss1 3y agoHow similar were the object to other objects? E.g., were you trying to distinguish an object vs nothing, a bicycle vs a fish, a bird vs a squirrel, or two different species of songbird at a feeder? How much would the training requirements increase or decrease moving up or down that scale?
- ilaksh 3y agoThe PaLM 2 stuff released yesterday has fine tuning for their newest large models as a core feature.
- moffkalast 3y agoUntil they actually make any of it available in anything but an obscure expensive API you have to request access to, they might as well not even exist.
- r_thambapillai 3y agothere are many services that integrate with them that would allow you to self-serve signup
- dkarras 3y agoI've been using it through poe and I prefer it to ChatGPT but can't pinpoint why. It just "gets" me better I guess?
- williamstein 3y agoThe landing page says "Easy integration via standard APIs Claude can be incorporated into any product or toolchain you’re building with minimal effort." Then there is a big button "Request Access", which for me right now just does nothing. OpenAI has really faced the pain to make their product available via an API to the general public at scale, but Anthropic/Google/etc. don't quite seem to be there yet. It's frustrating.
- akiselev 3y agoTry a browser or a clean profile without any ad blocking turned on. It took me a couple of tries to figure out how to get it working but you should see a modal with a form when it works. FYI the waitlist form submits a regular POST request so it'll reload the main page instead of closing the modal dialog. I opened network monitor with preserved logs to double check that I made it on the list :facepalm:
- chaxor 3y agoI don't think the person you're responding to wants a network based or cloud based solution. When someone says they want it available they mean running on their own device. This is hackernews, nearly everyone on this site should have their own self hosted LLM running on a computer/server or device they have at their house. Relying on 'the cloud' for everything makes us worse developers in just about every imaginable way, creates a ton of completely unnecessary and complicated source code, and creates far too many calls to the internet which are unnecessary. Using local hard drives for example is thousands of times faster than using cloud data storage, and we should take advantage of that in the software we write. So instead of making billions of calls to download a terabyte database query-by-query (seen this 'industry-standard' far too many times), maybe make one call and build it locally. This is effectively the same problem in LLMs/ML in general, and the same incredible stupidity is being followed. Download the model once, run your queries locally. That's the solution we should be using.
- winstonprivacy 3y agoDon't forget the ability to fine tune based on one's own data sources. For me, this is more important than any of the six reasons you mentioned.
- deleted 3y ago[deleted]
- srowaway2 3y ago7. Price! GPT4-32K costs ~$2 if you end up using the full 32K tokens, so if you're doing any chaining or back-and-forth it can get expensive very quickly.
- hesdeadjim 3y agoOof, got access to the 8k model recently and was wondering what costs would be on the 32k one. That's brutal.
- freediver 3y agoWhy? 32K tokens would cover creating source code for a small to medium programming project. THat would easilly cost hundreds of dollars if done by a freelancer, and could get it in almost real time for just $2.
- replygirl 3y ago32k tokens is 500-1000 lines of code, so more like thousands of dollars, unless you're comparing against a landing page or CRUD tool arranged through Upwork. On the flip side, before I dove into AutoGPT I couldn't get GPT-4 to iterate on something as simple as a TypeScript function that deeply converts { foo_bar: string } to { fooBar: Date } without running out of context and cyclically reintroducing regressions.
- hesdeadjim 3y agoAdvice on where to get started with auto-gpt?
- jiggawatts 3y agoOnly if you can get that result in one pass. The real utility of LLMs is that they can be called in a loop to scan through many web pages, many code files, issue tickets, emails, etc... There are already demos and experiments out there that for every input, 4x outputs are generated, then those are fed back into the LLM 4x for "review", then the best variant is then used to generate code which then automatically tested, errors are fed back in a loop, also with 4x parallel tries, etc... It's the throughput compared to humans that is the true differentiator. If hooking up the API in a loop up ends up costing more than a human, then it's not worth it.
- IshKebab 3y agoReliability surely? They still haven't managed to make a model that says "I don't know" rather than bullshitting. That's by far the biggest unsolved problem.
- jll29 3y agoMore languages?
- sanxiyn 3y agoAgreed. This is an important feature. Not all people speak English.
- ianhawes 3y agoWe use Anthropic Instant in production and it has been much faster than Davinci/GPT4 for awhile. In terms of quality, Instant is at least as good as GPT3.5.
- azeirah 3y agoPrivacy as well! I'd much rather run a model locally than in the cloud even if I lose out in performance in both speed and quality.