6 ms·
Can generalist foundation models beat special-purpose tuning?
- avereveard 3y agogpt-4 is not foundational, it's rlhf tuned. also I didn't find the actual gpt-4 model revision, so yeah. sloppy all around.
- cl42 3y agoI’m looking to run experiments like this for specific domains and prompting strategies as a way to showcase the next version of https://evals.phasellm.com https://evals.phasellm.com If the paper above resonates with you or if you have questions about promoting techniques, feel free to reply below or email me at hello #at# phaseai #dot# com Happy to help!
- jerpint 3y agoA real test would be fine tuning gpt4 and comparing that to gpt4 medprompt
- c7b 3y agoI think the point of the study was precisely to see how well the original model without adaptations (my understanding is that they only use prompting, which does not affect weights) can perform. I think that question is arguably even more interesting than 'How well can we train a model to perform on med school questions?'. I'm not saying this is generalization, because the training set surely included a lot of medical literature, but if the base model without fine-tuning can perform well in one important domain, that's a very interesting data point (especially if we find the same to hold true for multiple domains).
- ramoz 3y agoCaveat this with the fact that they did rag.
- practice9 3y agoYeah the Medprompt name is misleading
- owl_brawl 3y agoThe "RAG" part (over the training set) is by far the smallest contribution to the performance gains reported (see the ablation study in section 5.2). I don't think the model is actually learning in-context from the selected samples, but rather is continuing to be better conditioned to sample from the right part of the pre-training distribution here, which does a slightly better job when the samples are on topic (vs. random)
- sungam 3y agoWe found that GPT-4 performs at a very high level on dermatology specialist exam questions (which it would not have been trained on) https://www.medrxiv.org/content/10.1101/2023.07.13.23292418v1 https://www.medrxiv.org/content/10.1101/2023.07.13.23292418v...
- RandomLensman 3y agoThe questions were public available. The point is the model wasn't specifically trained for dermatology.
- 4death4 3y agoWhy wouldn’t it have been trained on such questions?
- sungam 3y agoNot publicly available
- RandomLensman 3y agoAbstract says the sample questions were public available. The point they are making is that the model wasn't specifically trained for dermatology.
- pyinstallwoes 3y agoWhat makes you think that the entire body of private access journals and books hasn’t been used?
- civilized 3y agoExtensive medical information is publicly available on websites like www.nhs.uk. So even if it wasn't literally trained on exam questions, it may have picked up substantially identical information elsewhere. Which is still a significant feat. But I doubt it's doing anything more than the sort of semantic lookup that is already well-known to be within its capabilities.
- evrydayhustling 3y agoThis is a very interesting study, but the generalization implied in the abstract -- that specialized models may not have advantages in specialized domains -- is an overreach for two reasons. They introduce medprompt, which combines chain of thought reasoning, supervision, and retrieval augmented generation to get better results on novel questions. This is a cool way to leverage supervision capacity outside of training/fine tuning! But they compare this strategy, applied to GPT-4, with old prompting strategies applied to MedPalm. This is apples and oranges -- you could easily have taken the same supervision strategy (possibly adapted for smaller attention window etc) to get a closer comparison. Second, MedPalm is a fine tuned 500b parameter model. GPT-4 is estimated to be 3-4x size. So the comparison confounds emerging capabilities at scale, in addition to prompting impact, with anything you can say generally about the value of model specialization.
- djha-skin 3y agoThe answer to most of these kinds of questions of generalist versus specialist when it comes to AI, according to Richard Sutton's "The Bitter Lesson"[1], is that specialized AI will win for a while, but generalized will always win the long-term because it's better at turning compute into results. Whether or not that will pan out with "fine tuned" versus "generalized" versions of the same data-eating algorithms remains to be seen, but I suspect the bitter lesson might still apply. 1: http://www.incompleteideas.net/IncIdeas/BitterLesson.html http://www.incompleteideas.net/IncIdeas/BitterLesson.html
- nuz 3y agoWell the counter argument is that with an equal amount of compute, specialized models will win over generalized ones. The reason is that generalized ones have to spread out their weights to be able to do a ton of irrelevant things to the niche subject that a finetuned model can accomplish with fewer and more focused weights.
- rschiavone 3y agoI'd argue that yours is not a counter argument, but the same thing said by grandparent rephrased. It's obvious that with the same amount of computing power, specialized > generalized. But in 2-5 years, the future generalized models can beat the current specialized ones.
- goldenkey 3y agoThe future of this is the same as what happened with PCs. The specificity will initially be thrown away, added back later as an accelerator, and eventually, brought back into the fold as an automatically used accelerator. It all comes full circle as optimizations, hinting, tiering, and heuristics. AI will be used to select the net to automatically load. Nets will be cached, branch predicted etc. The future of AI software and hardware doesn't yet support the scale we need for this type of generalized AI processor (think CPU but call it an AIPU.) And no, GPUs aren't an AIPU, we can't even fit whole some of the largest models on these things without running them in pieces. They don't have a higher level language yet, like C, which would compile down to more specific actions after optimizations are borne (not PTX/LLVM/Cuda/OpenCL.)
- RandomLensman 3y agoAny sufficiently large library has a lot of specialty books in it.
- mistrial9 3y agoyou have got to be joking. This is not true in any real way. source: six years in the book industry in California
- upghost 3y agoTechnical foul. Doing science based on closed-source model. 10 yard penalty for humanity. GPT-4 almost disappeared last week. Aside from the obvious AI bus factor and "continuity of science" concerns, I find it absurd that so many papers and open source ML/LLM frameworks and libraries exist that simply would not work at all if not for GPT-4. Have we simply given up? I thought this was hacker news.
- Philpax 3y agoI'm all in favour of doing reproducible work, especially against open-source ML (I'm the maintainer of a library for LLM inference), but what is the alternative here? GPT-4 is still the flagship model "of humanity"; it is still the best model that is publicly available. The intent of the paper is to determine whether the best general model can compete with specialized models - how do you do that without using the best general model?
- tensor 3y agoThis is not answering that question. It is answering whether a specific large model with unknown training set can compete with specific small models on known smaller training sets.
- rafram 3y agoHalf of science comes down to poking at black boxes to see how they react. The universe is a closed-source model!
- ZiiS 3y agoThe other half is publishing results to fix that.
- quickthrower2 3y agoThe black hole here is political though! The algorithm in question is here on earth.
- deleted 3y ago[deleted]
- PaulHoule 3y agoI would think you could apply an inference process like that to a smaller model and get results better than you get with zero-shot on a larger model... And be able to afford to run the smaller model as many times as it takes.
- make3 3y agoreally ? "gpt4 beats our tiny model", that's your paper?
- hallqv 3y agoDepends on your definition of winning - special-purpose tuning is vastly more cost effective since it allows you to train a smaller model that can perform specific tasks as good as a bigger one. A good analogy is building a webapp - would you prefer to hire a developer with 30+ years experience in various CS domains as well as a PhD or a specialized webdev with 5 years experience at a tenth of the rate?
- Xcelerate 3y agoNot strictly related to their study, but if we consider Solomonoff induction to be the most general model of all (albeit not computable), then I’d say yes to the question, if only because it will select the most specialized model(s) for the particular problem at hand automatically. One could argue universal general intelligence is simply the ability to optimally specialize as necessary. I think one aspect that is overlooked when people are involved is that we create specialized or fine-tuned models precisely for the reason that a general approach didn’t work well enough. If it had, we would have stopped there. So there’s a selection bias in that almost all fine-tuned models are initially better than general models, at least until the general models catch up.
- doingtheiroming 3y agoThere are lots of ways to define “good enough” as well. What are the costs of inferring several small experts contributing to a decomposed workflow versus using GPT4 for example. If you want to run multiple instances for different teams or departments, how do the costs escalate. Do you want to include patient data and have concerns about using a closed source model to do so. Etc. There’s little doubt that GPT4 is going to be most capable at most tasks either OotB or with prompt engineering as here. But that doesn’t mean it’s the right approach to use now.
- cyanydeez 3y agoI assume the next step is to find the right model size for semantic specialism then create a model of model that's filtering down to specialized models. same will happen for temporal information that stratifies and changes, ie, asking what a 1950s scientist understands about aquantim physics.
- VirusNewbie 3y agoI don't understand the reasoning. if you merged two of the most specialized models together and simply doubled the size/parameter count, there is no reason the smaller one can generalize better than the larger one. It would simply have two different submodels inside its ubermodel.
- Davidzheng 3y ago
- kjkjadksj 3y agoWhat I want to know is how these systems compare to simple keyword search algorithms. E.g. presumably the person using these systems are medical professionals, typing in some symptoms perhaps gleaming for answers. Theoretically, all these date could be cataloged in a way where you can type in symptoms x y z, demographic information a b c, medical history d e f, and get a confidence score for a potential diagnosis bases on matching these factors to results from past association studies. A system like that I imagine would be easier to feed in true information and importantly would also offer a confidence estimate of the result. It would also be a lot more computationally simpler to run I’d imagine given the compute required to train a machine learning model vs keyword search algorithms you can often run locally across massive datasets.
- shmageggy 3y agoWhy did "Case study in Medicine" get stripped from the submitted title? It's way more clickbaity now.
- hatmatrix 3y agoI think you answered your own question.
- menzoic 3y agoYou can always fine tune a generalist model to get better results
- tedivm 3y agoThis was something we thought about a bit at Rad AI. I think one of the problems this paper ignores isn't whether you can get a general purpose model to beat a special purpose one, but whether it's really worth it (outside of the academic sense). Accuracy is obviously important, and anything that sacrifices accuracy in a medical areas is potentially dangerous. So everything I'm about to say assumes that accuracy remains the same or improves for a special purpose model (and in general that is the case- papers such as this talk about shrinking models as a way of improvement without sacrificing accuracy: https://arxiv.org/abs/1803.03635 https://arxiv.org/abs/1803.03635). All things being equal for accuracy, the model that performs either the fastest or the cheapest is going to win. For both of these cases one of the easiest ways to accomplish the goal is to use a smaller model. Specialized models are just about always smaller. Lower latency on requests and less energy usage per query are both big wins that affect the economics of the system. There are other benefits as well. It's much easier to experiment and compare specialized models, and there's less area for errors to leak in. So even if it's possible to get a model like GPT-4 to work as well as a specialized model, if you're actually putting something into production and you have the data it almost always makes sense to consider a specialized model.
- theossuary 3y agoThat's interesting, I'm currently working on an idea that assumes the opposite. Having built specialized models for years, the cost of having a data science team clean the data and build a model is pretty high, and it can take quite a while (especially if part of the project is setting up the data collection). For prototyping and for smaller use cases, it makes a lot of sense to use a much more general model. Obviously this doesn't apply to things like medicine, etc. But for much more general things like: check if someone is on the train-tracks, or number of people currently queuing in a certain area, or if there's a fight in a stadium; I think multi-modal models are going to take over. Not because they're efficient, or particularly fast; but because it'll be quick to implement, test, and iterate on. The cost of building a specialized model, and keeping it up to date, will far exceed the cost of an LVM in most niche use cases.
- tedivm 3y ago
- TrevorJ 3y agoIf I had to guess at what the next leap for AI would be, it looks something like a collection of small and large special-purpose models, with a model on top that excels at understanding which sub-models (if any) should be applied to the current problem space.
- rdlecler1 3y agoThere is a reason we use forks and not sporks. Specialization can give you exactly what you want.