5 ms·
To add a voice of skepticism. The recent rush to open source these models may be indicative that the tens of millions that’s spent training these things has re
by lumost 4y ago
To add a voice of skepticism. The recent rush to open source these models may be indicative that the tens of millions that’s spent training these things has relatively poor roi. There may be a hope that someone else figures out how to make these commercially useful.
- mumblemumble 4y agoFrom what I've seen, using these huge models for inference at any kind of scale is expensive enough that it's difficult to find a business case that justifies the compute cost.
- f311a 4y agoYandex uses it for search and voice assistant
- sailingparrot 4y agoThose models aren't trained with the objective of being deployed in production. They are trained to be used as teachers during distillation into smaller models that fit the cost/latency requirements for whatever scenario those big companies have. That's where the real value is.
- gfodor 4y agoAn equally plausible frame is that once a technology becomes replicated across several companies, it makes sense to open source it since the marginal competitive advantage are the possible resultant external network effects. I don't know if that's the right way to think about the open sourcing of large language models. I just think we really can't read too much into such releases regarding their motivation.
- jenny91 4y agoYes, commoditize your complements.
- MivLives 4y agoWe're using these at where I work (large retail site) to help make filler text on generated articles. Think the summary blurb no one reads at the top. As for why we're writing these articles (we have a paid team that writes them too), the answer is SEO. This is probably the only thing I've seen done with a text model in production usage. I'm not 100% sure what model they're using.
- BonoboIO 4y agoContent made for machines. Probably a billion dollar industry.
- jquery 4y agoMade by machines, for machines. It’s poetic.
- blippage 4y agoThere's a line in one of Douglas Adams' books where he says something along the lines that things like VCRs were invented to watch TV programs so that you don't have to. Who would have thought that one man's joke would become a reality?
- fab1an 4y agoContent made for machines serving humans made by machines pretending to be human
- tobr 4y agoSorry but every part of that sounds so terrible.
- MivLives 4y agoYeah I'm not a huge fan of it. I'll never forget the look in our UX person's eyes when she realized that our team doesn't exist to make customer's experience better (there's a ton of other teams for that) but to make Googlebot's experience better. Right now we're in the process of getting publishers you've heard of to write blurbs for best lists, but we're supplying the products so it's not really a best list. I can't say I'm a big fan but my teams is great and I don't have time to look for a job right now.
- lostmsu 4y agoThey did not publish benchmarks about quality of the models, which is very suspicious. I personally squinted hard when they said removing dropout improves training speed (which is in iterations per second), but said nothing about how it affects the performance (rate of mistakes in inference) of the trained model.
- jasonphang 4y agoI agree that the lack of benchmarks makes it hard to determine how valuable this model is. But on the topic of dropout, dropout has been dropped for the pretraining stage of several other large models. Off the top of my head: GPT-J-6B, GPT-NeoX-20B, and T5-1.1/LM.
- HeavyStorm 4y agoMaybe training it is not that expensive? I know from practice that it takes a really really long time to train even a small nn (thousands of params) , so you'll need a lot more hardware to train one with billions... But, it's expensive to buy the hardware, not necessarily to use it. If you, for some reason, have a few hundred GPU lying around, it might be "cheap" to do the necessary training. Now, that's not your point - cost != price. But, still...
- can16358p 4y agoI can't think of anyone having a few hundred GPUs around unless: - They were into Ethereum mining and quit. - They've already built a cluster with them (e.g. in an academic setting). - They live in a datacenter. - They are a total psychopath. But even assuming one magically has all those GPUs available and ready to train, I don't want to calculate the power cost of it anyway. Unless one has access to free or extremely cheap electricity it would still be very expensive.
- raducu 4y ago> If you, for some reason, have a few hundred GPU lying around Not to nitpick, but that is like saying that if you have a Lamborghini lying around, a Sunday trip in one is not so expensive.
- vgel 4y agoMy guess is they're mostly vanity projects for large tech companies. While the models have some value, they also serve as interesting research projects and help them attract ML talent to work on more profitable models like ad-targeting.
- dandiep 4y agoThere are tons of commercial uses for these models. I've been experimenting with an app targeted toward language learners [1]. We use large language models to: - Generate vocabulary - e.g. for biking: handlebars, pedals, shifters, etc - Generate translation exercises for given topic a learner wants to learn about - e.g. I raised the seat on my bike - Generate questions for the user - e.g. What are the different types of biking? - Provide more fluent ways to say things - I went on my bike to the store -> I rode my bike to the store - Provide explanations of the difference in meaning between two words And we have fine tuned smaller models to do other thing like grammar correction, exercise grading, and embedded search. These models are going to completely change the field of education in my opinion. 1) https://squidgies.app https://squidgies.app - be kind it's still a bit alpha
- MasterScrat 4y agoHuggingFace will soon release their BigScience model: https://twitter.com/BigScienceLLM/status/1539941348656168961 https://twitter.com/BigScienceLLM/status/1539941348656168961 "a 176 billion parameter transformer model that will be trained on roughly 300 billion words in 46 languages" So anything smaller than that will become worthless. May be a factor, companies have a last chance to make a PR splash before it happens. Read more about it: https://bigscience.huggingface.co/blog/model-training-launched https://bigscience.huggingface.co/blog/model-training-launch...
- rahidz 4y agoNot necessarily, only ~30% of the database is in English, so it likely won't be as good as a smaller model trained solely or mostly on English words. https://bigscience.huggingface.co/blog/building-a-tb-scale-multilingual-dataset-for-language-modeling https://bigscience.huggingface.co/blog/building-a-tb-scale-m...
- TaylorAlexander 4y agoIt kinda seems like a model trained on multiple languages would to some extent be better at English than a model trained only on English? I mean so much of English comes from other languages, and understanding language as a concept transcends any specific language. Of course there are limits and it needs good English vocabulary and understanding, but I feel the extra languages would help rather than hinder English performance.
- lairv 4y ago"worthless" huh, not everyone can afford inference of a ~500gb models, depending on the the speed/rate you need you might definitely go for smaller model But maybe your sentence was more about "after BigScience model, open-sourcing anything smaller than that will be useless" which isn't necessarily true either, because there is still room to improve parameter efficiency, i.e. smaller models with comparabale performances
- t_mann 4y agoOnly half-joking use case: active communities like this one on HN make sites attractive to human visitors. A new site could use bots to fake activity. Not sure it would work in the long run though.