14 ms·

Although let's see if it is sustainable. I pay for ChatGPT Plus but most people aren't, and from what I've read they're losing money like crazy. We may wind up

Although let's see if it is sustainable. I pay for ChatGPT Plus but most people aren't, and from what I've read they're losing money like crazy. We may wind up seeing this as the technical high point before the various competitors cut back on expensive hardware/energy usage for the free tier, and the quality of the responses takes a dive. This might sound a bit conspiratorial so apologies for that but back when the competition for Google Assistant/Siri/Cortana/Alexa was super hot, the responses to the various voice assistance was almost eerie in that it could infer what you wanted/needed. Then as things cooled off, they got gradually dumber/worse every year since then. They're legitimately bad today (Siri in particular, but even Google Assistant is much worse than itself from back then). I suspect it is because hardware costs were too high so they found cheaper models that could run on a potato/locally. From what I've read to get ChatGPT responses in the times it currently takes, they may need to be running something like two 3080 GPUs, 64 GB of RAM, and a high end CPU, with the power draw associated. So to make the economics work (even with ad revenue for example), either a technological breakthrough has to occur to make running today's models much cheaper OR they will cut back making the responses objectively worse (they arguably already did this once with a 3.5 model change).

sharemywin 3y ago
isn't ChatGPT just an investor demo anyway? I doubt they run out of cash at least until they fund building GPT 5. If it flops then who knows.
- stavros 3y ago
  No, that was GPT, GPT-2, and GPT-3.
malwrar 3y ago
The only moat openai has is their financial resources to pay for training and their vast troves of data scraped before reddit/twitter/etc started trying to lock things down. I think at this point there are enough people knowledgeable and interested in open LM technology that, if openai begins to degrade in quality, we won’t lose this technology forever. I’ve personally been obsessively reading papers about new models and transformer modifications, and having succeeded in implementing some I think the experience has taught me that this tech is probably where web was in the 90s: easy in concept, expensive and hard to scale due to a few missing easy pieces thatll come later.
- viscanti 3y ago
  > The only moat openai has is their financial resources to pay for training and their vast troves of data scraped before reddit/twitter/etc started trying to lock things down. Do we know that there's zero benefit from the user generated data they're getting now? They know when someone clicks the button to regenerate, and in theory they can have GPT4 review all the responses and classify if they're good or not. I don't know how beneficial that data is, so I'm curious if it's been proven to be completely worthless or not. If it's valuable at all, and openai is using it to continually improve the performance of ChatGPT, then maybe it will be difficult for a competitor to ever get traffic and data to their alternative.
  - malwrar 3y ago
    I don’t know about “zero benefit”—certainly their access to customer queries gives them a gods eye view into how people are using their LM implementation and to some degree could probably be used to fine-tune their model—but I doubt it gives them a leg up that couldn’t be matched through the sheer scale of academics and volunteers who contribute to open solutions & research. Maybe we invent a method for directly fine-tuning based on user interactions, or we discover that there’s some shelf where further fine-tuning only confuses the model, or discover some magical architectural modification or dataset that enables high-quality interactions. We’re still at the stage where basic modifications to the model breaks seemingly insurmountable problems. ALiBi for example swaps in a new attention mechanism, literally just a matrix addition operation, and found that it enables inference far past the context limit. I’m ultimately optimistic, their data is valuable but I don’t think it’s insurmountable.
- BlueTemplar 3y ago
  reddit/twitter/etc(?) data might be vast, but it's very low quality...
  - fomine3 3y ago
    Sometimes low quality text is what people need.
spott 3y ago
If it was just computer that caused the various assistants to get bad, then why haven’t they gotten good again? Google assistant came out in 2016! That is 7 years ago! Since then, GPU compute in flops has 5xed… for single precision. If you include half-floats (not a thing back then), it is 75x the single precision flops. I think the reason that assistants have “gotten bad” is a combination of edge case controls (avoiding politically problematic results), increased capabilities mean more opportunities for failures, and their failures are amplified, and their successes ignored (in other words they aren’t necessarily worse, we just expect more out of them).
- assistantist 3y ago
  If it was just a matter of flops then AI would already be driving cars and making door to door food deliveries. The problem is neither the energy expenditure nor the compute capacity. No one knows what the proper architecture should be for general purpose intelligence assistants so everyone has to fine-tune the existing models on their specialized use cases and even then it's a hit or miss depending on how much training data you have available. No one currently has any ideas on how to deploy auto-adaptive ML software that continuously adapts to new data so the industry has settled on deploying snapshots and then hoping the data set that was used for the snapshot is going to be good enough for most use cases. It seems to work well enough for now and since Facebook has decided to open source most of their work someone in the open source community might figure out how to continously update these models without deterioration in output quality.
  - thfuran 3y ago
    The difficulty of improving AI capability towards AGI isn't really related to a purported worsening of an already-implemented system for financial reasons.
    - financialisty 3y ago
      If money is the bottleneck then we will never get AGI. A superior machine intelligence would have no use for money.
ColonelPhantom 3y ago
One thing about LLMs is that from what I know, batching is very effective with them, because to generate a token you need to basically go through the entire network for just that one sample. If you batch, you still need to stream the entire network, but that doesn't get any more expensive; you use each piece of data more often on hardware that used to just sit idle. So I assume that at an OpenAI-scale, they're more than able to batch up requests to their models, which gives a tiny latency increase (assuming they're getting many requests per second), but massively improves compute utilization.
Madmallard 3y ago
We need a strong localized version at least for software development that is free as soon as possible. It is such a competitive advantage there's no way it will stick around forever for as cheap as it is.
steno132 3y ago
It’s sustainable. Llama.cpp is orders of magnitude more efficient than Llama because it’s in Rust. OpenAI hasn’t even touched low level languages yet.
- donkeybeer 3y ago
  C++ not Rust. And most ML libraries being called from python or whatever are themselves C++ but of course there's always room for improvement, esp if you reimplement the entire corebase in C++ and specialize the inference program for your architecture. Specialization often leads to possibility of more efficient code. You are not running a generic neural net but you know the architecture beforehand so you can design its memory layout etc for efficiency yes.
- raincole 3y ago
  Are you seriously not knowing what .cpp in llama.cpp stands for?
NoLinkToMe 3y ago
> the responses to the various voice assistance was almost eerie in that it could infer what you wanted/needed. Then as things cooled off, they got gradually dumber/worse every year since then. They're legitimately bad today (Siri in particular It's a compelling argument but I find that difficult to believe given the timespan. Siri launched in 2011. The FLOP/s per dollar rates for GPUs involved in ML doubled every 2 years across this peripd [0]. Meaning in the past 12 years, you're talking about 64x GPU performance per dollar spent. Combine that with 36% general inflation + some increased spending on mobile computing, plus a decade of time on making algorithms for a big consumer product more efficient, and you get to wildly more favourable economics, as in, two orders of magnitude more favourable than when Siri launched in 2011. I'm not surprised this could well be happening for ChatGPT (would be cool to see the first public audit standards of AI products emerge, to test this hypothesis). But I don't think it holds true for the 12y period for Siri and co. https://epochai.org/blog/trends-in-gpu-price-performance https://epochai.org/blog/trends-in-gpu-price-performance
- Turskarama 3y ago
  The hardware being cheaper only matters if they truly want to improve the product, which it's not clear that they do. Lowering the number of FLOPs needed will always reduce the cost, as the hardware gets cheaper the cost is just reduced faster. The question is, would improving Siri sell more IPhones? If not, then why bother?
- mattnewton 3y ago
  I don’t think Siri et al got worse, we just had time for the hype of the excellent voice to text and back again to wear off, and realized that they were too limited at figuring out actions to be very useful.
- telchior 3y ago
  I do remember meeting the Siri team before Apple bought them (I was writing for a tech blog at the time) and having them show me a bunch of really cool queries. I was pretty impressed. Shortly after, Apple bought them, and significantly after that, actually launched it as a product. I was not at all impressed by what launched. Keeping in mind that the pre-sale demo I got was in pretty controlled conditions (i.e. I could have been duped): After Apple's launch of Siri, someone else who had been working with Siri told me that he was really unhappy with what Apple had done; according to him, they gutted a bunch of capabilities, mainly because A) they couldn't be launched to all customers or B) just didn't fit with Apple's product vision. To give a specific example, the pre-Apple Siri could make reservations at a hotel in SF / NYC / Austin for you but not in Dubuque or Tokyo, so that had to go. Of course, that original Siri was very different from ChatGPT: it was based on structured queries and the "cool stuff" it could do was basically a ton of API linkages to outside services. At that point, there'd already been a wave of "natural language" companies like Powerset and Semantic Web that had promised the world and then flamed out, so everyone was aware that unstructured queries were basically impossible. Funny how fast things change.
keyle 3y ago
I'm actually worried of the inevitable enshitification of Open AI. Right now, GPT 4 is great and I use it every day. I rely less and less on Google and save a ton of time. Google has gone downhill hard for a few years, thanks to all these silly blog engines like medium, substack and the like. I'm really worried and would like to 'freeze' GPT 4 as is, at least with the data up to Sep 2021, and for them to not change or tweak any of it further. I wish we could have an offline solution just as good, or a guarantee of GPT 4 LTS. I don't mind paying so as long as it's useful. I'm convinced that inevitably, every successful engineering driven business falls into a marketing business; and once market research and sales drive every business decision, the product is boned.
- version_five 3y ago
  This is a good point and I'm surprised it hasn't had more attention. We've seen history repeat itself enough times that it's inevitable. Probably the best you can do is download a few of the full precision 70B or bigger models (LLaMA etc) to at least have some state of the art weights around you could use yourself if you wanted if/when openAI goes to shit. The nice thing about the language models is as least they're relatively compact, compared to google search which you'd never be able to host locally. Though thinking about it, in a way LLMs are local copies of the "good" internet. The big ones mostly seem to know what information there is to know and have none of the crap and link/content farms that is now most of the internet.
  - MPSimmons 3y ago
    >... compared to google search which you'd never be able to host locally That's something I haven't thought about for a long time. How big is the Google index, would you say? Petabytes or exabytes, or am I vastly overestimating?
    - raincole 3y ago
      > over 100,000,000 gigabytes Accodrding to Google: https://www.google.com/search/howsearchworks/how-search-works/organizing-information https://www.google.com/search/howsearchworks/how-search-work...
scarface_74 3y ago
> This might sound a bit conspiratorial so apologies for that but back when the competition for Google Assistant/Siri/Cortana/Alexa was super hot, the responses to the various voice assistance was almost eerie in that it could infer what you wanted/needed. Siri at least was always a simple intent based system that was never very intelligent and very much a brute voice method where someone went in and manually configured patterns for it to recognize. You can tell by how the API for Siri works for third party integrations. On a much smaller scale, I’ve done something similar with Amazon Lex (the AWS version of Alexa). I am not saying the underlying architecture of Siri or Alexa is simple. But how you build on top of it to get it to do things is just giving it patterns for it to recognize and various permutations. The hardest part is the speech to text once you get everything in text form, the complexity is not that great.
- asynchronous 3y ago
  Reminds me of the memes/era when AI truly was just a million nested if statements
kromem 3y ago
There are several breakthroughs actually, every few weeks, inching closer and closer to the next generation of hardware for ML models with optoelectronic compute. This is just one of the most recent: https://news.mit.edu/2023/system-could-yield-more-powerful-efficient-llms-0822 https://news.mit.edu/2023/system-could-yield-more-powerful-e... Pretty much every single major tech university has their research/VC arm with a hand in photonics for AI. The initial rollout is around interconnect, but compute is moving along even faster than I'd expected, with significant steps forward occurring frequently. I'm skeptical we'll be seeing actual quantum computing any time soon for general purpose stuff due to error correction, but for the specific needs of training and running a neural network, it's pretty much the perfect marriage of hardware limitations with task specifications. So yes, this software trend is going to end up being much cheaper and efficient. By orders of magnitude leaps, not just at the creeping pace of Moore's law.
atleastoptimal 3y ago
That's why model hashes/transparency is necessary. There's absolutely no reason anyone should ever consider building a deployed app that uses an API whose underlying model could change any second to something far dumber. The only thing that could avoid this is competition and commodification of LLMs.
raincole 3y ago
> from what I've read they're losing money like crazy. Source? Random tweets don't count. > They're legitimately bad today (Siri in particular, but even Google Assistant is much worse than itself from back then) Your anecdote. My anecdote is Siri doesn't change, but our expectation becomes much, much higher. Siri blew everyone's mind by... making a phone call to nearby Starbucks. When it came out that phone call was a super big deal.
d_sem 3y ago
Assuming your hardware requirements are correct I would gladly buy this hardware for on premise support for always available ChatGPT4.
jdkee 3y ago
Didn't OpenAI get a ten billion dollar infusion from Microsoft for 20% of the company?
BeFlatXIII 3y ago
IMO, your post should be one of the canned responses to AI doomers. No, LLMs won't replace fiction writers and diffusion models won't replace artists because the people who are too cheap to pay writers and artists are also too stingy to pay AI hosting companies.
deleted 3y ago
[deleted]
wongarsu 3y ago
So far OpenAI's modus operandi was "does it get better if we make it bigger". But now we've reached "good enough" (arguably already with GPT3.5, as evidenced by free ChatGPT running on GPT3.5-turbo). There has been promising work about making models smaller without losing performance (e.g. by training them longer), or quantizing the weights (llama.cpp etc) to make the model cheaper to run. I believe we will see a lot more focus on this over the next couple years, and given how there's been comparatively little work done in this direction I wouldn't be surprised by 10x or 100x efficiency gains.
- SubiculumCode 3y ago
  efficiency efficiency efficiency will be the call. How do we reduce model size without losing ability. Can hardware be designed to run a particular set of weights efficiently?
- theRealMe 3y ago
  I don’t want to make unsubstantiated guesses, but we should remember that GPT3.5 was originally just GPT3.5 and when they came out with turbo it not only sped it up but also reduced the cost (to us) by like 10x of what it used to be if I remember right. So they are (or at the very least “were”) working on and succeeding on reducing its cost to run.
- sharemywin 3y ago
  let's not forget about good old fashion moore's law. which I believe in GPUs is even faster. and I'm not sure if some of these exotics Hardware architectures will pane out or not either.
  - flangola7 3y ago
    Faster is an understatement. On password hashing the margin of safety dropped by 30x since 2020 due to GPU improvements.
unshavedyak 3y ago
On the plus side i think at-home LLMs will continue to get a lot better. ChatGPT 3.5/4 both seem in reach for Llama2 and seeing all the amazing fine tunings that were available for Llama1, i suspect we have a very functional future of at home LLMs. I say functional because i don't think we've seen the larger reasoning skills scale well for smaller deployments, so that may be out of reach. But i think a future were we run something far better than Siri, entirely at home, hooked up to our home APIs in a functional spoken interface seems in reach and awesome (to me hah). I look forward to hooking all my at-home cameras up to a voice recognition, image recognition, and LLM detection suite and just speak aloud and have the AI-suite "do stuff". It sounds really, really fun to me. If it's cheap it'll be slow, but imo that's okay. For home use i think these systems can be the glue that tie our DBs/Ingestions/etc together, rather than the big-brain that knows everything.
- JimtheCoder 3y ago
  "I look forward to hooking all my at-home cameras up to a voice recognition, image recognition, and LLM detection suite and just speak aloud and have the AI-suite "do stuff". It sounds really, really fun to me." Alternatively, this sounds like the starting plot to a crossover sci-fi/horror movie...
  - renewiltord 3y ago
    Of course it does. That's the structure of all sci-fi horror https://dresdencodak.com/2009/09/22/caveman-science-fiction/ https://dresdencodak.com/2009/09/22/caveman-science-fiction/
  - jstarfish 3y ago
    They never read Demon Seed.
  - dandellion 3y ago
    The door refused to open. It said, “Five cents, please.” He searched his pockets. No more coins; nothing. “I’ll pay you tomorrow,” he told the door. Again he tried the knob. Again it remained locked tight. “What I pay you,” he informed it, “is in the nature of a gratuity; I don’t have to pay you.” “I think otherwise,” the door said. “Look in the purchase contract you signed when you bought this conapt.” In his desk drawer he found the contract; since signing it he had found it necessary to refer to the document many times. Sure enough; payment to his door for opening and shutting constituted a mandatory fee. Not a tip. “You discover I’m right,” the door said. It sounded smug. From the drawer beside the sink Joe Chip got a stainless steel knife; with it he began systematically to unscrew the bolt assembly of his apt’s money-gulping door. “I’ll sue you,” the door said as the first screw fell out. Joe Chip said, “I’ve never been sued by a door. But I guess I can live through it. -- Philip K. Dick, Ubik

sharemywin 3y ago

isn't ChatGPT just an investor demo anyway? I doubt they run out of cash at least until they fund building GPT 5. If it flops then who knows.

stavros 3y ago
No, that was GPT, GPT-2, and GPT-3.

malwrar 3y ago

The only moat openai has is their financial resources to pay for training and their vast troves of data scraped before reddit/twitter/etc started trying to lock things down. I think at this point there are enough people knowledgeable and interested in open LM technology that, if openai begins to degrade in quality, we won’t lose this technology forever. I’ve personally been obsessively reading papers about new models and transformer modifications, and having succeeded in implementing some I think the experience has taught me that this tech is probably where web was in the 90s: easy in concept, expensive and hard to scale due to a few missing easy pieces thatll come later.

viscanti 3y ago
> The only moat openai has is their financial resources to pay for training and their vast troves of data scraped before reddit/twitter/etc started trying to lock things down. Do we know that there's zero benefit from the user generated data they're getting now? They know when someone clicks the button to regenerate, and in theory they can have GPT4 review all the responses and classify if they're good or not. I don't know how beneficial that data is, so I'm curious if it's been proven to be completely worthless or not. If it's valuable at all, and openai is using it to continually improve the performance of ChatGPT, then maybe it will be difficult for a competitor to ever get traffic and data to their alternative.
- malwrar 3y ago
  I don’t know about “zero benefit”—certainly their access to customer queries gives them a gods eye view into how people are using their LM implementation and to some degree could probably be used to fine-tune their model—but I doubt it gives them a leg up that couldn’t be matched through the sheer scale of academics and volunteers who contribute to open solutions & research. Maybe we invent a method for directly fine-tuning based on user interactions, or we discover that there’s some shelf where further fine-tuning only confuses the model, or discover some magical architectural modification or dataset that enables high-quality interactions. We’re still at the stage where basic modifications to the model breaks seemingly insurmountable problems. ALiBi for example swaps in a new attention mechanism, literally just a matrix addition operation, and found that it enables inference far past the context limit. I’m ultimately optimistic, their data is valuable but I don’t think it’s insurmountable.
BlueTemplar 3y ago
reddit/twitter/etc(?) data might be vast, but it's very low quality...
- fomine3 3y ago
  Sometimes low quality text is what people need.

spott 3y ago

If it was just computer that caused the various assistants to get bad, then why haven’t they gotten good again? Google assistant came out in 2016! That is 7 years ago! Since then, GPU compute in flops has 5xed… for single precision. If you include half-floats (not a thing back then), it is 75x the single precision flops. I think the reason that assistants have “gotten bad” is a combination of edge case controls (avoiding politically problematic results), increased capabilities mean more opportunities for failures, and their failures are amplified, and their successes ignored (in other words they aren’t necessarily worse, we just expect more out of them).

assistantist 3y ago
If it was just a matter of flops then AI would already be driving cars and making door to door food deliveries. The problem is neither the energy expenditure nor the compute capacity. No one knows what the proper architecture should be for general purpose intelligence assistants so everyone has to fine-tune the existing models on their specialized use cases and even then it's a hit or miss depending on how much training data you have available. No one currently has any ideas on how to deploy auto-adaptive ML software that continuously adapts to new data so the industry has settled on deploying snapshots and then hoping the data set that was used for the snapshot is going to be good enough for most use cases. It seems to work well enough for now and since Facebook has decided to open source most of their work someone in the open source community might figure out how to continously update these models without deterioration in output quality.
- thfuran 3y ago
  The difficulty of improving AI capability towards AGI isn't really related to a purported worsening of an already-implemented system for financial reasons.
  - financialisty 3y ago
    If money is the bottleneck then we will never get AGI. A superior machine intelligence would have no use for money.

ColonelPhantom 3y ago

One thing about LLMs is that from what I know, batching is very effective with them, because to generate a token you need to basically go through the entire network for just that one sample. If you batch, you still need to stream the entire network, but that doesn't get any more expensive; you use each piece of data more often on hardware that used to just sit idle. So I assume that at an OpenAI-scale, they're more than able to batch up requests to their models, which gives a tiny latency increase (assuming they're getting many requests per second), but massively improves compute utilization.

Madmallard 3y ago

We need a strong localized version at least for software development that is free as soon as possible. It is such a competitive advantage there's no way it will stick around forever for as cheap as it is.

steno132 3y ago

It’s sustainable. Llama.cpp is orders of magnitude more efficient than Llama because it’s in Rust. OpenAI hasn’t even touched low level languages yet.

donkeybeer 3y ago
C++ not Rust. And most ML libraries being called from python or whatever are themselves C++ but of course there's always room for improvement, esp if you reimplement the entire corebase in C++ and specialize the inference program for your architecture. Specialization often leads to possibility of more efficient code. You are not running a generic neural net but you know the architecture beforehand so you can design its memory layout etc for efficiency yes.
raincole 3y ago
Are you seriously not knowing what .cpp in llama.cpp stands for?

NoLinkToMe 3y ago

> the responses to the various voice assistance was almost eerie in that it could infer what you wanted/needed. Then as things cooled off, they got gradually dumber/worse every year since then. They're legitimately bad today (Siri in particular It's a compelling argument but I find that difficult to believe given the timespan. Siri launched in 2011. The FLOP/s per dollar rates for GPUs involved in ML doubled every 2 years across this peripd [0]. Meaning in the past 12 years, you're talking about 64x GPU performance per dollar spent. Combine that with 36% general inflation + some increased spending on mobile computing, plus a decade of time on making algorithms for a big consumer product more efficient, and you get to wildly more favourable economics, as in, two orders of magnitude more favourable than when Siri launched in 2011. I'm not surprised this could well be happening for ChatGPT (would be cool to see the first public audit standards of AI products emerge, to test this hypothesis). But I don't think it holds true for the 12y period for Siri and co. https://epochai.org/blog/trends-in-gpu-price-performance https://epochai.org/blog/trends-in-gpu-price-performance

Turskarama 3y ago
The hardware being cheaper only matters if they truly want to improve the product, which it's not clear that they do. Lowering the number of FLOPs needed will always reduce the cost, as the hardware gets cheaper the cost is just reduced faster. The question is, would improving Siri sell more IPhones? If not, then why bother?
mattnewton 3y ago
I don’t think Siri et al got worse, we just had time for the hype of the excellent voice to text and back again to wear off, and realized that they were too limited at figuring out actions to be very useful.
telchior 3y ago
I do remember meeting the Siri team before Apple bought them (I was writing for a tech blog at the time) and having them show me a bunch of really cool queries. I was pretty impressed. Shortly after, Apple bought them, and significantly after that, actually launched it as a product. I was not at all impressed by what launched. Keeping in mind that the pre-sale demo I got was in pretty controlled conditions (i.e. I could have been duped): After Apple's launch of Siri, someone else who had been working with Siri told me that he was really unhappy with what Apple had done; according to him, they gutted a bunch of capabilities, mainly because A) they couldn't be launched to all customers or B) just didn't fit with Apple's product vision. To give a specific example, the pre-Apple Siri could make reservations at a hotel in SF / NYC / Austin for you but not in Dubuque or Tokyo, so that had to go. Of course, that original Siri was very different from ChatGPT: it was based on structured queries and the "cool stuff" it could do was basically a ton of API linkages to outside services. At that point, there'd already been a wave of "natural language" companies like Powerset and Semantic Web that had promised the world and then flamed out, so everyone was aware that unstructured queries were basically impossible. Funny how fast things change.

keyle 3y ago

I'm actually worried of the inevitable enshitification of Open AI. Right now, GPT 4 is great and I use it every day. I rely less and less on Google and save a ton of time. Google has gone downhill hard for a few years, thanks to all these silly blog engines like medium, substack and the like. I'm really worried and would like to 'freeze' GPT 4 as is, at least with the data up to Sep 2021, and for them to not change or tweak any of it further. I wish we could have an offline solution just as good, or a guarantee of GPT 4 LTS. I don't mind paying so as long as it's useful. I'm convinced that inevitably, every successful engineering driven business falls into a marketing business; and once market research and sales drive every business decision, the product is boned.

version_five 3y ago
This is a good point and I'm surprised it hasn't had more attention. We've seen history repeat itself enough times that it's inevitable. Probably the best you can do is download a few of the full precision 70B or bigger models (LLaMA etc) to at least have some state of the art weights around you could use yourself if you wanted if/when openAI goes to shit. The nice thing about the language models is as least they're relatively compact, compared to google search which you'd never be able to host locally. Though thinking about it, in a way LLMs are local copies of the "good" internet. The big ones mostly seem to know what information there is to know and have none of the crap and link/content farms that is now most of the internet.
- MPSimmons 3y ago
  >... compared to google search which you'd never be able to host locally That's something I haven't thought about for a long time. How big is the Google index, would you say? Petabytes or exabytes, or am I vastly overestimating?
  - raincole 3y ago
    > over 100,000,000 gigabytes Accodrding to Google: https://www.google.com/search/howsearchworks/how-search-works/organizing-information https://www.google.com/search/howsearchworks/how-search-work...

scarface_74 3y ago

> This might sound a bit conspiratorial so apologies for that but back when the competition for Google Assistant/Siri/Cortana/Alexa was super hot, the responses to the various voice assistance was almost eerie in that it could infer what you wanted/needed. Siri at least was always a simple intent based system that was never very intelligent and very much a brute voice method where someone went in and manually configured patterns for it to recognize. You can tell by how the API for Siri works for third party integrations. On a much smaller scale, I’ve done something similar with Amazon Lex (the AWS version of Alexa). I am not saying the underlying architecture of Siri or Alexa is simple. But how you build on top of it to get it to do things is just giving it patterns for it to recognize and various permutations. The hardest part is the speech to text once you get everything in text form, the complexity is not that great.

asynchronous 3y ago
Reminds me of the memes/era when AI truly was just a million nested if statements

kromem 3y ago

There are several breakthroughs actually, every few weeks, inching closer and closer to the next generation of hardware for ML models with optoelectronic compute. This is just one of the most recent: https://news.mit.edu/2023/system-could-yield-more-powerful-efficient-llms-0822 https://news.mit.edu/2023/system-could-yield-more-powerful-e... Pretty much every single major tech university has their research/VC arm with a hand in photonics for AI. The initial rollout is around interconnect, but compute is moving along even faster than I'd expected, with significant steps forward occurring frequently. I'm skeptical we'll be seeing actual quantum computing any time soon for general purpose stuff due to error correction, but for the specific needs of training and running a neural network, it's pretty much the perfect marriage of hardware limitations with task specifications. So yes, this software trend is going to end up being much cheaper and efficient. By orders of magnitude leaps, not just at the creeping pace of Moore's law.

atleastoptimal 3y ago

That's why model hashes/transparency is necessary. There's absolutely no reason anyone should ever consider building a deployed app that uses an API whose underlying model could change any second to something far dumber. The only thing that could avoid this is competition and commodification of LLMs.

raincole 3y ago

> from what I've read they're losing money like crazy. Source? Random tweets don't count. > They're legitimately bad today (Siri in particular, but even Google Assistant is much worse than itself from back then) Your anecdote. My anecdote is Siri doesn't change, but our expectation becomes much, much higher. Siri blew everyone's mind by... making a phone call to nearby Starbucks. When it came out that phone call was a super big deal.

d_sem 3y ago

Assuming your hardware requirements are correct I would gladly buy this hardware for on premise support for always available ChatGPT4.

jdkee 3y ago

Didn't OpenAI get a ten billion dollar infusion from Microsoft for 20% of the company?

BeFlatXIII 3y ago

IMO, your post should be one of the canned responses to AI doomers. No, LLMs won't replace fiction writers and diffusion models won't replace artists because the people who are too cheap to pay writers and artists are also too stingy to pay AI hosting companies.

deleted 3y ago

[deleted]

wongarsu 3y ago

So far OpenAI's modus operandi was "does it get better if we make it bigger". But now we've reached "good enough" (arguably already with GPT3.5, as evidenced by free ChatGPT running on GPT3.5-turbo). There has been promising work about making models smaller without losing performance (e.g. by training them longer), or quantizing the weights (llama.cpp etc) to make the model cheaper to run. I believe we will see a lot more focus on this over the next couple years, and given how there's been comparatively little work done in this direction I wouldn't be surprised by 10x or 100x efficiency gains.

SubiculumCode 3y ago
efficiency efficiency efficiency will be the call. How do we reduce model size without losing ability. Can hardware be designed to run a particular set of weights efficiently?
theRealMe 3y ago
I don’t want to make unsubstantiated guesses, but we should remember that GPT3.5 was originally just GPT3.5 and when they came out with turbo it not only sped it up but also reduced the cost (to us) by like 10x of what it used to be if I remember right. So they are (or at the very least “were”) working on and succeeding on reducing its cost to run.
sharemywin 3y ago
let's not forget about good old fashion moore's law. which I believe in GPUs is even faster. and I'm not sure if some of these exotics Hardware architectures will pane out or not either.
- flangola7 3y ago
  Faster is an understatement. On password hashing the margin of safety dropped by 30x since 2020 due to GPU improvements.

unshavedyak 3y ago

On the plus side i think at-home LLMs will continue to get a lot better. ChatGPT 3.5/4 both seem in reach for Llama2 and seeing all the amazing fine tunings that were available for Llama1, i suspect we have a very functional future of at home LLMs. I say functional because i don't think we've seen the larger reasoning skills scale well for smaller deployments, so that may be out of reach. But i think a future were we run something far better than Siri, entirely at home, hooked up to our home APIs in a functional spoken interface seems in reach and awesome (to me hah). I look forward to hooking all my at-home cameras up to a voice recognition, image recognition, and LLM detection suite and just speak aloud and have the AI-suite "do stuff". It sounds really, really fun to me. If it's cheap it'll be slow, but imo that's okay. For home use i think these systems can be the glue that tie our DBs/Ingestions/etc together, rather than the big-brain that knows everything.

JimtheCoder 3y ago
"I look forward to hooking all my at-home cameras up to a voice recognition, image recognition, and LLM detection suite and just speak aloud and have the AI-suite "do stuff". It sounds really, really fun to me." Alternatively, this sounds like the starting plot to a crossover sci-fi/horror movie...
- renewiltord 3y ago
  Of course it does. That's the structure of all sci-fi horror https://dresdencodak.com/2009/09/22/caveman-science-fiction/ https://dresdencodak.com/2009/09/22/caveman-science-fiction/
- jstarfish 3y ago
  They never read Demon Seed.
- dandellion 3y ago
  The door refused to open. It said, “Five cents, please.” He searched his pockets. No more coins; nothing. “I’ll pay you tomorrow,” he told the door. Again he tried the knob. Again it remained locked tight. “What I pay you,” he informed it, “is in the nature of a gratuity; I don’t have to pay you.” “I think otherwise,” the door said. “Look in the purchase contract you signed when you bought this conapt.” In his desk drawer he found the contract; since signing it he had found it necessary to refer to the document many times. Sure enough; payment to his door for opening and shutting constituted a mandatory fee. Not a tip. “You discover I’m right,” the door said. It sounded smug. From the drawer beside the sink Joe Chip got a stainless steel knife; with it he began systematically to unscrew the bolt assembly of his apt’s money-gulping door. “I’ll sue you,” the door said as the first screw fell out. Joe Chip said, “I’ve never been sued by a door. But I guess I can live through it. -- Philip K. Dick, Ubik