7 ms·

Dreaming of the day I run my own models and can be entirely responsible for my own outages.

hospitalJail 3y ago
This came up the other day. I decided to tease everyone with an 'I told you so' about using some third party hosting service instead of the offline one I had developed years prior. The offline service was still working, and people were doing their job. The online service was not working, and it was causing other people to be unable to do their job. We had 0 control over the third party. The other thing, I make software and I basically don't touch it for a few years or ever. These third party services are always updating and breaking causing us to update as well. IB4 let me write my own compilers so I have real control.
- dangero 3y ago
  Do you write your refrigerators firmware?
  - danShumway 3y ago
    If you are regularly updating your refrigerator's firmware or your refrigerator's firmware relies on an Internet connection to function, then I am very sorry to say this but you have lost control of your life :)
    - fennecfoxy 3y ago
      Depends on how much technology is in the fridge I guess. If it's just a plain old fridge then no need to be smart. If they say, found a way to make it xx% more efficient because they xyz and dynamically abc with some small arm processor then fair enough. Huh, the graph here is interesting: https://appliance-standards.org/blog/how-your-refrigerator-has-kept-its-cool-over-40-years-efficiency-improvements https://appliance-standards.org/blog/how-your-refrigerator-h... Shows you how creating and enforcing standards is the driver for stuff like this. I wonder how we could make them even more efficient, some way to stop the transfer of warm air when the door is opened? Wonder if it's possible to create some sort of air curtain at the front when it's opened to prevent warm air coming in, ie use driven air velocity to overcome the cold air wants to come out, hot air wants to come in. Hmmm.
      - danShumway 3y ago
        > I wonder how we could make them even more efficient, some way to stop the transfer of warm air when the door is opened? I wonder how we could make them even more efficient, some way to stop the transfer of warm air when the door is opened? Wonder if it's possible to create some sort of air curtain at the front when it's opened to prevent warm air coming in, ie use driven air velocity to overcome the cold air wants to come out, hot air wants to come in. Hmmm. That is an interesting idea, but I don't think an Internet connection would help with it :) > Shows you how creating and enforcing standards is the driver for stuff like this. Also agreed that is an interesting graph, I agree that it shows how standards and better production has led to decreased energy usage -- but notably, a lot of those standards are around better insulation and more efficient components. Putting an extra layer of foam in your fridge or having sensors in your fridge that help regulate temperature definitely doesn't mean you've lost control of your life. But needing to download a firmware update to your Internet-enabled fridge that uses a Samsung account where you now can't access your grocery list until you finish the mandated update which changes your fridge's UI on its mobile app -- I think that means you've lost control of your life :)
        fennecfoxy 3y ago
        Oh yeah for sure, but I think there are definitely reasons for it. The enshittification/technoshit that comes out of the iot world (and all other devices) because of corporate greed just ruins it even more. The whole signing up for a Samsung account thing etc for your fridge. Stuff like this really just needs to be legislated under some kind of "all technology should just work, locally and with one another with at least an agreed set of features" level. Apple should have been legally forced to use USB C (or whatever alternative was best) ages ago, even before the EU got to them. Apple were happy to use Wifi/Bluetooth/etc/etc standards yet still wanted to use other proprietary BS. Same goes for literally everything else: all technologies should work together using at least a common method (with say options for proprietary stuff) and iot/whatever should all work flawlessly locally without any account or internet connectivity (which should all be 100% optional). Devices should work flawlessly even if the company that produces them has shut down all servers and gone bankrupt. We need to force our governments to do this stuff for us.
  - joshuaissac 3y ago
    No, but the firmware runs locally, instead of in someone else's cloud, so an outage in their cloud cannot take down my fridge.
    - phyrex 3y ago
      https://www.samsung.com/us/explore/family-hub-refrigerator/overview/ https://www.samsung.com/us/explore/family-hub-refrigerator/o...
      - danShumway 3y ago
        Nice, looks like we finally got around to inventing refrigerator magnets! ---- That is a little bit dismissive of me though. There are some cool features here: I can now "entertain in my kitchen", which is definitely a normal thing that normal people do. I love getting everyone together to crowd around my refrigerator so that we can all watch Game of Thrones. And I can use Amazon Alexa from my fridge just in case I'm not able to talk out loud to the cheap unobtrusive device that has a microphone in it specifically so that it can be placed in any room of the house. So having that option is good. And perhaps the biggest deal of all, I can finally "shop from home." That was a huge problem for me before, I kept thinking, "if only I had a better refrigerator I could finally buy things on websites." And this is a great bargain for only 3-5 thousand dollars! I can't believe I was planning to buy some crappy normal refrigerator for less than a thousand bucks and then use the extra money I saved to mount a giant flat-screen TV hooked up to a Chromecast in my kitchen. That would have been a huge mistake for me to make. Honestly it's just the icing on the cake that I can "set as many timers as [I] want." That's a great feature for someone like me because I can't set any timers at all using my phone or a voice assistant. /s ---- <serious>Holy crud, smart-device manufacturers have become unhinged. The one feature that actually looks useful here is being able to take a picture of the inside of the fridge while you're away. That is basically the one feature that I would want from a fridge that isn't much-better handled using a phone or a tablet or a TV or a normal refrigerator button. Which, great, but the problem is that I know what the inside of my fridge looks like right now, and let me just say: if I was organized enough that a photograph of the inside of my fridge would be clear enough to tell me what food was in it, and if I was organized enough that the photo wouldn't just show 'a pile of old containers, some of them transparent and some of them not' -- I have a feeling that in that case I would no longer be the type of person that needed to take a photo of the inside of my refrigerator to know what was in it.
  - TeMPOraL 3y ago
    Why on Earth would a refrigerator need firmware?
    - dvaletin 3y ago
      To show you ads on a screen on a fancy door.
Philpax 3y ago
You can do that today depending on what you need from the models.
- jstummbillig 3y ago
  Well, for me that's still quite a bit more than the best ones provide, but I am sure we will get there.
- benterix 3y ago
  Let's say I'm writing Flask code all day, and I need help with various parts of my code. Can I do it today or not? With questions like, "How to add 'Log in with Google' to the login screen" etc.
  - capableweb 3y ago
    In short: no. Longer: In theory, but it'll require a bunch of glue and using multiple models depending on the specific task you need help with. Some models are great at working with code but suck at literally anything else, so if you want it to be able to help you with "Do X with Y" you need to at least have two models, one that can reason up with an answer, and another to implement said answer. There is no general-purpose ("FOSS") LLM that even come close to GPT4 at this point.
  - phyrex 3y ago
    There’s the code llama model that you can run locally I think which should be able to help you with that: https://ai.meta.com/blog/code-llama-large-language-model-coding/ https://ai.meta.com/blog/code-llama-large-language-model-cod...
  - wokwokwok 3y ago
    If you have sufficiently good hardware, the 34B code llama model [1] (hint: pick the quantised model you can use based on “Max RAM required”, eg. q5/q6) running on llama.cpp [2], can answer many generic python and flask related questions, but it’s not quite good enough to generate entire code blocks for you like gpt4. It’s probably as good as you can get at the moment though; and hey, trying it out costs you nothing but the time it takes to download llama.cpp and run “make” and then point it at the q6 model file. So if it’s no good, you’ve probably wasted nothing more than like 30 min giving it a try. [1] - https://huggingface.co/TheBloke/CodeLlama-34B-Instruct-GGUF https://huggingface.co/TheBloke/CodeLlama-34B-Instruct-GGUF [2] - https://github.com/ggerganov/llama.cpp https://github.com/ggerganov/llama.cpp
m3kw9 3y ago
And then you will have to endure not using a model by OpenAI that is 10x better than a local one
- cwkoss 3y ago
  I think there is probably a threshold of usefulness, local LLMs are expensive to run but pretty close to it for most use cases now. In a couple years, our smartphones will probably be powerful enough to run LLMs locally that are good enough for 80% of uses.
- grishka 3y ago
  What is so special about OpenAI's cloud hardware that one can't build themselves a similar server to run AI models of similar size?
  - objectivetruth 3y ago
    [dead]
  - baq 3y ago
    Nothing stopping you from buying an H100 and putting it in your desktop. As for me, I’ve got other uses for $45k.
    - dcchambers 3y ago
      If you wanted to buy one...are you even able to order one? Aren't they massively backordered by big companies? Is there a black marker for H100 cards?
      - baq 3y ago
        I took the quote from eBay. Not sure which shade of black that would be ;)
  - airspresso 3y ago
    The hardware is primarily standard Nvidia GPUs (A100s, H100s), but the scale of the infrastructure is on another level entirely. These models currently need clusters of GPU-powered servers to make predictions fast enough. Which explains why OpenAI partnered with Microsoft and got billions in funding to spend on compute. You can run (much) smaller LLM models on consumer-grade GPUs though. A single Nvidia GPU with 8 GB RAM is enough to get started with models like Zephyr, Mistral or Llama2 in their smallest versions (7B parameters). But it will be both slower and lower quality than anything OpenAI currently offers.
deleted 3y ago
[deleted]
boh 3y ago
You can do that today. Oobabooga + Hugging Face models.
- nomel 3y ago
  Having something that executes, and having something that's genuinely useful, are two different things. For my hand typed use case's, GPT-4 is the only acceptable model that doesn't leave me frustrated and angry at wasting time. For some automated stuff (converting text to json, etc), the local models are fine.

This came up the other day. I decided to tease everyone with an 'I told you so' about using some third party hosting service instead of the offline one I had developed years prior. The offline service was still working, and people were doing their job. The online service was not working, and it was causing other people to be unable to do their job. We had 0 control over the third party. The other thing, I make software and I basically don't touch it for a few years or ever. These third party services are always updating and breaking causing us to update as well. IB4 let me write my own compilers so I have real control.

dangero 3y ago
Do you write your refrigerators firmware?
- danShumway 3y ago
  If you are regularly updating your refrigerator's firmware or your refrigerator's firmware relies on an Internet connection to function, then I am very sorry to say this but you have lost control of your life :)
  - fennecfoxy 3y ago
    Depends on how much technology is in the fridge I guess. If it's just a plain old fridge then no need to be smart. If they say, found a way to make it xx% more efficient because they xyz and dynamically abc with some small arm processor then fair enough. Huh, the graph here is interesting: https://appliance-standards.org/blog/how-your-refrigerator-has-kept-its-cool-over-40-years-efficiency-improvements https://appliance-standards.org/blog/how-your-refrigerator-h... Shows you how creating and enforcing standards is the driver for stuff like this. I wonder how we could make them even more efficient, some way to stop the transfer of warm air when the door is opened? Wonder if it's possible to create some sort of air curtain at the front when it's opened to prevent warm air coming in, ie use driven air velocity to overcome the cold air wants to come out, hot air wants to come in. Hmmm.
    - danShumway 3y ago
      > I wonder how we could make them even more efficient, some way to stop the transfer of warm air when the door is opened? I wonder how we could make them even more efficient, some way to stop the transfer of warm air when the door is opened? Wonder if it's possible to create some sort of air curtain at the front when it's opened to prevent warm air coming in, ie use driven air velocity to overcome the cold air wants to come out, hot air wants to come in. Hmmm. That is an interesting idea, but I don't think an Internet connection would help with it :) > Shows you how creating and enforcing standards is the driver for stuff like this. Also agreed that is an interesting graph, I agree that it shows how standards and better production has led to decreased energy usage -- but notably, a lot of those standards are around better insulation and more efficient components. Putting an extra layer of foam in your fridge or having sensors in your fridge that help regulate temperature definitely doesn't mean you've lost control of your life. But needing to download a firmware update to your Internet-enabled fridge that uses a Samsung account where you now can't access your grocery list until you finish the mandated update which changes your fridge's UI on its mobile app -- I think that means you've lost control of your life :)
      - fennecfoxy 3y ago
        Oh yeah for sure, but I think there are definitely reasons for it. The enshittification/technoshit that comes out of the iot world (and all other devices) because of corporate greed just ruins it even more. The whole signing up for a Samsung account thing etc for your fridge. Stuff like this really just needs to be legislated under some kind of "all technology should just work, locally and with one another with at least an agreed set of features" level. Apple should have been legally forced to use USB C (or whatever alternative was best) ages ago, even before the EU got to them. Apple were happy to use Wifi/Bluetooth/etc/etc standards yet still wanted to use other proprietary BS. Same goes for literally everything else: all technologies should work together using at least a common method (with say options for proprietary stuff) and iot/whatever should all work flawlessly locally without any account or internet connectivity (which should all be 100% optional). Devices should work flawlessly even if the company that produces them has shut down all servers and gone bankrupt. We need to force our governments to do this stuff for us.
- joshuaissac 3y ago
  No, but the firmware runs locally, instead of in someone else's cloud, so an outage in their cloud cannot take down my fridge.
  - phyrex 3y ago
    https://www.samsung.com/us/explore/family-hub-refrigerator/overview/ https://www.samsung.com/us/explore/family-hub-refrigerator/o...
    - danShumway 3y ago
      Nice, looks like we finally got around to inventing refrigerator magnets! ---- That is a little bit dismissive of me though. There are some cool features here: I can now "entertain in my kitchen", which is definitely a normal thing that normal people do. I love getting everyone together to crowd around my refrigerator so that we can all watch Game of Thrones. And I can use Amazon Alexa from my fridge just in case I'm not able to talk out loud to the cheap unobtrusive device that has a microphone in it specifically so that it can be placed in any room of the house. So having that option is good. And perhaps the biggest deal of all, I can finally "shop from home." That was a huge problem for me before, I kept thinking, "if only I had a better refrigerator I could finally buy things on websites." And this is a great bargain for only 3-5 thousand dollars! I can't believe I was planning to buy some crappy normal refrigerator for less than a thousand bucks and then use the extra money I saved to mount a giant flat-screen TV hooked up to a Chromecast in my kitchen. That would have been a huge mistake for me to make. Honestly it's just the icing on the cake that I can "set as many timers as [I] want." That's a great feature for someone like me because I can't set any timers at all using my phone or a voice assistant. /s ---- <serious>Holy crud, smart-device manufacturers have become unhinged. The one feature that actually looks useful here is being able to take a picture of the inside of the fridge while you're away. That is basically the one feature that I would want from a fridge that isn't much-better handled using a phone or a tablet or a TV or a normal refrigerator button. Which, great, but the problem is that I know what the inside of my fridge looks like right now, and let me just say: if I was organized enough that a photograph of the inside of my fridge would be clear enough to tell me what food was in it, and if I was organized enough that the photo wouldn't just show 'a pile of old containers, some of them transparent and some of them not' -- I have a feeling that in that case I would no longer be the type of person that needed to take a photo of the inside of my refrigerator to know what was in it.
- TeMPOraL 3y ago
  Why on Earth would a refrigerator need firmware?
  - dvaletin 3y ago
    To show you ads on a screen on a fancy door.

Philpax 3y ago

You can do that today depending on what you need from the models.

jstummbillig 3y ago
Well, for me that's still quite a bit more than the best ones provide, but I am sure we will get there.
benterix 3y ago
Let's say I'm writing Flask code all day, and I need help with various parts of my code. Can I do it today or not? With questions like, "How to add 'Log in with Google' to the login screen" etc.
- capableweb 3y ago
  In short: no. Longer: In theory, but it'll require a bunch of glue and using multiple models depending on the specific task you need help with. Some models are great at working with code but suck at literally anything else, so if you want it to be able to help you with "Do X with Y" you need to at least have two models, one that can reason up with an answer, and another to implement said answer. There is no general-purpose ("FOSS") LLM that even come close to GPT4 at this point.
- phyrex 3y ago
  There’s the code llama model that you can run locally I think which should be able to help you with that: https://ai.meta.com/blog/code-llama-large-language-model-coding/ https://ai.meta.com/blog/code-llama-large-language-model-cod...
- wokwokwok 3y ago
  If you have sufficiently good hardware, the 34B code llama model [1] (hint: pick the quantised model you can use based on “Max RAM required”, eg. q5/q6) running on llama.cpp [2], can answer many generic python and flask related questions, but it’s not quite good enough to generate entire code blocks for you like gpt4. It’s probably as good as you can get at the moment though; and hey, trying it out costs you nothing but the time it takes to download llama.cpp and run “make” and then point it at the q6 model file. So if it’s no good, you’ve probably wasted nothing more than like 30 min giving it a try. [1] - https://huggingface.co/TheBloke/CodeLlama-34B-Instruct-GGUF https://huggingface.co/TheBloke/CodeLlama-34B-Instruct-GGUF [2] - https://github.com/ggerganov/llama.cpp https://github.com/ggerganov/llama.cpp

m3kw9 3y ago

And then you will have to endure not using a model by OpenAI that is 10x better than a local one

cwkoss 3y ago
I think there is probably a threshold of usefulness, local LLMs are expensive to run but pretty close to it for most use cases now. In a couple years, our smartphones will probably be powerful enough to run LLMs locally that are good enough for 80% of uses.
grishka 3y ago
What is so special about OpenAI's cloud hardware that one can't build themselves a similar server to run AI models of similar size?
- objectivetruth 3y ago
  [dead]
- baq 3y ago
  Nothing stopping you from buying an H100 and putting it in your desktop. As for me, I’ve got other uses for $45k.
  - dcchambers 3y ago
    If you wanted to buy one...are you even able to order one? Aren't they massively backordered by big companies? Is there a black marker for H100 cards?
    - baq 3y ago
      I took the quote from eBay. Not sure which shade of black that would be ;)
- airspresso 3y ago
  The hardware is primarily standard Nvidia GPUs (A100s, H100s), but the scale of the infrastructure is on another level entirely. These models currently need clusters of GPU-powered servers to make predictions fast enough. Which explains why OpenAI partnered with Microsoft and got billions in funding to spend on compute. You can run (much) smaller LLM models on consumer-grade GPUs though. A single Nvidia GPU with 8 GB RAM is enough to get started with models like Zephyr, Mistral or Llama2 in their smallest versions (7B parameters). But it will be both slower and lower quality than anything OpenAI currently offers.

deleted 3y ago

[deleted]

boh 3y ago

You can do that today. Oobabooga + Hugging Face models.

nomel 3y ago
Having something that executes, and having something that's genuinely useful, are two different things. For my hand typed use case's, GPT-4 is the only acceptable model that doesn't leave me frustrated and angry at wasting time. For some automated stuff (converting text to json, etc), the local models are fine.