4 ms·

"LLM backends: Anthropic, OpenAI, OpenRouter." And here I was hoping that this was local inference :)

by throwa356262 8mo ago

deleted 8mo ago
[deleted]
micw 8mo ago
Sure. Why purchase a H200 if you can go with an ESP32 ^^
- sigmoid10 8mo ago
  Blowing more than 800kb on essentially an http api wrapper is actually kinda bad. The original Doom binary was 700kb and had vastly more complexity. This is in C after all, so by stripping out nonessential stuff and using the right compiler options, I'd expect something like this to come in under 100kb.
  - __tnm 8mo ago
    yeah i sandbagged the size just a little to start (small enough to fit on the c3, 888 picked for good luck & prosperity; I even have a build that pads to get 888 exactly), so i can now try reduce some of it as an exercise etc. but 100kb you’re not gonna see :) this has WiFi, tls, etc. doom didn’t need those
  - pitched 8mo ago
    Doom had the benefit of an OS that included a lot of low-level bits like a net stack. This doesn’t! That 800kB includes everything it would need from an OS too.
    - __tnm 8mo ago
      yah my back of the envelope math.. the “app logic”/wrapper pieces come out to about 25kb WiFi is 350 Tls is 120 and certs are 90!
    - oso2k 8mo ago
      Maybe you’re misremembering or referring to Doom (2016). The original Doom was developed for DOS and id had to build a lot of its own network stack. BSD style socket based networking wasn’t a given in DOS. Still, zclaw is an impressive achievement.
  - epcoa 8mo ago
    > vastly more complexity. Doom is ingenious, but it is not terribly complex IMHO, not compared to a modern networking stack including WiFi driver. The Doom renderer charm is in its overall simplicity. The AI is effective but not sophisticated.
  - madduci 8mo ago
    The whole ESP32 Libraries are kind of bloated. To enable Bluetooth, WiFi or HTTP handling, you need to embed some large libraries
peterisza 8mo ago
right, 888 kB would be impossible for local inference however, it is really not that impressive for just a client
- Dylan16807 8mo ago
  It's not completely impossible, depending on what your expectations are. That language model that was built out of redstone in minecraft had... looks like 5 million parameters. And it could do mostly coherent sentences.
  - godelski 8mo ago
    > built out of redstone in minecraft Ummm... > 5 million parameters Which is a lot more than 888kb... Supposing your ESP32 could use qint8 (LOL) that's still 1 byte per parameter and the k in kb stands for thousand, not million.
    - Dylan16807 8mo ago
      https://www.youtube.com/watch?v=VaeI9YgE1o8 https://www.youtube.com/watch?v=VaeI9YgE1o8 Yes I know how much a kilobyte is. But cutting down to 2 million 3 bit parameters or something like that would definitely be possible. And a 32 bit processor should be able to pack and unpack parameters just fine. Edit: Hey look what I just found https://github.com/DaveBben/esp32-llm https://github.com/DaveBben/esp32-llm "a 260K parameter tinyllamas checkpoint trained on the tiny stories dataset"
      - godelski 8mo ago
        > But cutting down to 2 million 3 bit parameters or something like that would definitely be possible. Sure, but there's no free lunch > Hey look what I just found I've even personally built smaller "L"LMs. The first L is in quotes because it really isn't large (So maybe lLM?) and they aren't anything like what you'd expect and certainly not what the parent was looking for. The utility of them is really not that high... (there are special cases though) Can you "do" it? Yeah. I mean you can make a machine learning model of essentially arbitrary size. Will it be useful? Obviously that's not guaranteed. Is it fun? Yes. Is it great for leaning? Also yes. And remember, Tiny Stories is 1GB of data. Can you train it for longer and with more data? Again, certainly, BUT again, there are costs. That Minecraft one is far more powerful than this thing. Also, remember that these models are not RLHF'd, so you really shouldn't expect it to act like you're expecting a LLM to work. It is only at stage 0, the "pre-training", or what Karpathy calls a "babbler".
__tnm 8mo ago
haha well I got something ridiculous coming soon for zclaw that will kinda work on board.. will require the S3 variant tho, needs a little more memory. Training it later today.
- throwa356262 8mo ago
  Sounds interesting, please keep us posted. I dont think I have an S3, but plenty of C3. I thought they had the same amount of memory

"LLM backends: Anthropic, OpenAI, OpenRouter." And here I was hoping that this was local inference :)

deleted 8mo ago

[deleted]

micw 8mo ago

Sure. Why purchase a H200 if you can go with an ESP32 ^^

sigmoid10 8mo ago
Blowing more than 800kb on essentially an http api wrapper is actually kinda bad. The original Doom binary was 700kb and had vastly more complexity. This is in C after all, so by stripping out nonessential stuff and using the right compiler options, I'd expect something like this to come in under 100kb.
- __tnm 8mo ago
  yeah i sandbagged the size just a little to start (small enough to fit on the c3, 888 picked for good luck & prosperity; I even have a build that pads to get 888 exactly), so i can now try reduce some of it as an exercise etc. but 100kb you’re not gonna see :) this has WiFi, tls, etc. doom didn’t need those
- pitched 8mo ago
  Doom had the benefit of an OS that included a lot of low-level bits like a net stack. This doesn’t! That 800kB includes everything it would need from an OS too.
  - __tnm 8mo ago
    yah my back of the envelope math.. the “app logic”/wrapper pieces come out to about 25kb WiFi is 350 Tls is 120 and certs are 90!
  - oso2k 8mo ago
    Maybe you’re misremembering or referring to Doom (2016). The original Doom was developed for DOS and id had to build a lot of its own network stack. BSD style socket based networking wasn’t a given in DOS. Still, zclaw is an impressive achievement.
- epcoa 8mo ago
  > vastly more complexity. Doom is ingenious, but it is not terribly complex IMHO, not compared to a modern networking stack including WiFi driver. The Doom renderer charm is in its overall simplicity. The AI is effective but not sophisticated.
- madduci 8mo ago
  The whole ESP32 Libraries are kind of bloated. To enable Bluetooth, WiFi or HTTP handling, you need to embed some large libraries

peterisza 8mo ago

right, 888 kB would be impossible for local inference however, it is really not that impressive for just a client

Dylan16807 8mo ago
It's not completely impossible, depending on what your expectations are. That language model that was built out of redstone in minecraft had... looks like 5 million parameters. And it could do mostly coherent sentences.
- godelski 8mo ago
  > built out of redstone in minecraft Ummm... > 5 million parameters Which is a lot more than 888kb... Supposing your ESP32 could use qint8 (LOL) that's still 1 byte per parameter and the k in kb stands for thousand, not million.
  - Dylan16807 8mo ago
    https://www.youtube.com/watch?v=VaeI9YgE1o8 https://www.youtube.com/watch?v=VaeI9YgE1o8 Yes I know how much a kilobyte is. But cutting down to 2 million 3 bit parameters or something like that would definitely be possible. And a 32 bit processor should be able to pack and unpack parameters just fine. Edit: Hey look what I just found https://github.com/DaveBben/esp32-llm https://github.com/DaveBben/esp32-llm "a 260K parameter tinyllamas checkpoint trained on the tiny stories dataset"
    - godelski 8mo ago
      > But cutting down to 2 million 3 bit parameters or something like that would definitely be possible. Sure, but there's no free lunch > Hey look what I just found I've even personally built smaller "L"LMs. The first L is in quotes because it really isn't large (So maybe lLM?) and they aren't anything like what you'd expect and certainly not what the parent was looking for. The utility of them is really not that high... (there are special cases though) Can you "do" it? Yeah. I mean you can make a machine learning model of essentially arbitrary size. Will it be useful? Obviously that's not guaranteed. Is it fun? Yes. Is it great for leaning? Also yes. And remember, Tiny Stories is 1GB of data. Can you train it for longer and with more data? Again, certainly, BUT again, there are costs. That Minecraft one is far more powerful than this thing. Also, remember that these models are not RLHF'd, so you really shouldn't expect it to act like you're expecting a LLM to work. It is only at stage 0, the "pre-training", or what Karpathy calls a "babbler".

__tnm 8mo ago

haha well I got something ridiculous coming soon for zclaw that will kinda work on board.. will require the S3 variant tho, needs a little more memory. Training it later today.

throwa356262 8mo ago
Sounds interesting, please keep us posted. I dont think I have an S3, but plenty of C3. I thought they had the same amount of memory