6 ms·

Small AI Models Gain Traction In places with unreliable networks

277 points by sscaryterry 4mo ago

bombcar 4mo ago
99% of the model "work" (meaning the connection to your computer) is just spinning a spinner - something that makes me want to wrap it with a mosh shell so I can just keep moving from network to network.
enoint 4mo ago
Fascinating to wonder whether the bigger model finds fewer or more counterfeits than the on-device one.
tim-fan 4mo ago
Is anyone making LLM-in-a-box for emergency supply kits yet? I feel that would be handy in all sorts of situations when networks are down.
- cdnsteve 4mo ago
  Can you expand what you mean?
  - wahnfrieden 4mo ago
    They want to ask the iOS Foundation model (frontier on device intelligence for something small) for instance about emergency procedures and life-saving info. I wouldn’t trust that model with much at all though. More likely to find what you need from miniature survival guides.
    - dofm 4mo ago
      > They want to ask the iOS Foundation model (frontier on device intelligence for something small) This is a bit of a straw man, TBH. For one thing, "LLM-in-a-box" doesn't necesssarily imply a device as small as a phone. For another, you'd need to convince people that the iOS Foundation model is the "frontier" of LLMs that run on phones when it is really not. AFAIK it is noticeably outperformed by the Gemma 4 E2B model and certainly the E4B. https://blog.google/innovation-and-ai/technology/developers-tools/quantization-aware-training-gemma-4/ https://blog.google/innovation-and-ai/technology/developers-... Here is a common-or-garden youtube video that includes a demonstration of how much better the E2B is: https://www.youtube.com/watch?v=sTxyBUbdZcA https://www.youtube.com/watch?v=sTxyBUbdZcA Whether this idea (LLMs for emergency/survival scenarios) has value, I don't know, so I am not offering an opinion, but you should approach it with a good faith argument. I am an LLM cynic but I suppose if I was to be without connectivity but with power for a while, a device with the Gemma 4 E2B or E4B model on it might be helpful or interesting to have. If such a device had the 12B QAT model on it, that really would cross the line to utility. Not sure it has value in the OP's scenario, still.
      - wahnfrieden 4mo ago
        I meant the new 20B param Foundation models, not the ones from last year. But sure Gemma might be better
- SwellJoe 4mo ago
  This is couched in prepper nonsense, but it's got LLM, WikiPedia, maps, etc. A bunch of genuinely useful stuff to keep on a USB stick or whatever: https://www.projectnomad.us/ https://www.projectnomad.us/ But, the current model you really want for an emergency kit is Gemma 4 12B QAT 4-bit. At ~7GB on disk, it's small enough to run on a tablet or any modern computer, slowly if you don't have a GPU or modern Apple silicon, but exceedingly smart for its size, excellent vision capabilities, good tool user, surprisingly good reasoning.
  - dofm 4mo ago
    The 12B QAT model is really overlooked because the tech industry has been so desperate for the LLM bet to play out to "product market fit" (which means please IPO now) that it has become convinced that coding models are the only things that matter.
- Terr_ 4mo ago
  > LLM-in-a-box for emergency For most actual emergency scenarios, a device that focuses on storage of large amounts of prepared normal reference material [0] will be wayyyyy cheaper, more durable, portable, and able to run on batteries or being constantly plugged into a somehow-still-normal electrical grid. (Think an e-ink tablet that can run off a 5V battery pack buffering a literal handcrank.) In contrast, imagine spending the money to build a beefy LLM-running computer with good GPU/RAM, and somehow mothballing it (to depreciate, unused) in a "safe" location for the big earthquake/flood/etc... Then when the disaster strikes and you dig it out, how will you power it when you need it, and for long enough to do anything useful? Even if wall-current civilization is 20 miles away on the other side of the mountain, are you going to carry it on your back, or are you going to carry food and water to live? If you do drag it there, are they going to let you run it when it cuts into light for surgery or heat to sterilize drinking water?
  - skybrian 4mo ago
    You will probably want a search engine though. Perhaps a small LLM would work well as a component for that?
    - rtpg 4mo ago
      grep works well!
    - iamflimflam1 4mo ago
      You may benefit from an embedding approach for semantic search. Not sure what an LLM would give you on top of that.
      - crsx_ 4mo ago
        If the emergency is in a foreign country, being able to communicate with locals would likely be a benefit - and a domain specific trained model could translate better than general purpose translators. In general, I think speech as input/output is under-explored. In the emergency scenario, in a stressful environment, having an expert in your ear you can talk to should work much better than having a big manual book to look up specific cases.
- vessenes 4mo ago
  I've been mulling over a good use of a large philanthropy spend in the next decade, and I would love to build a bunch of hardware "oracles" that include an LLM. Ideally solid state, visual/audio, solar + usb-c, so, good in a lot of doomsday scenarios as well as just out hiking. It's a fun thought experiment. I imagine making like 1 million of them, they could be sold and genuinely useful, but also given away; once owned, you could use them, or store and put in an emergency box, bury next to the 10k year clock.. a lot of possibilities.
  - adrianN 4mo ago
    I feel like you could get a lot more quality of life improvement for more people with the money if you spent it on low tech solutions, eg more efficient cooking stoves for people still cooking with biomass, or solar microgrids for areas without electricity.
    - vessenes 4mo ago
      No doubt there are a hundred more direct things one could do. On the other hand this would include instructions for all those things and could advise on the building! I think it would be a nice thing to have in the world.
- rasz 4mo ago
  Smallest local model able to work with offline wikipedia dump would be one step above just having an offline wikipedia dump.
- burgerone 4mo ago
  This is HN, not Reddit.
- sumitkumar 4mo ago
  electricity outage and battery running out is the end game for any real prolonged external emergency. Internet connection is just the soft edge.
- robotswantdata 4mo ago
  https://developers.google.com/edge/gallery https://developers.google.com/edge/gallery Put that on a spare phone
- bluerooibos 4mo ago
  > Is anyone making LLM-in-a-box for emergency supply kits yet? Maybe someone should be making this, but for rebuilding society in the event of a disaster - a solar-powered black box with most of humanity's knowledge within. Even something running one of the Qwen models would be useful. "So, we had a nuclear war and need to start from scratch. How do I turn this rock into a computer chip?"
  - bluGill 4mo ago
    If you are rebuilding society most of this knowledge is useless for centuries. You don't have enough labor to build and maintain factories. You will spend centuries in the hunter gather phase struggling to survive, while slowly building agriculture. You will be lucky if you can teach your grandkids to read - since that will be a useless skill. Print important knowledge on paper and store it in a desert. in 2000 years society and population will advance enough to get a jump start based on our knowledge.
    - jonbodner 4mo ago
      There is no rebuilding society because of energy. All of the oil and coal that could be extracted by a civilization that only has wood for fuel has been used. If we go back to the Stone Age, we aren’t returning to the present. If we’re lucky, we will get to the Middle Ages.
      - bluGill 4mo ago
        There is still a lot of coal and oil to be had. It isn't as easy as it used to be, but there is still a lot. If we get to the middle ages then society can build a lot with knowledge. More efficient iron production than what the middle ages (when still using wood) had for example would be useful, and our knowledge of technology may allow bicycles sooner (bicycles need higher technology than a steam engine)
- tristor 4mo ago
  I don't if anyone is doing this yet, but I think a small LLM w/ data sets for RAG reachable via APRS and LoRa would be very useful, not just as an individual but for the community around you.
- RetroTechie 4mo ago
  Oh my... I can think of a 101 things more useful in actual emergencies than an LLM-in-a-box. Unless you have a weird definition of "emergency" (if so: please define).
- theshrike79 4mo ago
  Project NOMAD: https://andrew.ooo/posts/project-nomad-offline-survival-computer/ https://andrew.ooo/posts/project-nomad-offline-survival-comp... I think it was on HN at some point too. Basically it's a bunch of offline data dumps (wikipedia, maps etc) with a local AI model on top that you can use to query them with natural language.
jdonaldson 4mo ago
I think neuro-symbolic AI has a lot of potential here, since small models can handle a lot of conversational inputs, while relying on wired-in solvers for more complex symbolic math/computation needs. https://jjd.io/posts/swollm-bbh-leaderboard.html https://jjd.io/posts/swollm-bbh-leaderboard.html
bix6 4mo ago
Has anyone used the Rx Scanner mentioned in the opening? https://rxall.net/rxscanner/ https://rxall.net/rxscanner/
mountainriver 4mo ago
I looked into this a bit but unfortunately because of starlink most of this won’t be needed
N_Lens 4mo ago
I strongly believe this premise in the article is correct - we will see a lot of tiny, hyper specialized models for individual tasks, and perhaps that will converge with an orchestration layer for a generalized intelligence that controls these specialized tiny models, that will be quite capable. I don't foresee AGI arising out training bigger LLMs (Though investors won't realise that for a while yet). It's actually how organic brains work - specialized tasks are offloaded to local cortical columns. The overall coordination between these sub-brains creates emergent skills/abilities.
- looofooo0 4mo ago
  What about recent models providing correct proofs to open math problems?
  - 16bitvoid 4mo ago
    What about it?
  - TJSomething 4mo ago
    I haven't tried it, but I saw Leanstral, an LLM specialized in writing Lean proofs, posted on HN recently and it claims to outperform some larger general purpose models. It didn't beat Claude Opus, but it seems to do decently at one tenth the cost. It's plausible that further research could yield other models that are smaller and more effective at limited tasks, reversing the trend of ever growing models.
- stingraycharles 4mo ago
  > It's actually how organic brains work - specialized tasks are offloaded to local cortical columns. How are small isolated language models more similar to that than MoE in LLMs?
  - simianwords 4mo ago
    Right MoE is a tradeoff between efficiency and intelligence.
tonysbuildsx 4mo ago
[flagged]
fpauser 4mo ago
SLMs for the rescue!
monkeydust 4mo ago
Where is a good place to start with training SLM these days if you don't have the compute locally?
- calgoo 4mo ago
  I rent cheap GPU based instances by the hour and run on those. Nothing fancy, but $20 can get you a decent amount of compute on a A6000 or H100.
egormakarov 4mo ago
Can't wait to be killed by my toaster because some sexy mossad agent seduced it.
- a96 4mo ago
  Have you ever tried to indulge an all-consuming urge to kill when you don't have opposable thumbs? Or hands? Or anything other than a bread slot?
  - masfuerte 4mo ago
    Fire!
- sscaryterry 4mo ago
  This made me laugh out loud :)
tauridev 4mo ago
[flagged]
dmezzetti 4mo ago
100% agree on this. I've been working on small local models for years with txtai (https://github.com/neuml/txtai https://github.com/neuml/txtai). I've published close to 100 models that can run local for RAG, Agents, Vector Search and more (https://huggingface.co/NeuML/collections https://huggingface.co/NeuML/collections).
HelloUsername 4mo ago
What about small (offline) AI Models in places with weak hardware?
Pakvothe 4mo ago
[flagged]
littlerobot 4mo ago
[dead]
prmph 4mo ago
> The RxScanner is a handheld spectrometer that scans a pill with infrared light, then sends the item’s molecular profile to an AI model equipped with a pharmaceutical database. In seconds, the AI identifies the medication from its molecular profile—or reports that it’s phony. Is every tech, including database search "AI" now?
rnagulapalle 4mo ago
[flagged]

277 points by sscaryterry 4mo ago

99% of the model "work" (meaning the connection to your computer) is just spinning a spinner - something that makes me want to wrap it with a mosh shell so I can just keep moving from network to network.

enoint 4mo ago

Fascinating to wonder whether the bigger model finds fewer or more counterfeits than the on-device one.

tim-fan 4mo ago

Is anyone making LLM-in-a-box for emergency supply kits yet? I feel that would be handy in all sorts of situations when networks are down.

cdnsteve 4mo ago
Can you expand what you mean?
- wahnfrieden 4mo ago
  They want to ask the iOS Foundation model (frontier on device intelligence for something small) for instance about emergency procedures and life-saving info. I wouldn’t trust that model with much at all though. More likely to find what you need from miniature survival guides.
  - dofm 4mo ago
    > They want to ask the iOS Foundation model (frontier on device intelligence for something small) This is a bit of a straw man, TBH. For one thing, "LLM-in-a-box" doesn't necesssarily imply a device as small as a phone. For another, you'd need to convince people that the iOS Foundation model is the "frontier" of LLMs that run on phones when it is really not. AFAIK it is noticeably outperformed by the Gemma 4 E2B model and certainly the E4B. https://blog.google/innovation-and-ai/technology/developers-tools/quantization-aware-training-gemma-4/ https://blog.google/innovation-and-ai/technology/developers-... Here is a common-or-garden youtube video that includes a demonstration of how much better the E2B is: https://www.youtube.com/watch?v=sTxyBUbdZcA https://www.youtube.com/watch?v=sTxyBUbdZcA Whether this idea (LLMs for emergency/survival scenarios) has value, I don't know, so I am not offering an opinion, but you should approach it with a good faith argument. I am an LLM cynic but I suppose if I was to be without connectivity but with power for a while, a device with the Gemma 4 E2B or E4B model on it might be helpful or interesting to have. If such a device had the 12B QAT model on it, that really would cross the line to utility. Not sure it has value in the OP's scenario, still.
    - wahnfrieden 4mo ago
      I meant the new 20B param Foundation models, not the ones from last year. But sure Gemma might be better
SwellJoe 4mo ago
This is couched in prepper nonsense, but it's got LLM, WikiPedia, maps, etc. A bunch of genuinely useful stuff to keep on a USB stick or whatever: https://www.projectnomad.us/ https://www.projectnomad.us/ But, the current model you really want for an emergency kit is Gemma 4 12B QAT 4-bit. At ~7GB on disk, it's small enough to run on a tablet or any modern computer, slowly if you don't have a GPU or modern Apple silicon, but exceedingly smart for its size, excellent vision capabilities, good tool user, surprisingly good reasoning.
- dofm 4mo ago
  The 12B QAT model is really overlooked because the tech industry has been so desperate for the LLM bet to play out to "product market fit" (which means please IPO now) that it has become convinced that coding models are the only things that matter.
Terr_ 4mo ago
> LLM-in-a-box for emergency For most actual emergency scenarios, a device that focuses on storage of large amounts of prepared normal reference material [0] will be wayyyyy cheaper, more durable, portable, and able to run on batteries or being constantly plugged into a somehow-still-normal electrical grid. (Think an e-ink tablet that can run off a 5V battery pack buffering a literal handcrank.) In contrast, imagine spending the money to build a beefy LLM-running computer with good GPU/RAM, and somehow mothballing it (to depreciate, unused) in a "safe" location for the big earthquake/flood/etc... Then when the disaster strikes and you dig it out, how will you power it when you need it, and for long enough to do anything useful? Even if wall-current civilization is 20 miles away on the other side of the mountain, are you going to carry it on your back, or are you going to carry food and water to live? If you do drag it there, are they going to let you run it when it cuts into light for surgery or heat to sterilize drinking water?
- skybrian 4mo ago
  You will probably want a search engine though. Perhaps a small LLM would work well as a component for that?
  - rtpg 4mo ago
    grep works well!
  - iamflimflam1 4mo ago
    You may benefit from an embedding approach for semantic search. Not sure what an LLM would give you on top of that.
    - crsx_ 4mo ago
      If the emergency is in a foreign country, being able to communicate with locals would likely be a benefit - and a domain specific trained model could translate better than general purpose translators. In general, I think speech as input/output is under-explored. In the emergency scenario, in a stressful environment, having an expert in your ear you can talk to should work much better than having a big manual book to look up specific cases.
vessenes 4mo ago
I've been mulling over a good use of a large philanthropy spend in the next decade, and I would love to build a bunch of hardware "oracles" that include an LLM. Ideally solid state, visual/audio, solar + usb-c, so, good in a lot of doomsday scenarios as well as just out hiking. It's a fun thought experiment. I imagine making like 1 million of them, they could be sold and genuinely useful, but also given away; once owned, you could use them, or store and put in an emergency box, bury next to the 10k year clock.. a lot of possibilities.
- adrianN 4mo ago
  I feel like you could get a lot more quality of life improvement for more people with the money if you spent it on low tech solutions, eg more efficient cooking stoves for people still cooking with biomass, or solar microgrids for areas without electricity.
  - vessenes 4mo ago
    No doubt there are a hundred more direct things one could do. On the other hand this would include instructions for all those things and could advise on the building! I think it would be a nice thing to have in the world.
rasz 4mo ago
Smallest local model able to work with offline wikipedia dump would be one step above just having an offline wikipedia dump.
burgerone 4mo ago
This is HN, not Reddit.
sumitkumar 4mo ago
electricity outage and battery running out is the end game for any real prolonged external emergency. Internet connection is just the soft edge.
robotswantdata 4mo ago
https://developers.google.com/edge/gallery https://developers.google.com/edge/gallery Put that on a spare phone
bluerooibos 4mo ago
> Is anyone making LLM-in-a-box for emergency supply kits yet? Maybe someone should be making this, but for rebuilding society in the event of a disaster - a solar-powered black box with most of humanity's knowledge within. Even something running one of the Qwen models would be useful. "So, we had a nuclear war and need to start from scratch. How do I turn this rock into a computer chip?"
- bluGill 4mo ago
  If you are rebuilding society most of this knowledge is useless for centuries. You don't have enough labor to build and maintain factories. You will spend centuries in the hunter gather phase struggling to survive, while slowly building agriculture. You will be lucky if you can teach your grandkids to read - since that will be a useless skill. Print important knowledge on paper and store it in a desert. in 2000 years society and population will advance enough to get a jump start based on our knowledge.
  - jonbodner 4mo ago
    There is no rebuilding society because of energy. All of the oil and coal that could be extracted by a civilization that only has wood for fuel has been used. If we go back to the Stone Age, we aren’t returning to the present. If we’re lucky, we will get to the Middle Ages.
    - bluGill 4mo ago
      There is still a lot of coal and oil to be had. It isn't as easy as it used to be, but there is still a lot. If we get to the middle ages then society can build a lot with knowledge. More efficient iron production than what the middle ages (when still using wood) had for example would be useful, and our knowledge of technology may allow bicycles sooner (bicycles need higher technology than a steam engine)
tristor 4mo ago
I don't if anyone is doing this yet, but I think a small LLM w/ data sets for RAG reachable via APRS and LoRa would be very useful, not just as an individual but for the community around you.
RetroTechie 4mo ago
Oh my... I can think of a 101 things more useful in actual emergencies than an LLM-in-a-box. Unless you have a weird definition of "emergency" (if so: please define).
theshrike79 4mo ago
Project NOMAD: https://andrew.ooo/posts/project-nomad-offline-survival-computer/ https://andrew.ooo/posts/project-nomad-offline-survival-comp... I think it was on HN at some point too. Basically it's a bunch of offline data dumps (wikipedia, maps etc) with a local AI model on top that you can use to query them with natural language.

jdonaldson 4mo ago

I think neuro-symbolic AI has a lot of potential here, since small models can handle a lot of conversational inputs, while relying on wired-in solvers for more complex symbolic math/computation needs. https://jjd.io/posts/swollm-bbh-leaderboard.html https://jjd.io/posts/swollm-bbh-leaderboard.html

bix6 4mo ago

Has anyone used the Rx Scanner mentioned in the opening? https://rxall.net/rxscanner/ https://rxall.net/rxscanner/

mountainriver 4mo ago

I looked into this a bit but unfortunately because of starlink most of this won’t be needed

N_Lens 4mo ago

I strongly believe this premise in the article is correct - we will see a lot of tiny, hyper specialized models for individual tasks, and perhaps that will converge with an orchestration layer for a generalized intelligence that controls these specialized tiny models, that will be quite capable. I don't foresee AGI arising out training bigger LLMs (Though investors won't realise that for a while yet). It's actually how organic brains work - specialized tasks are offloaded to local cortical columns. The overall coordination between these sub-brains creates emergent skills/abilities.

looofooo0 4mo ago
What about recent models providing correct proofs to open math problems?
- 16bitvoid 4mo ago
  What about it?
- TJSomething 4mo ago
  I haven't tried it, but I saw Leanstral, an LLM specialized in writing Lean proofs, posted on HN recently and it claims to outperform some larger general purpose models. It didn't beat Claude Opus, but it seems to do decently at one tenth the cost. It's plausible that further research could yield other models that are smaller and more effective at limited tasks, reversing the trend of ever growing models.
stingraycharles 4mo ago
> It's actually how organic brains work - specialized tasks are offloaded to local cortical columns. How are small isolated language models more similar to that than MoE in LLMs?
- simianwords 4mo ago
  Right MoE is a tradeoff between efficiency and intelligence.

tonysbuildsx 4mo ago

[flagged]

fpauser 4mo ago

SLMs for the rescue!

monkeydust 4mo ago

Where is a good place to start with training SLM these days if you don't have the compute locally?

calgoo 4mo ago
I rent cheap GPU based instances by the hour and run on those. Nothing fancy, but $20 can get you a decent amount of compute on a A6000 or H100.

egormakarov 4mo ago

Can't wait to be killed by my toaster because some sexy mossad agent seduced it.

a96 4mo ago
Have you ever tried to indulge an all-consuming urge to kill when you don't have opposable thumbs? Or hands? Or anything other than a bread slot?
- masfuerte 4mo ago
  Fire!
sscaryterry 4mo ago
This made me laugh out loud :)

tauridev 4mo ago

[flagged]

dmezzetti 4mo ago

100% agree on this. I've been working on small local models for years with txtai (https://github.com/neuml/txtai https://github.com/neuml/txtai). I've published close to 100 models that can run local for RAG, Agents, Vector Search and more (https://huggingface.co/NeuML/collections https://huggingface.co/NeuML/collections).

HelloUsername 4mo ago

What about small (offline) AI Models in places with weak hardware?

Pakvothe 4mo ago

[flagged]

littlerobot 4mo ago

[dead]

prmph 4mo ago

> The RxScanner is a handheld spectrometer that scans a pill with infrared light, then sends the item’s molecular profile to an AI model equipped with a pharmaceutical database. In seconds, the AI identifies the medication from its molecular profile—or reports that it’s phony. Is every tech, including database search "AI" now?

rnagulapalle 4mo ago

[flagged]