4 ms·

So, its not a general language model, focused on tool call strictly for tiny edge-devices. There are solutions everywhere for high-capacity devices, Needle is f

So, its not a general language model, focused on tool call strictly for tiny edge-devices. There are solutions everywhere for high-capacity devices, Needle is for sub-$200 devices.

fwipsy 2mo ago
14mb? More like sub-$20 devices.
- TomatoCo 2mo ago
  Most pi pico's come with 16mb of flash. I wonder what kind of performance that can eek out.
  - SequoiaHope 2mo ago
    Well running from QSPI flash (even the internal memory versions use SPI internally) so any inference would be very slow streaming from that compared to RAM. The featured article says: “With a peak session RAM around 28MB, Needle runs on newer microcontrollers like ESP32-S3.” So I don’t see this doing anything useful on a Pico. The Pico 2 (RP2350) for example has 520k of RAM.
    - Rohansi 2mo ago
      An ESP32 has the same amount of SRAM as the Pi Pico. You can hook up PSRAM to the Pi Pico just like ESP32 to get 16MB more RAM.
      - SequoiaHope 2mo ago
        [dead]
anon373839 2mo ago
It seems to me that the model struggles to have enough general intelligence, knowledge, or reasoning capacity for arbitrary prompted tool calling. At this size, not surprising. I am VERY interested in seeing how it could perform with some fine-tuning for a specific family of tools/tasks. That would be a great addition to the demo.
- electroglyph 2mo ago
  i would assume a model this size would require finetuning tbh. even functiongemma recommends that.
- londons_explore 2mo ago
  It also seems to have far more tokens per second than needed for general "close the blinds" "tool_call(blinds, CLOSED)". I do wonder if more smartness could be had by using sparser experts.... And possibly even having some kind of expert switching penalty to try to reduce the amount of data read from read only flash memory by encouraging subsequent tokens to use already loaded experts.
- yunbiao 2mo ago
  [flagged]

So, its not a general language model, focused on tool call strictly for tiny edge-devices. There are solutions everywhere for high-capacity devices, Needle is for sub-$200 devices.

fwipsy 2mo ago

14mb? More like sub-$20 devices.

TomatoCo 2mo ago
Most pi pico's come with 16mb of flash. I wonder what kind of performance that can eek out.
- SequoiaHope 2mo ago
  Well running from QSPI flash (even the internal memory versions use SPI internally) so any inference would be very slow streaming from that compared to RAM. The featured article says: “With a peak session RAM around 28MB, Needle runs on newer microcontrollers like ESP32-S3.” So I don’t see this doing anything useful on a Pico. The Pico 2 (RP2350) for example has 520k of RAM.
  - Rohansi 2mo ago
    An ESP32 has the same amount of SRAM as the Pi Pico. You can hook up PSRAM to the Pi Pico just like ESP32 to get 16MB more RAM.
    - SequoiaHope 2mo ago
      [dead]

anon373839 2mo ago

It seems to me that the model struggles to have enough general intelligence, knowledge, or reasoning capacity for arbitrary prompted tool calling. At this size, not surprising. I am VERY interested in seeing how it could perform with some fine-tuning for a specific family of tools/tasks. That would be a great addition to the demo.

electroglyph 2mo ago
i would assume a model this size would require finetuning tbh. even functiongemma recommends that.
londons_explore 2mo ago
It also seems to have far more tokens per second than needed for general "close the blinds" "tool_call(blinds, CLOSED)". I do wonder if more smartness could be had by using sparser experts.... And possibly even having some kind of expert switching penalty to try to reduce the amount of data read from read only flash memory by encouraging subsequent tokens to use already loaded experts.
yunbiao 2mo ago
[flagged]