4 ms·

I wish this could be self hosted or at least e2e encrypted in a way where our private data couldn't be leaked somehow.

• transfire 4y ago
  Agree. I wonder how well it would handle mechanical design drawings. If it could, it might be very helpful in my industry, but our drawings have strict privacy restrictions. Eventually I hope we’ll be able to able get a single high end server appliance to run an instance in house for such use cases.
• wsgeorge 4y ago
  Even better, run locally. That should be the goal.
  • barbazoo 4y ago
    Is this realistic though anytime soon? How long until this can be self hosted and how long after that until it'll run on a laptop or phone? Granted I know nothing about the ML hardware that's in Apple hardware and even Google phones so maybe we're closer than I imagine. Do you know if that hardware is capable of running LLM or is that optimized for something else?
    • wsgeorge 4y ago
      > Do you know if that hardware is capable of running LLM or is that optimized for something else? I do! Checkout llama.cpp [0] and related projects. The game changer with this was Gerganov's ggml format that makes inference on CPUs realistic. I'm currently running alpaca.cpp and llama.cpp, using 7B models 4-bit quantized (so they're the lowest powered, lightest weight models) on an M1 Pro with 16gig RAM. You could run same on an M1 Macbook Air with 8gig RAM. Gerganov shows the Whisper model running on an Android phone. [0]https://github.com/ggerganov/llama.cpp https://github.com/ggerganov/llama.cpp
• fortyseven 4y ago
  Free, open source alternatives are in development. Or so my alpaca tells me.