4 ms·

I cannot really wrap my head around what is this. Is this the whole model? Is this some sort of middleware between user and OpenAI?

sp332 3y ago
"Nitro is a drop-in replacement for OpenAI's REST API" It's not a model, it puts an API on local models.
- adaboese 3y ago
  so I still need to download and host models myself. I found that to be incredibly impractical. I tried to do it for my project AIMD, but the cost and quality just made absolutely no sense even with the top models.
  - bbor 3y ago
    Well, the market for local inference is already quite large, to say the least. “It didn’t pencil out in my business favor” doesn’t seem like a fair criticism, especially for an app clearly focused on the hobbyist—>SMB market where compute costs are dwarfed by the costs of wages and increased mental load. I definitely see your specific point tho, and have found the same for high-level usecases. Local models become really useful when you need smaller models for ensemble systems, to give one class of use case you might want to try out —- e.g. proofreading, simple summarization, tone detection, etc.
sho_hn 3y ago
It's someone taking llama.cpp, strapping it to a HTTP server lib and implementing the OpenAI REST API, then putting some effort into a shiny website and docs, because AI-anything is a reputation and/or investment magnet right now. Not that it isn't a nice idea -- making it easy to test existing OpenAI-based apps against competing open models is a pretty good thing.
- toddmorey 3y ago
  Or… it’s just someone who wanted to build cool stuff. I see the value in being able to locally run apps designed to talk to OpenAI apis.
  - brucethemoose2 3y ago
    There are dozens of other endpoints that will do this, and possibly do it better. I'm all for cool innovations, but... I'm not really seeing the advantage here beyond the shiny website to attract VCs.
gmerc 3y ago
Our you could use something like omnitool (https://github.com/omnitool-ai/omnitool https://github.com/omnitool-ai/omnitool) and interface with both cloud and local AI, not limited to llms.

sp332 3y ago

"Nitro is a drop-in replacement for OpenAI's REST API" It's not a model, it puts an API on local models.

adaboese 3y ago
so I still need to download and host models myself. I found that to be incredibly impractical. I tried to do it for my project AIMD, but the cost and quality just made absolutely no sense even with the top models.
- bbor 3y ago
  Well, the market for local inference is already quite large, to say the least. “It didn’t pencil out in my business favor” doesn’t seem like a fair criticism, especially for an app clearly focused on the hobbyist—>SMB market where compute costs are dwarfed by the costs of wages and increased mental load. I definitely see your specific point tho, and have found the same for high-level usecases. Local models become really useful when you need smaller models for ensemble systems, to give one class of use case you might want to try out —- e.g. proofreading, simple summarization, tone detection, etc.

sho_hn 3y ago

It's someone taking llama.cpp, strapping it to a HTTP server lib and implementing the OpenAI REST API, then putting some effort into a shiny website and docs, because AI-anything is a reputation and/or investment magnet right now. Not that it isn't a nice idea -- making it easy to test existing OpenAI-based apps against competing open models is a pretty good thing.

toddmorey 3y ago
Or… it’s just someone who wanted to build cool stuff. I see the value in being able to locally run apps designed to talk to OpenAI apis.
- brucethemoose2 3y ago
  There are dozens of other endpoints that will do this, and possibly do it better. I'm all for cool innovations, but... I'm not really seeing the advantage here beyond the shiny website to attract VCs.

gmerc 3y ago

Our you could use something like omnitool (https://github.com/omnitool-ai/omnitool https://github.com/omnitool-ai/omnitool) and interface with both cloud and local AI, not limited to llms.