5 ms·

This is a really interesting question. I think there's definitely a world for both deployment models. Maybe a good analogy is database engines: both SQLite (a l

This is a really interesting question. I think there's definitely a world for both deployment models. Maybe a good analogy is database engines: both SQLite (a library) and Postgres (a long-running service) have widespread use cases with tradeoffs.

jameshart 3y ago
But these are typically filling the usecases of productivity applications, not ‘engines’. Microsoft Word doesn’t run its grammar checker as an external service and shunt JSON over a localhost socket to get spelling and style suggestions. Photoshop doesn’t install a background service to host filters. The closest pattern I can think of is the ‘language servers’ model used by IDEs to handle autosuggest - see https://microsoft.github.io/language-server-protocol/ https://microsoft.github.io/language-server-protocol/ - but the point of that is to enable many to many interop - multiple languages supporting multiple IDEs. Is that the expected usecase for local language assistants and image generators?
- bri3d 3y ago
  Funny choice of example. You’ve always been able to use Word as a remote spellchecker over COM, and as of Windows 8, spellchecking is available system wide and runs in a separate process (again over COM) for sandboxing reasons. JSON over TCP is perhaps a silly IPC mechanism for local services, but this kind of composition doesn’t seem unreasonable to me.
  - jameshart 3y ago
    > use Word as a remote spellchecker over COM That's not how COM works. You can load Word's spellchecker into your process. Windows added a spellchecking API in Windows 8. I've not dug into the API in detail, but don't see any indication that spellchecker providers run in a separate process (you can probably build one that works that way, but it's not intrinsic to the provider model).
    - bri3d 3y ago
      Are you not familiar with out of process COM servers? A lot of Office automation is out of process, even inside of Office itself. Admittedly I’m not sure about the grammar checker specifically. As for the Spellcheck API, external providers are explicitly out of proc: https://learn.microsoft.com/en-us/windows/win32/intl/about-the-spell-checker-api https://learn.microsoft.com/en-us/windows/win32/intl/about-t... Anyway, my point still stands - building desktop apps using composition over RPC is neither new nor a bad idea, although HTTP might not be the best RPC mechanism (although… neither was COM…)
- pseudosavant 3y ago
  The language server pattern is actually a very good comparison. The web service + web UI approach enables you do use different local and/or cloud AI services. That is why most of these servers/services support the OpenAI API.
  - jameshart 3y ago
    Which means most of these servers limit themselves to the capabilities exposed by the OpenAI API.

jameshart 3y ago

But these are typically filling the usecases of productivity applications, not ‘engines’. Microsoft Word doesn’t run its grammar checker as an external service and shunt JSON over a localhost socket to get spelling and style suggestions. Photoshop doesn’t install a background service to host filters. The closest pattern I can think of is the ‘language servers’ model used by IDEs to handle autosuggest - see https://microsoft.github.io/language-server-protocol/ https://microsoft.github.io/language-server-protocol/ - but the point of that is to enable many to many interop - multiple languages supporting multiple IDEs. Is that the expected usecase for local language assistants and image generators?

bri3d 3y ago
Funny choice of example. You’ve always been able to use Word as a remote spellchecker over COM, and as of Windows 8, spellchecking is available system wide and runs in a separate process (again over COM) for sandboxing reasons. JSON over TCP is perhaps a silly IPC mechanism for local services, but this kind of composition doesn’t seem unreasonable to me.
- jameshart 3y ago
  > use Word as a remote spellchecker over COM That's not how COM works. You can load Word's spellchecker into your process. Windows added a spellchecking API in Windows 8. I've not dug into the API in detail, but don't see any indication that spellchecker providers run in a separate process (you can probably build one that works that way, but it's not intrinsic to the provider model).
  - bri3d 3y ago
    Are you not familiar with out of process COM servers? A lot of Office automation is out of process, even inside of Office itself. Admittedly I’m not sure about the grammar checker specifically. As for the Spellcheck API, external providers are explicitly out of proc: https://learn.microsoft.com/en-us/windows/win32/intl/about-the-spell-checker-api https://learn.microsoft.com/en-us/windows/win32/intl/about-t... Anyway, my point still stands - building desktop apps using composition over RPC is neither new nor a bad idea, although HTTP might not be the best RPC mechanism (although… neither was COM…)
pseudosavant 3y ago
The language server pattern is actually a very good comparison. The web service + web UI approach enables you do use different local and/or cloud AI services. That is why most of these servers/services support the OpenAI API.
- jameshart 3y ago
  Which means most of these servers limit themselves to the capabilities exposed by the OpenAI API.