3 ms·

VM is not the right abstraction because of performance and resource requirements. VMs are used because nothing exists that provides same or better isolation. Us

VM is not the right abstraction because of performance and resource requirements. VMs are used because nothing exists that provides same or better isolation. Using a throwaway VM for each AI agent would be highly inefficient (think wasted compute and other resources, which is the opposite of what DeepSeek exemplified).

tptacek 2y ago
To which performance and resource requirements are you referring? A cloud VM runs as long as the agent runs, then stops running.
- ATechGuy 2y ago
  I mean performance overheads of an OS process running in a VM to (vs no VM) and additional resource requirements for running a VM, including memory and additional kernel. You can pull relevant numbers from academic papers.
  - tptacek 2y ago
    OK. Thanks for clarifying. I think you're pretty wrong on this one, for what it's worth.
  - transpute 2y ago
    A linear bar graph comparing compute/memory requirements? - OS process - virtual machine - LLM inference Could have longevity as PC master race meme template.
vineyardmike 2y ago
Is “DeepSeek” going to be the new trendy way to say to not be wasteful? I don’t think DS is a good example here. Mostly because it’s a trendy thing, and the company still has $1B in capex spend to get there. Firecracker has changed the nature of “VMs” into something cheap and easy to spin up and throw away while maintaining isolation. There’s no reason not to use it (besides complexity, I guess). Besides, the entire rest of this is a python notebook. With headless browsers. Using LLMs. This is entirely setting silicon on fire. The overhead from a VM the least of the compute efficiency problems. Just hit a quick cloud API and run your python or browser automation in isolation and move on.
- tptacek 2y ago
  I'm not even talking about Firecracker; for the duration of time things like these run, you could get a satisfactory UX with basic EC2.
- ATechGuy 2y ago
  I think you are assuming that inference happens on the same machine/VM that executes code generated by an AI agent.

tptacek 2y ago

To which performance and resource requirements are you referring? A cloud VM runs as long as the agent runs, then stops running.

ATechGuy 2y ago
I mean performance overheads of an OS process running in a VM to (vs no VM) and additional resource requirements for running a VM, including memory and additional kernel. You can pull relevant numbers from academic papers.
- tptacek 2y ago
  OK. Thanks for clarifying. I think you're pretty wrong on this one, for what it's worth.
- transpute 2y ago
  A linear bar graph comparing compute/memory requirements? - OS process - virtual machine - LLM inference Could have longevity as PC master race meme template.

vineyardmike 2y ago

Is “DeepSeek” going to be the new trendy way to say to not be wasteful? I don’t think DS is a good example here. Mostly because it’s a trendy thing, and the company still has $1B in capex spend to get there. Firecracker has changed the nature of “VMs” into something cheap and easy to spin up and throw away while maintaining isolation. There’s no reason not to use it (besides complexity, I guess). Besides, the entire rest of this is a python notebook. With headless browsers. Using LLMs. This is entirely setting silicon on fire. The overhead from a VM the least of the compute efficiency problems. Just hit a quick cloud API and run your python or browser automation in isolation and move on.

tptacek 2y ago
I'm not even talking about Firecracker; for the duration of time things like these run, you could get a satisfactory UX with basic EC2.
ATechGuy 2y ago
I think you are assuming that inference happens on the same machine/VM that executes code generated by an AI agent.