3 ms·

Looks like a cool project, but I am not sure I understand the need for one process per user. Some questions: - Why do you need one process per user? For low l

Looks like a cool project, but I am not sure I understand the need for one process per user. Some questions: - Why do you need one process per user? For low latency, would you just need to make sure you have idle CPU to serve their request, even if that CPU time is multiplexed onto an event loop (one event loop serves many users)? - Wouldn't this "event loop" actually be more efficient that one user/process, as there would be less context switching cost from the OS? - Can I just keep a map of (connection, thread_id) on my server, and spawn one thread per user on my own server? - Could I just load up my server with many cores, and give each user a SQLite database which runs each query in its own thread? - This way a multi GB database would not be loaded into RAM, the query would filter it down to a result set.

paulgb 5y ago
Good questions! > Why do you need one process per user? / Wouldn't this "event loop" actually be more efficient that one user/process, as there would be less context switching cost from the OS? We're particularly interested in apps that are often CPU-bound, so a traditional event-loop would be blocked for long periods of time. A typical solution is to put the work into a thread, so there would still be a context switch, albeit a smaller one. The process-per-user approach makes the most sense when a significant amount of the data used by each user does not overlap with other users. VS Code (in client/server mode) is a good example of this -- the overhead of siloing each process is relatively low compared to the benefits it gives. We think more data-heavy apps will make the same trade-offs. > Can I just keep a map of (connection, thread_id) on my server, and spawn one thread per user on my own server? If you don't have to scale beyond one server, this approach works fine, but it makes scaling horizontally complicated because you suddenly can't just use a plain old load balancer. It's not just about routing requests to the right server; deciding which server to run the threads on becomes complicated because you ideally want to decide based on the server load of each. We started going down this path, realized we'd end up re-inventing Kubernetes, so decided to embrace it instead. > Could I just load up my server with many cores, and give each user a SQLite database which runs each query in its own thread? This way a multi GB database would not be loaded into RAM, the query would filter it down to a result set. If, for a particular use case, it's economical to keep the data ready in a database that supports the query pattern users will make, it's probably not a good fit for a session-lived backend. In database terms, where our architecture makes sense is when you need to create an index on a dataset (or subset of a dataset) during the runtime of an application. For example, if you have thousands of large parquet files in blob storage and you want a user to be able to load one and run Falcon-type[1] analysis on it. [1] https://github.com/vega/falcon https://github.com/vega/falcon
- justsomeuser 5y ago
  So it is kind of like server-less, but each instance: - Persists for the lifetime of the user session. - Only processes a single user session. - Has large amounts of CPU/RAM and writable disk to handle large datasets.
  - paulgb 5y ago
    Yes, that sounds about right.
- mike_d 5y ago
  > A typical solution is to put the work into a thread, so there would still be a context switch, albeit a smaller one. Car wasn't fast enough, so we removed the rear view mirror to lower weight. You are looking at the sexy fun to solve problem rather than the useful boring solution of throwing away the stack. Users can already run things like Solidworks in a web browser with near native performance using VDI. > deciding which server to run the threads on becomes complicated because you ideally want to decide based on the server load of each High end load balancers have done this since the 90s. This is now easily done with the nginx API. Honestly I am sure there is some need somewhere for your stack. But hiring a good server/network operations team instead would have saved you a lot of code.
  - paulgb 5y ago
    One way to look at it is that it’s like the architecture Github Codespaces uses internally, made available off-the-shelf. I don’t think using a VDI approach would make Codespaces a better product. In fact, I was partly motivated to build this by frustration with laggy VDI setups I had to deal with (though I don’t think VDI has to be bad.) > High end load balancers have done this since the 90s. This is now easily done with the nginx API. A load balancer doesn’t (or at least shouldn’t) do everything we need to do, which involves statefully mapping hostnames generated on-the-fly to servers in a cluster. This allows our users to create instances that multiple clients can connect to, as opposed to just using “sticky sessions” or something like that. Our approach takes less code than you might think —- we lean heavily on nginx and Kubernetes where we can, so we only need to fill in the missing pieces.

paulgb 5y ago

Good questions! > Why do you need one process per user? / Wouldn't this "event loop" actually be more efficient that one user/process, as there would be less context switching cost from the OS? We're particularly interested in apps that are often CPU-bound, so a traditional event-loop would be blocked for long periods of time. A typical solution is to put the work into a thread, so there would still be a context switch, albeit a smaller one. The process-per-user approach makes the most sense when a significant amount of the data used by each user does not overlap with other users. VS Code (in client/server mode) is a good example of this -- the overhead of siloing each process is relatively low compared to the benefits it gives. We think more data-heavy apps will make the same trade-offs. > Can I just keep a map of (connection, thread_id) on my server, and spawn one thread per user on my own server? If you don't have to scale beyond one server, this approach works fine, but it makes scaling horizontally complicated because you suddenly can't just use a plain old load balancer. It's not just about routing requests to the right server; deciding which server to run the threads on becomes complicated because you ideally want to decide based on the server load of each. We started going down this path, realized we'd end up re-inventing Kubernetes, so decided to embrace it instead. > Could I just load up my server with many cores, and give each user a SQLite database which runs each query in its own thread? This way a multi GB database would not be loaded into RAM, the query would filter it down to a result set. If, for a particular use case, it's economical to keep the data ready in a database that supports the query pattern users will make, it's probably not a good fit for a session-lived backend. In database terms, where our architecture makes sense is when you need to create an index on a dataset (or subset of a dataset) during the runtime of an application. For example, if you have thousands of large parquet files in blob storage and you want a user to be able to load one and run Falcon-type[1] analysis on it. [1] https://github.com/vega/falcon https://github.com/vega/falcon

justsomeuser 5y ago
So it is kind of like server-less, but each instance: - Persists for the lifetime of the user session. - Only processes a single user session. - Has large amounts of CPU/RAM and writable disk to handle large datasets.
- paulgb 5y ago
  Yes, that sounds about right.
mike_d 5y ago
> A typical solution is to put the work into a thread, so there would still be a context switch, albeit a smaller one. Car wasn't fast enough, so we removed the rear view mirror to lower weight. You are looking at the sexy fun to solve problem rather than the useful boring solution of throwing away the stack. Users can already run things like Solidworks in a web browser with near native performance using VDI. > deciding which server to run the threads on becomes complicated because you ideally want to decide based on the server load of each High end load balancers have done this since the 90s. This is now easily done with the nginx API. Honestly I am sure there is some need somewhere for your stack. But hiring a good server/network operations team instead would have saved you a lot of code.
- paulgb 5y ago
  One way to look at it is that it’s like the architecture Github Codespaces uses internally, made available off-the-shelf. I don’t think using a VDI approach would make Codespaces a better product. In fact, I was partly motivated to build this by frustration with laggy VDI setups I had to deal with (though I don’t think VDI has to be bad.) > High end load balancers have done this since the 90s. This is now easily done with the nginx API. A load balancer doesn’t (or at least shouldn’t) do everything we need to do, which involves statefully mapping hostnames generated on-the-fly to servers in a cluster. This allows our users to create instances that multiple clients can connect to, as opposed to just using “sticky sessions” or something like that. Our approach takes less code than you might think —- we lean heavily on nginx and Kubernetes where we can, so we only need to fill in the missing pieces.