9 ms·

> Ideally in a web application, aside from some in-memory caches, no object allocated as part of a request should survive longer than the request itself. This

> Ideally in a web application, aside from some in-memory caches, no object allocated as part of a request should survive longer than the request itself. This is one of those areas where out of process caching wins. In process caching has a nasty habit of putting freshly created objects into collections that have survived for days or hours, creating writes in the old generation and back references from old to new. Going out of process makes it someone else’s problem. And if it’s a compiled language with no or a better GC, all the better.

byroot 2y ago
Author here. Agreed. We have some facility for out of process caching (node local memcached), and I frequently have to argue with colleagues that it's generally preferable to in-process caching.
- hinkley 2y ago
  I accept that it is true but I bristle at the fact of it. It shouldn’t be true.
  - byroot 2y ago
    Depends, it's not just about access time and GC pressure, it's also about sharing that cache with other processes on the node.
- foobazgt 2y ago
  A similar strategy is to serialize the object and store it in-process but off heap. This is useful when the values are private to the process, and/or they don't need to survive a crash, and/or you need to avoid the network overhead. Access times are often 100x-1000x faster.
  - kgeist 2y ago
    This is something I've been thinking about for a while. What if we create a language where all object references are explicitly specified to be either request-scoped or application-scoped. Don't allow application-scoped objects reference request-scoped objects. Allow to manually upgrade a reference from request-scoped to application-scoped if needed. That would allow us to have ephemeral per-request heaps which are torn down after every request at once. In-request garbage collections are super-fast. Application-scoped objects are never collected (i.e. no major collections). Wouldn't this simple model solve most problems? Basically, a very simple equivalent to Rust's lifetimes tailored to web services without all the complexity, and much less GC overhead than in traditional GC systems.
    - JonChesterfield 2y ago
      You could call them "arenas" to be consistent with the prior art. Yes, if you can partition the heaps into ones with distinct lifetimes, good plan.
    - ryukoposting 2y ago
      Given that Rust seems to be the generalized solution to this problem, would a viable prototype just be a Rust HTTP server with a Ruby interpreter embedded in it? Write the code in some kind of Ruby DSL which then feeds back into Rust? I ask because I have embedded Ruby in applications before, and I'm looking for an excuse to do it in Rust.
    - jnwatson 2y ago
      "What if we create a language where all object references are explicitly specified to be either request-scoped or application-scoped." I've done this in both C and C++. The downside of automatic memory management is you have to accept the decisions the memory manager makes. Still, generational GC like in Ruby and Python essentially attempts to discern the lifetime of allocations, and it gets it right most of the time.
echelon 2y ago
Rust is emerging as a major contender for HTTP/gRPC backend services. Actix, Axum, sqlx, diesel, and a whole host of other utilities and frameworks make writing Rust for HTTP just as easy and developer efficient as Golang or Java, but the code will never have to deal with GC. It's easy to pull request scoped objects into durable caches.
tsimionescu 2y ago
This could also be a case for non-managed objects in the same process. APIs aren't typically very friendly, but I would expect they could be made so, especially if it was for a dedicated use-case like caching.
deleted 2y ago
[deleted]
the_mitsuhiko 2y ago
One of the challenges I see in general is that languages don't have enough capabilities to express intent of lifetimes / control flow. What I mean by that is that there is a significant difference between spawning a thread with the intention of joining, allocating memory with the intention of only lasting to the end of the request etc. vs spawning a permanent background thread or stashing away an object into a global cache. This is starting to really become a problem in the observability space and async locals. Node.js for instance currently will keep async locals around for too long of a time because they are propagated everywhere. For instance if you call `console.log` in a promise you will leak an async local forever. Next.js famously keeps around way too many async locals past the request boundary for caching related reasons. A solution would be to have a trampoline to call things through that make it explicit that everything happening past that point is supposed to "detach" from the current flow. An allocator or a context local system can then use that information to change behavior.
- dinosaurdynasty 2y ago
  You could do something like that in Rust with a request-scooped arena. But then you'd have to do Rust. In node you could use worker threads (which create a new V8 instance in a separate OS thread) but that's probably too heavy handed.

byroot 2y ago

Author here. Agreed. We have some facility for out of process caching (node local memcached), and I frequently have to argue with colleagues that it's generally preferable to in-process caching.

hinkley 2y ago
I accept that it is true but I bristle at the fact of it. It shouldn’t be true.
- byroot 2y ago
  Depends, it's not just about access time and GC pressure, it's also about sharing that cache with other processes on the node.
foobazgt 2y ago
A similar strategy is to serialize the object and store it in-process but off heap. This is useful when the values are private to the process, and/or they don't need to survive a crash, and/or you need to avoid the network overhead. Access times are often 100x-1000x faster.
- kgeist 2y ago
  This is something I've been thinking about for a while. What if we create a language where all object references are explicitly specified to be either request-scoped or application-scoped. Don't allow application-scoped objects reference request-scoped objects. Allow to manually upgrade a reference from request-scoped to application-scoped if needed. That would allow us to have ephemeral per-request heaps which are torn down after every request at once. In-request garbage collections are super-fast. Application-scoped objects are never collected (i.e. no major collections). Wouldn't this simple model solve most problems? Basically, a very simple equivalent to Rust's lifetimes tailored to web services without all the complexity, and much less GC overhead than in traditional GC systems.
  - JonChesterfield 2y ago
    You could call them "arenas" to be consistent with the prior art. Yes, if you can partition the heaps into ones with distinct lifetimes, good plan.
  - ryukoposting 2y ago
    Given that Rust seems to be the generalized solution to this problem, would a viable prototype just be a Rust HTTP server with a Ruby interpreter embedded in it? Write the code in some kind of Ruby DSL which then feeds back into Rust? I ask because I have embedded Ruby in applications before, and I'm looking for an excuse to do it in Rust.
  - jnwatson 2y ago
    "What if we create a language where all object references are explicitly specified to be either request-scoped or application-scoped." I've done this in both C and C++. The downside of automatic memory management is you have to accept the decisions the memory manager makes. Still, generational GC like in Ruby and Python essentially attempts to discern the lifetime of allocations, and it gets it right most of the time.

echelon 2y ago

Rust is emerging as a major contender for HTTP/gRPC backend services. Actix, Axum, sqlx, diesel, and a whole host of other utilities and frameworks make writing Rust for HTTP just as easy and developer efficient as Golang or Java, but the code will never have to deal with GC. It's easy to pull request scoped objects into durable caches.

tsimionescu 2y ago

This could also be a case for non-managed objects in the same process. APIs aren't typically very friendly, but I would expect they could be made so, especially if it was for a dedicated use-case like caching.

deleted 2y ago

[deleted]

the_mitsuhiko 2y ago

One of the challenges I see in general is that languages don't have enough capabilities to express intent of lifetimes / control flow. What I mean by that is that there is a significant difference between spawning a thread with the intention of joining, allocating memory with the intention of only lasting to the end of the request etc. vs spawning a permanent background thread or stashing away an object into a global cache. This is starting to really become a problem in the observability space and async locals. Node.js for instance currently will keep async locals around for too long of a time because they are propagated everywhere. For instance if you call `console.log` in a promise you will leak an async local forever. Next.js famously keeps around way too many async locals past the request boundary for caching related reasons. A solution would be to have a trampoline to call things through that make it explicit that everything happening past that point is supposed to "detach" from the current flow. An allocator or a context local system can then use that information to change behavior.

dinosaurdynasty 2y ago
You could do something like that in Rust with a request-scooped arena. But then you'd have to do Rust. In node you could use worker threads (which create a new V8 instance in a separate OS thread) but that's probably too heavy handed.