3 ms·

I've always wondered why there is no API to tell the memory allocator to take some time to reorganize itself. E.g. a http server just processed a request and s

I've always wondered why there is no API to tell the memory allocator to take some time to reorganize itself. E.g. a http server just processed a request and sent a response. Before it sits in the queue to pick up a next request, it could tell: Now is a great time to stop this thread if you really need to. Ditto a GUI processed an event. GC could do a minor collection there. Malloc/free could take or give back some chunks to a lower level or the OS. Java had System.gc() but it was more a 'take as much time as you want on every thread and go deep'. I am thinking more of 'Here is avtime limit of a few ms, reorganize this thread only'.

williamcotton 4y ago
There’s a concept of a memory arena… allocate a bunch of memory per frame/request ahead of time and the just increase a pointer when you need memory and then release everything at the end of the frame/request.
- vitiral 4y ago
  You are confounding two subjects: a bump allocator and an arena. "just increase the pointer" is a bump allocator. An arena can be used with any allocator, not just bump allocators.
  - foota 4y ago
    I think there's two common usages of the term arena, I might call them homogenous and non homogenous. Is there much reason to use a non homogenous arena with something other than a bump allocator?
    - vitiral 4y ago
      I'm not sure what homogeneous means in this context? Arenas allow you to not worry about freeing memory, since it will all be freed in a single chunk. It also allows using allocators which are less concerned with fragmentation, since it is known that the arena memory will eventually be dropped. The amount you are concerned with fragmentation ranges from zero concern (bump allocator) to moderate concern (maybe buddy allocator) to larger concern depends on the application and its tradeoffs, like most things in software engineering.
  - hyperman1 4y ago
    I think apache httpd works like this, and there the 2 subjects are mixed. They have a per-request arena. As it gets destroyed at the end of the request, you can just use a bump allocater which is one of the fastest ways to alloc memory.
oddity 4y ago
Most people just want an allocator that works reasonably well, and maintaining that expectation means not exposing too many details that you might be held accountable to. If you care deeply, there are usually alternatives. There's nothing really stopping systems from exposing a hint for controlling this, but usually the people who might care about it don't just want hints, but actual guarantees, and then you have to consider all the users who hint incorrectly (or correctly, but for a different system/version). So, the benefit/cost ratio is low. Integration of GC with thread scheduling was once an active area of research, but the world has mostly moved on (perhaps prematurely, but so goes).
xenadu02 4y ago
Thread-local compacting GCs already handle this. Requests tend to be served by a single thread and the GC dishes out memory from a thread-local pool to avoid contention. Server-side GCs can also be concurrent and avoid suspending threads as much as possible. If you aren't generating garbage that escapes the nursery generation they may basically never suspend threads or block at all. Arenas are a similar concept that work well in non-GC environments if you don't let objects allocated during a request escape to live beyond the request. Thread-local pools to satisfy allocations are also a well-known technique used in various allocators so your system's malloc might be doing this for you. It might also be able to efficiently collapse free lists which more or less behaves like an arena allocator when your request pipeline doesn't let allocations escape. In practical terms: most approaches generalize well enough that if you follow the rules you'd need to follow anyway you get the benefits without the drawbacks. In most of the remaining edge cases you get even better benefits by adopting other strategies. For example Unity uses C# which has a GC... but games shouldn't allocate on the game loop _anyway_ so GC vs ref counts vs malloc isn't very important compared to avoiding allocations altogether. Or if you already obey the rules you'd need for an arena allocator in your server then letting the GC or allocator deal with it might mean you can coalesce cleanup across multiple requests which may actually be better for performance anyway. And without manual arena management you don't risk problems if some esoteric edge case in the code needs to allocate something that will outlive that specific request.
pklausler 4y ago
exit(2) can be pretty effective as an arena collector, actually.
samatman 4y ago
Lua's garbage collector can be asked to do a "step", which is a not-especially-well-defined concept, but which can be tuned until it takes the amount of time you intend. Works well in practice.

williamcotton 4y ago

There’s a concept of a memory arena… allocate a bunch of memory per frame/request ahead of time and the just increase a pointer when you need memory and then release everything at the end of the frame/request.

vitiral 4y ago
You are confounding two subjects: a bump allocator and an arena. "just increase the pointer" is a bump allocator. An arena can be used with any allocator, not just bump allocators.
- foota 4y ago
  I think there's two common usages of the term arena, I might call them homogenous and non homogenous. Is there much reason to use a non homogenous arena with something other than a bump allocator?
  - vitiral 4y ago
    I'm not sure what homogeneous means in this context? Arenas allow you to not worry about freeing memory, since it will all be freed in a single chunk. It also allows using allocators which are less concerned with fragmentation, since it is known that the arena memory will eventually be dropped. The amount you are concerned with fragmentation ranges from zero concern (bump allocator) to moderate concern (maybe buddy allocator) to larger concern depends on the application and its tradeoffs, like most things in software engineering.
- hyperman1 4y ago
  I think apache httpd works like this, and there the 2 subjects are mixed. They have a per-request arena. As it gets destroyed at the end of the request, you can just use a bump allocater which is one of the fastest ways to alloc memory.

oddity 4y ago

Most people just want an allocator that works reasonably well, and maintaining that expectation means not exposing too many details that you might be held accountable to. If you care deeply, there are usually alternatives. There's nothing really stopping systems from exposing a hint for controlling this, but usually the people who might care about it don't just want hints, but actual guarantees, and then you have to consider all the users who hint incorrectly (or correctly, but for a different system/version). So, the benefit/cost ratio is low. Integration of GC with thread scheduling was once an active area of research, but the world has mostly moved on (perhaps prematurely, but so goes).

xenadu02 4y ago

Thread-local compacting GCs already handle this. Requests tend to be served by a single thread and the GC dishes out memory from a thread-local pool to avoid contention. Server-side GCs can also be concurrent and avoid suspending threads as much as possible. If you aren't generating garbage that escapes the nursery generation they may basically never suspend threads or block at all. Arenas are a similar concept that work well in non-GC environments if you don't let objects allocated during a request escape to live beyond the request. Thread-local pools to satisfy allocations are also a well-known technique used in various allocators so your system's malloc might be doing this for you. It might also be able to efficiently collapse free lists which more or less behaves like an arena allocator when your request pipeline doesn't let allocations escape. In practical terms: most approaches generalize well enough that if you follow the rules you'd need to follow anyway you get the benefits without the drawbacks. In most of the remaining edge cases you get even better benefits by adopting other strategies. For example Unity uses C# which has a GC... but games shouldn't allocate on the game loop _anyway_ so GC vs ref counts vs malloc isn't very important compared to avoiding allocations altogether. Or if you already obey the rules you'd need for an arena allocator in your server then letting the GC or allocator deal with it might mean you can coalesce cleanup across multiple requests which may actually be better for performance anyway. And without manual arena management you don't risk problems if some esoteric edge case in the code needs to allocate something that will outlive that specific request.

pklausler 4y ago

exit(2) can be pretty effective as an arena collector, actually.

samatman 4y ago

Lua's garbage collector can be asked to do a "step", which is a not-especially-well-defined concept, but which can be tuned until it takes the amount of time you intend. Works well in practice.