18 ms·

Rust: Dropping heavy things in another thread can make your code 10000x faster

• saagarjha 6y ago
  Why would you ever write a get_size function that drops the object you call it on? Surely in an actual, non-contrived usecase spawning another thread and letting the drop occur there would just be plain worse?
  • Areading314 6y ago
    Right there is no reason to pass ownership to a function like this.
  • nickm12 6y ago
    I took this to be a contrived example to illustrate the point. I could imagine a process that creates a big data structure (e.g. parse an xml file), pulls some data out, and then drops the data structure. If you want to use that data sooner, you can push the cleanup off your thread.
  • pjmlp 6y ago
    Not at all, Herb Sutter has a CppCon talk about this kind of optimisations. It is also the approach taken by C++/WinRT, COM and UWP components get moved into a background cleaning thread, to avoid application pauses on complex data structures reaching zero count.
  • ashtonkem 6y ago
    It’s a contrived example to demonstrate the technique.
  • epage 6y ago
    I believe this is contrived to prove a point. And this isn't just a help in these contrived examples. I believe process cleanup (an extreme case of cleaning up objects) is one of cases where garbage collection performs better because it doesn't have to unwind the stack, call cleanup functions that are not in the cache, and make a lot of `free` calls to the allocator. I vaguely remember reading about Google killing processes rather than having them clean up correctly, relying on the OS to properly clean up any resources of significance. Now this doesn't mean you should do this in all cases. Profile first, see if you can avoid the large objects, and then look into deferred de-allocations ... if the timing of resource cleanup meets your application's guarantees.
    • Reelin 6y ago
      > killing processes rather than having them clean up correctly, relying on the OS I recall Firefox preventing cleanup code from running when you quit a few years ago. Prior to that, quitting with a lot of pages open (ie hundreds) could cause it to lock up for quite some time.
    • seventh-chord 6y ago
      Killing a process without freeing all allocations is, as far as I can tell, routine in C. Especially for memory it makes no sense "freeing" allocations, the whole memory space is getting scrapped anyways. Of course, once you add RAAI the compiler cant reason about which destructors it can skip on program exit, and if programmers are negligent of this you get programs that are slow to close.
      • gpderetta 6y ago
        exit(2) will only call destructors of static objects. Quick_exit not even those.
      • estebank 6y ago
        > Killing a process without freeing all allocations is, as far as I can tell, routine in C. Many times by accident :) > if programmers are negligent of this you get programs that are slow to close. I wouldn't call that negligence, just not fully optimized.
  • Reelin 6y ago
    I think the contrived use case is just for illustrative purposes? If I'm understanding correctly, the combination of cleanup code and deallocation can sometimes consume enough time that it's worth dispatching it on another thread. That's hardly specific to Rust though. As you note that will certainly add some overhead, although that could be minimized by not spawning a fresh thread each time. It could easily reduce latency for a thread the UI is waiting on in many cases.
    • tedunangst 6y ago
      It would be helpful to see an example from a real application, too.
      • masklinn 6y ago
        A very large Vec<String> (say a few million non-empty strings) would do I'd guess, Rust would drop the Vec which would recursively drop each String.
        • burpsnard 6y ago
          I start thinking about c++ 'placement new' and COBOL copy record structures. And aquire/release objects from resource pools that manage themselves. And message queue handlers. And pass-by-reference, and object pools..
• andrewfromx 6y ago
  hmm my first thought its, having to do that is a lot like c and cleaning up my own allocations. This feels like something rust should automatically do for me?
  • ReaLNero 6y ago
    In C, if you forget to clean up, you have a memory leak which is hard to track down. In Rust, if you don't do this, you're not sacrificing memory leaks, only performance. A profiler can tell you when you should drop asynchronously.
    • madmax96 6y ago
      >A profiler can tell you when you should drop asynchronously Is there any profiler that does this today? What are the drawbacks with asynchronous drops?
      • ehsanu1 6y ago
        See some discussion here: https://www.reddit.com/r/rust/comments/gntv7l/dropping_heavy_objects_in_another_thread_can_make/ https://www.reddit.com/r/rust/comments/gntv7l/dropping_heavy...
        • deleted 6y ago
          [deleted]
      • MaulingMonkey 6y ago
        > Is there any profiler that does this today? It requires interpretation, but yes. > What are the drawbacks with asynchronous drops? You pay some overhead for enqueuing work for later. Taken too far, lock contention / false sharing can make performance way worse. The allocators and destructors must be thread safe if offloading to a worker thread. The core running your thread is likely to have some of this data in L1/L2/L3 cache, which might not be true for whatever core would deque the work for asyncronously dropping. It can be harder to attribute the costs of dropping to the right code if it all gets mixed up into a single work queue cleaned up by opaque worker threads when profiling. If you don't use some kind of backpressure mechanism, allocations can potentially outpace deallocations and run you out of memory. ---- So, concrete example: Using Telemetry - a flamegraph style realtime profiler requiring invasive annotations - I was able to track down the cause of a framerate hitch in a game I was working on, to the sudden release of several graphics resources in a game. During events which would significantly restyle the look of some of the terrain, we'd eat several 10s/100s of milliseconds of overhead freeing things - more than enough to cause us to miss vsync. Would've stuck out like a sore thumb in any profiler capable of giving you a rough idea of the stack(s) involved in a 100ms timeframe that you can correlate to a vsync miss / missed frames. D3D9 isn't thread safe (although freeing resources might've been?), but I didn't need to offload the work onto another thread just to amortize the cost over a few frames. Instead, a simple work queue did the trick. Problem solved! New problem: level transitions took significantly longer when doing mass frees of the same resources - more than doubling the cost of deallocation IIRC, for reasons I never did fully understand. Cache thrashing of some sort? We were still maxing out the core running the main thread with mostly cleanup logic... Final code we shipped with used a hybrid solution that would choose between syncronous (high-throughput) and asyncronous (non-stalling) cleanup logic depending on what was happening in-game. Worked like a charm. Of course, this logic was hideously project specific and unable to be automatically chosen correctly for you by the programming language...
• dirtydroog 6y ago
  Oh my good god. I'm hoping this is down to developer naivety rather than being a feature of rust.
  • sockgrant 6y ago
    1) he should pass by reference to avoid the extra copy. So in his example yes it’s dev naivety 2) but somewhere somehow this object will deallocate, so his trick of putting it to another thread would work if the deal location takes awhile. Same for cpp if you have a massive object in a unique ptr. So it’s not a rust issue
    • renewiltord 6y ago
      Where's the extra copy? I don't see one. He's moving the struct into the function, getting size and then dropping it.
    • VWWHFSfQ 6y ago
      > avoid the extra copy there is no copy happening here
  • ReactiveJelly 6y ago
    The same could happen in C++, I think. Destructors are supposed to be called recursively.
  • wizzwizz4 6y ago
    It's not a feature of Rust; it's a "feature" of the way we design operating systems and processors. This is the same in C.
• maxton 6y ago
  I'm not very familiar with Rust, but I don't understand why you wouldn't just use a reference-to-HeavyThing as the function argument, so that the object isn't moved and then dropped in the `get_size` function?
  • epage 6y ago
    For these contrived cases, yes, you would just pass a reference to the function but I think the point is to simplify the case down to demonstrate a point.
    • burpsnard 6y ago
      In the olden days, it was just out.flush(); out.close();
  • Cyph0n 6y ago
    You’re spot on: this is simply a bad example that you would never see in a real application.
  • ehsanu1 6y ago
    If you never drop it, you have a memory leak. If the caller drops it, it's still the same as the `get_size` dropping it in terms of performance impact. Generally you'd only pass ownership when that's needed for some reason. So this toy example might not be realistic but it does demonstrate the performance impact.
  • heavenlyblue 6y ago
    So the caller of the function still needs to free HeavyThing in the same thread.
• cperciva 6y ago
  If freeing the data structure in question takes this long, how much time are you wasting duplicating the data structure?
  • deleted 6y ago
    [deleted]
  • saagarjha 6y ago
    I’m actually very curious why it takes this long; is Rust memseting the buffer when dropping it? Edit: it seems like turning on optimizations seems to improve the situation quite a bit. Not sure why they were profiling the debug build.
    • deleted 6y ago
      [deleted]
    • firethief 6y ago
      > Edit: it seems like turning on optimizations seems to improve the situation quite a bit. Not sure why they were profiling the debug build. This is the most important point in the thread, since it invalidates the results for most purposes.
      • saagarjha 6y ago
        Not completely, it's still 2-3 orders of magnitude slower.
        • firethief 6y ago
          You're right, I expected it would make a bigger difference
          • dathinab 6y ago
            The thing is it's not slow because rust is doing anything wrong or unoptimized, is slow because cleaning up insane amounts of memory allocations is slow. Also if you run this: ``` fn main() { ::std::thread::spawn(move || { println!("end")}); println!("Hello, world!"); } ``` You might notice that "end" might not be printed because the main thread exists before it prints and terminates the process. This means that the dropping might actually not happen if it's at the end of the program and nothing is faster then not doing the work. Also it's a not uncommon pattern in small user facing CLI to leak (memory) resources, as they (should) be cleaned up with the process termination.
• Ididntdothis 6y ago
  I used to do this sometimes with C++ when I realized that clearing out a vector with lots of objects was slow. Is Rust basically based on unique_ptr? One problem with this approach was that you still had to wait for these threads when the application would shut down.
  • saagarjha 6y ago
    Rust basically gives the compiler understanding of unique_ptr and prevents you from using it after you’ve moved it.
    • Ididntdothis 6y ago
      Would you have to keep track of these threads in Rust? I have done a lot of desktop development where you have to be aware of what happens during shutdown. Seems a lot of server guys write their code under the assumption that it will never shut down.
      • firethief 6y ago
        The answer to this question is the same for any language without a heavy runtime. You can choose to join the worker thread, detach it, or kill it.
      • pornel 6y ago
        You would need to add `thread.join()` at the end of main, or have some RAII guard that does it for you. In practice that's probably optional, because the heap and all resources are usually torn down with the process anyway. Important things, like saving data or committing transactions, shouldn't be done in destructors.
    • deleted 6y ago
      [deleted]
  • qcoh 6y ago
    Out of curiosity, how did you do that in C++?
• epage 6y ago
  For those wanting a real world example where this can be useful: I am writing a static site generator. When run in "watch" mode, it deletes everything and starts over (I'd like to reduce these with partial updates but can't always do it). Moving that cleanup to a thread would make "watch" more responsive.
  • firethief 6y ago
    Why can't it cleanup right after the work?
    • pmontra 6y ago
      Or no cleanup at all. A CLI command that runs for a very short time can allocate memory to perform its job, print the result and exit. Then the OS releases all the memory of the process. No idea if Rust can work like this.
      • estebank 6y ago
        std::mem::forget, which doesn't run destructors: https://doc.rust-lang.org/std/mem/fn.forget.html https://doc.rust-lang.org/std/mem/fn.forget.html
      • ReactiveJelly 6y ago
        "Watch mode" for static site gens would mean you leave the process running and let it rebuild the site whenever a file changes, probably 10s to 100s of times in a typical run
  • elcomet 6y ago
    That's not really the same issue that is mentionned in the article though, is it ? The issue from the article would be solved by just passing a reference to the variable. In your case, cleanup is an action that needs to be done before writing new files. So you have to wait for cleanup anyway, don't you ?
    • 6y ago
• deleted 6y ago
  [deleted]
• staticfloat 6y ago
  It seems that this would be a great reason to not pass the entire heavy object through your function, and to instead pass it as a reference. When passing an object (rather than a reference to an object) there's a lot more work going on both in function setup, and in object dropping. I'm not a rust guru, so I don't know the precise wording, but it's simple enough to realize that if this function, as claimed, must drop all the sub-objects within the `HeavyObject` type, then those objects must have been copied from the original object. If you instead define the function to take in a reference (by adding just two `&` characters into your program), the single-threaded case is now almost 100x faster than the multithreaded case. Here's a link to a Rust Playground with just those two characters changed: https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=9bbe169c77cd31b6cd25b74c24635e12 https://play.rust-lang.org/?version=stable&mode=debug&editio... Note that the code that drops the data in a separate thread is not timing the amount of time your CPU is spinning, dropping the data. So while this does decrease the latency of the original thread, the best solution is to avoid copying and then freeing large, complex objects as much as possible. While it is of course necessary to do this sometimes, this particular example is just not one of them. :) As an aside, I'm somewhat surprised that the Rust compiler isn't inlining and eliminating all the copying and dropping; this would seem to be a classic case where compiler analysis should be able to determine that `a.size()` should be computable without copying `a`, and it should be able to eliminate the function call cost as well. Manually doing this gives the exact same timing as my gist above, so I assume that this is happening when passing a reference, but not happening when passing the object itself.
  • fpgaminer 6y ago
    Rust isn't copying anything; everything in the original code would be a move.
  • heavenlyblue 6y ago
    If your function takes a reference to the object, something still needs to free it.
  • heftig 6y ago
    As already mentioned, Rust wasn't copying anything; the `HashMap` is not a `Copy`-able type, so it was just moved around (it's also not very large: all its items are behind a pointer to the heap). All you did was move the drop from the `fn_that_drops_heavy_things` to the end of `main`, where it is outside the timing function.
• chowells 6y ago
  This is the standard problem with tracing data structures to free them. You frequently run into it with systems based on malloc/free or reference counting. The underlying problem is that freeing the structure takes time proportional to the number of pointers in the structure it has to chase. Generational/compacting GC has the opposite problem. Garbage collection takes time proportional to the live set, and the amount of memory collected is unimportant. It's actually a lot to be said for rust that the ownership system lets you transfer freeing responsibility off-thread safely and cheaply in order to not have it block the critical path. But overall, there's nothing really unexpected here, if you're familiar with memory management.
  • loufe 6y ago
    I've not worked with any language thus far without automatic garbage collecting, so this was definitely a neat read for me. It sounds rather elegant.
    • burpsnard 6y ago
      It's worth popping the hood and getting your fingers dirty. C was written in an era where memory was a scarce and precious resource to be grudgingly used if absolutely necessary
      • tsimionescu 6y ago
        Please note that C is quite a few years older than the first GC languages. LISP1.5, ALGOL-68 and APL all had garbage collectors before C even existed. Not to say that it's not worth it to learn manual memory management as well, but it is important not to think that GCs are a fancy modern tool, and that greybeards would never touch one. There were greybeards using punch cards and programming in a GC language, with output printed out on paper.
  • Reelin 6y ago
    > the ownership system lets you transfer freeing responsibility off-thread safely and cheaply in order to not have it block the critical path This can also trivially be done in other languages. Atomically append your pointer to a queue of "large things that need to be freed" and move on as though you had actually called free. Within a particularly time sensitive loop you can even opt to place pointers into a preallocated array locally. Then once per loop iteration swap that array with the thread handling the deallocations for you. It eats up a bit of CPU time but can significantly reduce latency.
• rhacker 6y ago
  Pass by reference?
  • bszupnick 6y ago
    If you pass by reference the heavy object won't be dropped. If your goal is to drop a heavy object, this is a cool way to do it.
• deleted 6y ago
  [deleted]
• cs702 6y ago
  In other words, Rust's automagical memory deallocation is NOT a zero-cost abstraction: fn get_len1(things: HeavyThings) -> usize { things.len() } fn get_len2(things: HeavyThings) -> usize { let len = things.len(); thread::spawn(move || drop(things)); len } The OP shows an example in which a function like get_len2 is 10000x faster than a function like get_len1 for a hashmap with 1M keys. See also this comment by chowells: https://news.ycombinator.com/item?id=23362925 https://news.ycombinator.com/item?id=23362925
  • DasIch 6y ago
    Nothing about how Rust handles deallocation is magical in any way. It's also definitely a zero-cost abstraction as I can see because the manual solution that's equivalent to get_len1() would be to essentially call free() on things. That would ultimately suffer from the same problem.
    • cs702 6y ago
      Yeah, you're right. In hindsight this was a poorly thought-out and poorly written post on my part.
  • dathinab 6y ago
    No the zero-cost refers to the abstraction (and runtime cost), which still is zero-cost. Deallocating is part of the normal work load not the abstraction. Also this isn't rust specific. Most (all?) RAII languages are affected and many GC approaches have this effect, too. Some do add additional abstraction to magically always or sometimes put the de-allocation into another thread. But de-allocating in another thread is not generally good or bad. There are a lot of use-cases where doing so is rather bad or can't be done (in case TLS is involved). Rust and other similar RAII languages at least let you decide what you want to do. Now it's (I think) generally known that certain kinds (not all) of GC do make some thinks simpler for GUI-like usage. Through they also tend to have less control. Note that it's a common pattern for small user CLI facing tools (which are not GC'ed) to leak resources instead of cleaning them up properly. You can do so in rust too if you want but it's a potential problem for longer running applications. Also here is a faster get `get_len` then both which is also more idiomatic rust then both: ``` fn get_len1(things: &HeavyThings) -> usize { things.len() } ``` If you have a certain thread (e.g. UI thread) in which you never want to do any cleanup work you can consider using a container like: ``` struct DropElsewhere<T: Send>(pub Option<T>); impl<T: Send> Drop for DropElsewhere<T> { fn drop(&mut self) { if let Some(value) = self.take() { thread::spawn(move || drop(value)); } } } ``` You can optimize this with `ManualDrop` to have close to zero-runtime overhead (removes the `take` and `if let` part).
• floppy123 6y ago
  Why should i ever need to drop a heavy object for only getting a size? Not in C++ and also not in Rust, the diffent thread idea is just creativ stupidity, sorry
• fpgaminer 6y ago
  Some important things I think people should note before blindly commenting: * The example code is obviously contrived. The real gist is that massive deallocations in the UI thread cause lag, which the example code proves. That very thing can easily happen in the real world. * I didn't see any difference on my machine between a debug build and a release build. * The example is preforming 1 _million_ deallocations. That's why it's so pathological. It's not just a "large" vector. It's a vector of 1 million vectors. While that may seem contrived, consider a vector of 1 million strings, something that's not too uncommon, and which would likely suffer the same performance penalty. * Rust is not copying anything, nor duplicating the structures here. In the example code the structures would be moved, not copied, which costs nothing. The deallocation is taking up 99% of the time. * As an aside, compilers have used the trick of not free-ing data structures before, because it provides a significant performance boost. Instead of calling free on all those billions of tiny data structures a compiler would generate during its lifetime, they just let them leak. Since a compiler is short lived its not a problem, they get a free lunch (pun unintended), and the OS takes care of cleaning up after all is said and done. My point is that this post isn't theoretical, we do deallocation trickery in the real world.
  • papaf 6y ago
    This deallocation trick is neat but in C and C++ you could use a memory pool to do this. In theory, you could also use a memory pool in Rust but I think the standard library uses malloc without some way of overriding this behaviour.
    • deleted 6y ago
      [deleted]
    • orf 6y ago
      You can change the global allocator in any rust project. You can write your own easy enough, or use one like jemalloc
      • bauerd 6y ago
        Think with LD_PRELOAD it is always possible to override the allocator?
• cesarb 6y ago
  Just be careful, because moving heavy things to be dropped to another thread can change the semantics of the program. For instance, consider what happens if within that heavy thing you had a BufWriter: unless its buffer is empty, dropping it writes the buffer, so now your file is being written and closed in a random moment in the future, instead of being guaranteed to have been sent to the kernel and closed when the function returns. And it can even be worse if it's holding a limited resource, like a file descriptor or a database connection. That is, I wouldn't recommend using this trick unless you're sure that the only thing the "heavy thing" is holding is memory (and even then, keep in mind that memory can also be a limited resource).
  • lostmyoldone 6y ago
    I only know a very little rust, but since it's generally a good practice to never defer writing (or other side effects) to an ambiguous future point in time - with memory allocations as the only plausible exception - is there any way in rust to make sure one doesn't accidentally move complex objects with drop side-effects into other threads? Granted the way the type system work you usually know the type of a variable quite well, but could this happen with opaque types? I'm very much out of my depth, but it felt like one of those things that could really bite you if you are unaware, as happened with finalizers in Java decades ago.
    • masklinn 6y ago
      > I only know a very little rust, but since it's generally a good practice to never defer writing (or other side effects) to an ambiguous future point in time - with memory allocations as the only plausible exception - is there any way in rust to make sure one doesn't accidentally move complex objects with drop side-effects into other threads? If you're the one creating the structure, you could opt it out of Send, that'd make it… not sendable. So it wouldn't be able to cross thread-boundaries. For instance Rc is !Send, you simply can not send it across a thread-boundary (because it's a non-threadsafe reference-counting handle). If you don't control the type, then you'd have to wrap it (newtype pattern) or remember to manually mem::drop it. The latter would obviously have no safety whatsoever, the former you might be able to lint for I guess, though even that is limited or complicated (because of type inference the problematic type might never get explicitly mentioned).
• deleted 6y ago
  [deleted]
• andreygrehov 6y ago
  Does anyone know how would this work in Go?
  • arendtio 6y ago
    I have no idea, but my guess is that it doesn't matter, as the deallocation is being done by the garbage collection.
  • echlebek 6y ago
    Lots to be learned at https://blog.golang.org/ismmkeynote https://blog.golang.org/ismmkeynote
• grogers 6y ago
  Contrived examples like this are ridiculous. Creating such a heavy thing is likely even more expensive than tearing it down. So unless you create it on a separate thread, you probably shouldn't be freeing it on a separate one. It's not going to solve your interactivity problem. If you are creating the object on a separate thread then it's already going to be natural to free it on a separate one too.
  • ReactiveJelly 6y ago
    Something is better than nothing.
• heftig 6y ago
  If I seriously wanted to move object destruction off-thread, I would use at least a dedicated thread with a channel, so I could make sure the dropper is done at some point (before the program terminates, at the latest). It also avoids starting and stopping threads constantly. Something like this: https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=bf35e0f30b0a730f41b215e5bff8270a https://play.rust-lang.org/?version=stable&mode=debug&editio... You could have an even more advanced version spawning tasks into something like rayon's thread pool, I assume.
  • ReactiveJelly 6y ago
    Someone is working on this as a direct response to this blog: https://www.reddit.com/r/rust/comments/go4xcp/new_crate_deferdrop_defer_dropping_your_data_to_a/ https://www.reddit.com/r/rust/comments/go4xcp/new_crate_defe... And yes, spawning a thread for every drop is horrible. It's just to prove the concept. The defer_drop crate uses a global worker thread.
• SilasX 6y ago
  Completely different dynamic (because no Rust GC), but this reminds me of how Twitch made their server, written in Go, a lot faster by allocating a bunch of dummy memory at the beginning so the garbage collector doesn't trigger nearly as often: https://news.ycombinator.com/item?id=21670110 https://news.ycombinator.com/item?id=21670110
  • the8472 6y ago
    The java equivalent to the Go case would simply be adjusting the -Xms flag. The Go approach is a needlessly convoluted because the runtime doesn't offer any tuning knobs. As for the rust case, if you squint then it's similar to a concurrent collector.
• Animats 6y ago
  There's a worse case in deallocation. Tracing through a data structure being released for a long-running program can cause page faults, unused data having been swapped out. This is part of why some programs take far too long to exit.
• dathinab 6y ago
  One thing I just noticed is that the example doesn't make sure to actually run the new thread to completion before the main thread exists. This means that if you do a "drop in other thread" and then main exists, the drop might never run. Which is often fine as the exit of main causes process termination and as such will free the memory normally anyway. But it would be a problem one some systems where memory cleanup on process exit is less reliable. Through such systems are more rare by now I think.
  • ReactiveJelly 6y ago
    It would have to be a non-desktop system. I'm pretty sure Linux will always free process-private memory, and threads, and file descriptors when a process exits. The only things that can leak in typical cases are some kinds of shared memory and maybe child processes?
• littlestymaar 6y ago
  The title is slightly wrong: it's not going to make your code faster, it's going to reduce latency on the given thread. It maybe a net win if this is the UI thread of a desktop app, but overall, it will come at a performance cost: because modern allocators have thread-local memory pools, and now you're moving away from it. And if you're running you code on a NUMA system (most server nowadays), when moving from one thread to another, you can end up freeing non-local memory instead of local one. Also, you won't have any backpressure on your allocations, and you are susceptible to run out of memory (especially because your deallocations now occur more slowly than they should) Main takeaway: if you use it blindly it's an anti-pattern, but it can be a good idea in its niche: the UI thread of a GUI.
  • kevingadd 6y ago
    this could make your code faster by providing more consistent control flow: the main thread is always doing Work and your gc threads are always cleaning up dead objects. this provides fewer, better-predicted branches and code that's more likely to stay in the icache. most gc based environments use dedicated threads for gc and finalizers, this is one reason to do so edit: to be more specific: your normal flow is to alloc at the top of your function, and at the bottom you dealloc. so in basically every case you are paying the cost of deallocs, but if the alloc is conditional the dealloc is now also conditional which is more branches to predict. the dealloc is probably also handled by functions so you have jumps/calls eating up branch prediction table space in the gc/offloaded dealloc scenario, your deallocs on the work thread are no longer conditional because you're just handing addresses off to the gc. if your gc is STW you've added 'if (stop_requested) stop()' branches throughout your workload, but those are effectively 0-cost because stop_requested is always false (when it's true, the cost of the mispredict has no significance because your thread is about to suspend). the gc thread is always doing the same thing or waiting, and again when it's about to wait a branch mispredict cost has no significance.
  • pshc 6y ago
    Yes it’ll reduce latency, but doesn’t it also increase parallelism? A single-threaded program ought to improve overall, unless the extra overhead you mentioned dominates. A parallel program might improve or not. I think if you wanted to do deferred destruction right, ideally you’d mod an allocator to have functions like (alloc_local, alloc_global, free_now, free_deferred) to avoid exhausting memory. Traits could make this ergonomic. Also I admit I don’t understand why “you won’t have any backpressure on your allocations,” shouldn’t deferred destruction give you more backpressure if anything? I am probably confused.
• thickice 6y ago
  Is this applicable for Go as well ?
• jeffdavis 6y ago
  Speedup numbers should be given when optimizing constant factors -- e.g. "I made this operation 5X faster using SIMD" or "By employing readahead, I sped up this file copy by 10X". The points raised in this article are really different: * don't do slow stuff in your latency-critical path * threads are a nice way to unload slow stuff that you don't need done right away (especially if you have spare cores) * dropping can be slow The first and second points are good, but not really related to rust, deallocations, or the number 10000. The last point is worth discussing, but still not really related to the number 10000 and barely related to rust. Rust encourages an eager deallocation strategy (kind of like C), whereas many other languages would use a more deferred strategy (like many GCs). It seems like deferred (e.g. GC) would be better here, because after the main object is dropped, the GC doesn't bother to traverse all of the tiny allocations because they are all dead (unreachable by the root), and it just discards them. But that's not the full story either. It's not terribly common to build up zillions of allocations and then immediately free them. What's more common is to keep the structure (and its zillions of allocations) around for a while, perhaps making small random modifications, and then eventually freeing them all at once. If using a GC, while the large structure is alive, the GC needs to scan all of those objects, causing a pause each time, which is not great. The eager strategy is also not great: it only needs to traverse the structure once (at deallocation time), but it needs to individually deallocate. The answer here is to recognize that all of the objects in the structure will be deallocated together. Use a separate region/arena/heap for the entire structure, and wipe out that region/arena/heap when the structure gets dropped. You don't need to traverse anything while the structure is alive, or when it gets dropped. In rust, probably the most common way to approximate this is by using slices into a larger buffer rather than separate allocations. I wish there was a little better way of doing this, though. It would be awesome if you could make new heaps specific to an object (like a hash table), then allocate the keys/values on that heap. When you drop the structure, the memory disappears without traversal.
• pierrebai 6y ago
  I've seen variations on this trick multiple times. Using threads, using a message sent to self, using a list and a timer to do the work "later", using a list and waiting for idle time... They all have one thing in common: pampering over a bad design. In the particular example given, the sub-vector probably come from a common source. One could keep a big buffer (a single allocation) and an array of internal pointers. For example of such a design to hold a large array of text strings, see for example this blog entry and its associated github repo: https://www.spiria.com/en/blog/desktop-software/optimizing-shared-data/ https://github.com/pierrebai/FastTextContainer Roughly it is this: struct TextHolder { const char* common_buffer; std::vector<const char*> internal_pointers; }; This is of course addressing the example, but the underlying message is generally applicable: change your flawed design, don't hide your flaws.
  • viraptor 6y ago
    Yes. There's also a number of pool/arena allocators in rust which could be used here instead to drop All entries at once.
• thePunisher 6y ago
  The obvious solution would be to borrow the HeavyThing instead of having it dropped inside the function.
• wmichelin 6y ago
  Minor typo, `froget` instead of `forget`
• jkoudys 6y ago
  It'd be interesting to implement this on a type that would defer all of these drop threads (or one big drop threads built off a bunch of futures) until the end of some major action, like sending the http response on an actix-web thread. Could be a great way to get the fastest possible response time, since then the client has their response before any delay on cleanup.
  • AaronFriel 6y ago
    There is no such thing as a free lunch here, so it would reduce the unloaded response time but should have no effect (or a negative impact) on a highly loaded server's response time. I'm finding this out when benchmarking a message passing/queue management system. Anything I do to defer work onto a separate threadpool improves latency up to a point, then reduces throughput.
    • jkoudys 6y ago
      If you're bottlenecked, then certainly. There's no free lunch, but for us, problems that can be solved by simply scaling up the resources on the host as relatively cheap as free vs expensive developer time. When we're purely focused on sales and not anywhere close to hitting a full mem/cpu bottleneck, this would bee good. This situation you describe sounds a lot like dealing with garbage-collection cycles, so you give a good recommendation on something to watch out for, as rust performing at the level of a GC'd language removes a big reason for choosing rust.
• ncmncm 6y ago
  There is nothing unique to Rust about this; it is a very old technique. It is usually much inferior to the "arena allocator" method, where all the discarded allocations are coalesced and released in a single, cheap operation that could as well be done without another thread. That method is practical in many languages, Rust possibly included. C++ supports it in the Standard Library, for all the standard containers. If important work must be done in the destructors, it is still better to farm the work out to a thread pool, rather than starting another thread. Again, C++ supports this in its Standard Library, as I think Rust does too. One could suggest that the only reason to present the idea in Rust is the cynical one that Rust articles get free upvotes on HN.
  • ShroudedNight 6y ago
    > C++ supports it in the Standard Library, for all the standard containers. I don't know what the situation is today, but in the past, the GCC standard library containers had non-trivial destructors when running in debug mode. Ensuring their proper invocation was required to avoid dangling pointers in their book keeping. Non-obvious and painful to debug.
• thu123 6y ago
  Thu test
• earthboundkid 6y ago
  Maybe some sort of “collector” could come by a clean up “garbage” memory periodically to improve performance…
• chubot 6y ago
  Looks like Evan Wallace ran into the same issue in practice in esbuild https://news.ycombinator.com/item?id=22336284 https://news.ycombinator.com/item?id=22336284 I actually originally wrote esbuild in Rust and Go, and Go was the clear winner. The parser written in Go was both faster to compile and faster to execute than the parser in Rust. The Go version compiled something like 100x faster than Rust and ran at something around 10% faster (I forget the exact numbers, sorry). Based on a profile, it looked like the Go version was faster because GC happened on another thread while Rust had to run destructors on the same thread. ESBuild is a really impressive performance-oriented project: https://github.com/evanw/esbuild https://github.com/evanw/esbuild The Rust version also had other problems. Many places in my code had switch statements that branched over all AST nodes and in Rust that compiles to code which uses stack space proportional to the total stack space used by all branches instead of just the maximum stack space used by any one branch: https://github.com/rust-lang/rust/issues/34283 https://github.com/rust-lang/rust/issues/34283 (copy of lobste.rs comment)
• snicker7 6y ago
  I wonder if it might be possible for OS's to provide a fast, asynchronous way of deallocating memory.
• crimsonalucard1 6y ago
  I guess choosing when or how a program deallocates is important in a language that's close to the metal. Rust tries to be zero cost while providing abstractions that make it seemingly a high level language but ultimately things like this show that it's not exactly zero cost because abstractions can incur hidden penalties. There needs to be some internal syntax that allows a rust user to explicitly control deallocation when needed. If I started reading code where people would randomly move a value into another thread and essentially do nothing I would be extremely confused. Any language that begins to rely on "trick" or "hacks" as standard patterns exposes a design flaw. Maybe if rust provided special syntax that a function can be decorated with so that it does deallocation in another thread automatically? Or maybe an internal function called drop_async...? This would make this pattern an explicit part of the language rather than a strange hack/trick.