4 ms·
Next Generation Out of Band Garbage Collection
- jeeyoungk 2y agoThis is from several years ago (2017), but this has very similar vibe as Instagram disabling Python GC - https://instagram-engineering.com/dismissing-python-garbage-collection-at-instagram-4dca40b29172 https://instagram-engineering.com/dismissing-python-garbage-...
- IshKebab 2y agoA 10% performance improvement on Python code is laughable. You can get a 5000% performance improvement if you switch to a better language.
- ziddoap 2y agoI bet the engineers at Instagram were unaware of pythons performance profile when they chose it, you should let them know that they should just switch to a different language.
- byroot 2y agoMeta is just a small startup though, they probably don't have enough resources nor the skills to switch to a better language even after they've heard the gospel.
- immibis 2y agoThe joke is that Facebook literally did, right?
- byroot 2y agoDid what? Rewrite Instagram into another language? Do you have any source on this? Last time I checked they're working on improving Python performance instead (yes I know they forked it into Cinder, but they're trying to upstream their optimizations [0]). Which is very similar to what we're doing at Shopify. Of course 100% of Instagram isn't in Python, I'm certain there's lots of supporting services in C++ etc, but AFAIK the Instagram "frontend" is still largely a Python/Django app. The joke is that if Meta thought that replacing all the Python code they have with something else was worth it, they'd have done it already. [0] https://github.com/facebookincubator/cinder https://github.com/facebookincubator/cinder
- lucianbr 2y ago> The joke is that if Meta thought that replacing all the Python code they have with something else was worth it, they'd have done it already. "Worth it" depends on both how much performance improvement you get, and how hard it is to replace. Did you consider maybe the rewriting effort is so humongous that it is not worth doing despite large performance improvements? Thus making the joke not funny at all...
- byroot 2y agoThat's exactly the joke though. Every time Ruby (or Python) is discussed on HN we get the same old tired question of "why don't they just rewrite in Rust". But that's some silly engineer tunnel vision, squeezing the very last bit of performance out of a system isn't a goal in itself. You just need it to be efficient enough that it cost you significantly less to run that the amount of revenue it brings you. I can bet you that moving off Python must have been pitched dozens and dozens of time by Meta engineers, but deemed not worth it, because execution speed isn't the only important characteristic. So yes, I find it hilarious when HN commenters suggests companies should rewrite all their software into whatever is seen as the most performant one.
- IshKebab 2y agoIt's usually dismissed because companies think short term, and switching languages is a project with huge short term disadvantages and huge long term advantages.
- HumanOstrich 2y agoIt's usually dismissed because it's almost always a huge strategic blunder.
- debugnik 2y agoI think they meant Facebook switched their PHP code to Hack and HHVM, their own PHP-like language and implementation.
- IshKebab 2y agoI don't know if you're joking or not but that is exactly true. Meta went as far as creating their own PHP engine and then a new PHO compatible language because they didn't have the resources to switch from PHP. Instagram is presumably in the same position. Switching language is basically impossible once you have a certain amount of code. I'm sure they were aware of the performance issues with Python but they probably said "we'll worry about it later" when they were a small startup and now it's too late.
- eru 2y agoWell, Facebook also created their own hacked up version of PHP (called Hack) that's presumably easier to migrate PHP to. Hack is actually surprisingly pleasant, basically about the best language they could have made starting from PHP. (I know, that's damning with faint praise. But I actually mean this unironically. It has TypeScript vibes.)
- krapp 2y agoI was excited about Hack when it came out. Unfortunately PHP took just enough from it to kill it. I gave up on it once Composer stopped supporting it, after backwards compatibility with PHP no longer became a goal. IMHO Hack's best feature was native support for XHP... which (also unfortunately) isn't something PHP decided to take.
- eru 2y agoI only used Hack when I was very briefly working for Facebook. (And I used PHP once before nearly 20 years ago by now for some web site I had 'inherited', back when PHP was truly an awful language)
- Twirrim 2y agoHack really gave PHP/Zend the kick up the arse that it seemed to need.
- eru 2y ago
- electricwire 2y agoThey may well have been initially, its a pretty puzzling choice
- worik 2y ago> I bet the engineers at Instagram were unaware of pythons performance profile when they chose it, Is the Instagram stack Python? I doubt it, but stranger things have happened I suspect it is actually some derivative of Apache, or Nginx. Something sensible
- rty32 2y agoVery naive take. 1. 10% performance improvement at Instagram could lead to many millions of revenue "instantly". It is not laughable at any company. 2. It won't be a 5000% performance improvement. Facebook uses its own fork of Python that is heavily optimized. Probably still far from C++, but you should be thinking about languages like Java when talking about performance. "Better" is a very subjective term when discussing languages, and I hope such discussions can be more productive and meaningful.
- foobazgt 2y agoCinder's benchmarks don't seem "like Java" performance, given they aren't that far off cython. https://github.com/facebookincubator/cinder/blob/cinder/3.8/PERF https://github.com/facebookincubator/cinder/blob/cinder/3.8/...
- eru 2y agoCPython itself has seen lots of performance improvements recently. Benchmarks on CPython 3.12 take about half the time they took on CPython 3.9.
- IshKebab 2y agoYeah it's definitely welcome, but even if it is double the performance (doesn't seem to be quite there in my experience) fast languages are still 25-50x faster. It's like walking twice as fast when the alternative is driving.
- eru 2y agoYes. Well, it really depends on whether that alternative is open to you, and at what cost. So eg lots of machine learning code is held together by duct tape and Python. Most of the heavy lifting is done by Python modules implemented in (faster) non-Python languages. The parts that remain in Python could potentially be sped up by migrating them, too. But that migration would likely not do too much for the overall performance, but still be pretty expensive to do (in terms of engineering effort). For organisations in these kinds of situations, it makes a lot of sense to hope for / contribute to a faster Python. Especially if it's a drop-in replacement (like Python 3.12 is for 3.9). What makes really me hopefully is actually JavaScript: on the face of it, JavaScript is actually about the worst language to have a fast implementation. But thanks to advances in clever compiler and interpreter techniques, JavaScript is one of the decently fast languages these days. Especially if you are willing to work in a restricted subset of the language for substantial parts of your code. I'm hoping Python can benefit from similar efforts. Especially since they don't need to re-invent the wheel, but can learn from the earlier and ongoing JavaScript efforts. (I myself made some tiny efforts for CPython performance and correctness. Some of them were even accepted into their repository.)
- deleted 2y ago[deleted]
- deleted 2y ago[deleted]
- hinkley 2y ago> Ideally in a web application, aside from some in-memory caches, no object allocated as part of a request should survive longer than the request itself. This is one of those areas where out of process caching wins. In process caching has a nasty habit of putting freshly created objects into collections that have survived for days or hours, creating writes in the old generation and back references from old to new. Going out of process makes it someone else’s problem. And if it’s a compiled language with no or a better GC, all the better.
- byroot 2y agoAuthor here. Agreed. We have some facility for out of process caching (node local memcached), and I frequently have to argue with colleagues that it's generally preferable to in-process caching.
- hinkley 2y agoI accept that it is true but I bristle at the fact of it. It shouldn’t be true.
- byroot 2y agoDepends, it's not just about access time and GC pressure, it's also about sharing that cache with other processes on the node.
- foobazgt 2y agoA similar strategy is to serialize the object and store it in-process but off heap. This is useful when the values are private to the process, and/or they don't need to survive a crash, and/or you need to avoid the network overhead. Access times are often 100x-1000x faster.
- kgeist 2y agoThis is something I've been thinking about for a while. What if we create a language where all object references are explicitly specified to be either request-scoped or application-scoped. Don't allow application-scoped objects reference request-scoped objects. Allow to manually upgrade a reference from request-scoped to application-scoped if needed. That would allow us to have ephemeral per-request heaps which are torn down after every request at once. In-request garbage collections are super-fast. Application-scoped objects are never collected (i.e. no major collections). Wouldn't this simple model solve most problems? Basically, a very simple equivalent to Rust's lifetimes tailored to web services without all the complexity, and much less GC overhead than in traditional GC systems.
- henning 2y agoThey built a large codebase on a language that doesn't let you control memory, because that makes you "more productive". So just having Rails allocate a per-request arena that is asynchronously freed which would force the programmer not to have any objects that outlive the request, or just pre-allocating memory for a fixed amount of request handling per server instance, or whatever allocation behavior you want to do that is generally possible in C/C++/Zig/Rust/Odin/etc, requires hacking on the language itself. Which means your changes have to go through the Ruby team first. Any additional changes would also need to go through them, which increases the cost of change. Then there is a permanent layer of indirection between your GC callbacks and the semantics of what those callbacks do. Instead of just writing out the custom allocators you want, because that's impossible. How depressing.
- jrockway 2y agoI'm guessing that Zig, Rust, Oden, and "etc." didn't exist when they started the codebase. Now they need to keep moving in their imperfect state. I don't think anyone would start a large company on Ruby today. (They would on Python, though, which is equally unfortunate.)
- bhaak 2y agoStartups are not large companies in the beginning. Although I'm not sure what the preferred language for quickly getting a startup up and running would be these days.
- floating-io 2y agoWhatever works and you can find enough developers for. The language (or the rest of the stack even) is rarely a barrier to success. What matters are a good idea, good motivation, and decent availability of competence. JMHO.
- byroot 2y agoI don't see how it is imperfect. Per request arenas sound super cool on paper, and work very well on system with clear constraints. But if suddenly a request start allocating more than the arena can accommodate you're in a bit of a pickle. They're absolutely not a panacea. Setting aside the challenge of refactoring the Ruby VM to allow this sort of arenas, they'd be a terrible fit for Shopify's monolith. Ultimately, while it's a bit counter intuitive, GCs can perform extremely well in term of throughput. Ruby's GC isn't quite there yet, but still perform quite well and is improving every versions.
- sys64739 2y agoWhat dashboard software is that?
- mhh__ 2y agoLooks like grafana
- spullara 2y agoAll the other virtual machines that support GC need to look at the JVM's ZGC and Shenandoah. Sub-millisecond pause times with terabyte heaps.
- oorza 2y agoC4 still smokes them both, doesn't it?
- spullara 2y agohard to smoke sub-millisecond pauses but there may be other axes where it is better. it used to be that people thought azul was better because it was generational but now zgc is as well. my guess is that c4 doesn't have enough of an edge at this point but happy to see benchmarks that prove otherwise.
- deleted 2y ago[deleted]
- algorithmsRcool 2y agoHow much of a throughput penalty do those options incur on the application?
- spullara 2y agoI think it only affects throughput at the limit at the 5% level. All the portfolio companies that implemented it got a net increase in performance as they avoid redlining their servers.
- bob1029 2y agoI think we should be careful when correlating heap size with how long the collection should take. Also, I really want ZGC in .NET runtime, but I don't think I'll ever get support for it first party. There's some kind of principled ideologue holdout situation going on over at Microsoft. Every time I get into it with one of their engineers I'm sent to some impotent "please may I have a temporary GC exemption" API. All I want is it to do nothing. How hard is it to just not clean up the goddamn garbage? Give me a registry flag + env variable + cli arg all required at the same time if you're so worried someone might trip over it.
- voiper1 2y ago> Ideally in a web application, aside from some in-memory caches, no object allocated as part of a request should survive longer than the request itself. Any object that does is probably something that should be eagerly loaded during boot, or some state that is leaking between requests. As such, any object promoted to the old generation during a request cycle is very unlikely to be immortal, so promoting it is wasteful. So could each request clean up it's own garbage when it finishes, so then they should never need any global garbage collection?
- byroot 2y agoThat's pretty much what I'm hinting at at the end when I mention minor GC. I don't think doing it after each request would be sensible, but counter intuitively, the time it takes to run GC isn't proportional to amount of garbage to collect, but to the number of live objects left (ignoring some minor things like finalizers). So on paper at least we could run a minor GC for very cheap after each request, but there's likely some better heuristics given currently the median request already spent less than 1ms in GC, so after every requests might be overdoing it. Also even if we were doing that, many requests would still have to run GC because they allocate more than there is memory available, so they need to clean their own garbage to continue, you can't delay GC indefinitely. But at least now, endpoints that spend too much time in GC are responsible for their own demise, so engineers responsible for a given endpoint performance have a clear signal that they should allocate less, whereas before it could easily discounted as being caused by lots of garbage left over by another collocated endpoint.
- hgo 2y agoWould it be possible for the allocator/GC to know what allocations are made within a request and make a generation for specifically for it? Allocations too big to fit would be made like usual
- byroot 2y agoThat's already what we effectively have. Since objects cannot be promoted to the old generation inside the request cycle, objects in the new gen are request allocated objects. So if we were to eagerly trigger a minor GC after a request, we'd have very little objects to scan, and only need to sweep garbage, which is only a small fraction of time spent in GC.
- cletus 2y ago> no object allocated as part of a request should survive longer than the request itself So I've spent a lot of time doing Hack (and PHP) as well as Java, Python and other languages. For me, as far as serving HTTP requests goes, Hack/PHP are almost the perfect language. Why? 1. A stateless functional core. There's no loading of large libraries, which is an issue with Python and Java in certain paradigms. The core API us just functions that mean startup costs for a non-stateful service are near zero; 2. The model, as alluded to the above quote, basically creates temporary objects and then tears everything down at the end of the request. It's so much more difficult to leak resources this way as opposed to, say, a stateful Java or C++ server. PHP got a lot of hate unjustly for its "global" scope when in fact it's not global at all. "Global" in PHP/Hack is simply request-scoped and pretty much every language offers request-scoping; 3. There's no threading. Hack, in particular, uses a cooperative async/await model. Where you'd normally create threads (eg making a network request), that's handled by the runtime to make an async/await call out of non-blocking I/O. You never have to deal with mutexes, thread starvation, thread pools, lock ups, etc. You never want to deal with that in "application" or "product" code. Never. So this article is specific to Ruby-on-Rails, which obviously still has persistent objects, hence the need for GC still. How Facebook deals with this is kinda interesting. Most FB product code uses an in-memory write-through graph database (called TAO, backed to MySQL). There is an entity model in Hack on top of this that does a whole bunch of stuff like enforcing privacy (ie you basically never talk to TAO directly and if you do, you're going to have to explain why that's necessary, and you absolutely never talk to MySQL directly). But the point is that persistent entities are request-scoped as well (unlike RoR I guess?).
- BoppreH 2y agoIf the core API is just functions, how do stateful applications handle connections to this persistent storage? Can you still have a connection pool, or does every request pay the extra latency to start a new connection and re-authenticate?
- spullara 2y agoi think they do now but originally one of the reasons people use mysql with languages that are connection per request was that mysql connections were very cheap.