10 ms·
> When our notifications system was running on Java, it was on 30 c32.xl instances. When we switched over to Elixir, we could run on 15. Would be curious to kn
by threeseed 3y ago
> When our notifications system was running on Java, it was on 30 c32.xl instances. When we switched over to Elixir, we could run on 15.
Would be curious to know how they tried to optimise the Java stack.
Because on every benchmark I've seen the JVM is faster in every which way than Elixir. Except for memory where often people will over-provision the JVM rather than look at where their code might be over-allocating or leaking.
- TacticalCoder 3y agoThey rewrote, which is known to help too. Going from 30 to 15 instances is not bad but it's very likely that a Java-to-Java rewrite would have helped go down too. The big one however is going from 200 Python servers to 4 Erlang ones: a 50x reduction is quite something and a Python-to-Python rewrite would not have allowed to achieved a 50x gain: > All this is possible because Elixir, and the Erlang platform underneath, are fundamentally designed for always-online software with many users. When you use the right tool for the job, the benefits are clear.
- chii 3y ago> it's very likely that a Java-to-Java rewrite would have helped go down too. it depends. The elixir programming paradigm might be one where you are able to more easily write efficient, but still highly concurrent code, where as it would take more work to do the same in java.
- elcritch 3y agoElixir / Erlang does open up a lot of paradigms. However it'll be interesting to see how Java fairs with its new virtual threads. There was already Akka so it should happen fairly quickly. Still I'd prefer Elixir. The BEAM VM just runs lighter.
- OkayPhysicist 3y agoIf it's cludged on like Java's attempt at supporting basic functional programming, it's guaranteed to painful regardless.
- ddorian43 3y agoProject loom should make it better in dev experience and easier to reason + better library evosystem.
- ddorian43 3y agoIs this a cpu-heavy workload? If it's mostly io-heavy, I could see a gevent + pypy approach getting 50x better.
- retrocryptid 3y ago[flagged]
- infecto 3y agoWhy is that?
- dragonwriter 3y agoI don’t tend rewrite Python code any more often than is needed due to feature changes or occasionally refactoring to pay off some maintenance friction caused by design that has in practice turned out to be suboptimal. What are you talking about?
- retrocryptid 3y agoWhen you move from one minor rev of python to the next, some language feature changes (either syntax or semantics or features no longer work.) For instance... if you use async io in 2.x, the debugger stops working. Between 2.3, 2.5 and 2.7, the syntax of package variable scoping changed and then the semantics changed from package to class variables. If you used a feature like package variables in your code in 2.3, that code would not work in 2.5. If you fixed it in 2.5, the semantics changed so that if you defined a package variable according to the 2.5 syntax, but it was defined within a class, it became a class variable. This is just an example.
- dragonwriter 3y ago> When you move from one minor rev of python to the next, some language feature changes Even if there was a breaking change affecting your project every minor version, to have the cadence of backward-compatibility induced changes you siggest you’d have to be switching Python versions forward about four times as fast as they are released, which, if you started with the oldest in support version at the beginning of the project, you could only sustain for about a year and a half, before running out of versions to switch forward to. > If you used a feature like package variables in your code in 2.3, Then you are probably out yelling at kids to get off your lawn; 2.3 being out of support for 12 years.
- tyingq 3y agoThere's some actual detail on the Python -> Elixir move here: https://medium.com/pinterest-engineering/fighting-spam-with-guardian-a-real-time-analytics-and-rules-engine-938e7e61fa27 https://medium.com/pinterest-engineering/fighting-spam-with-... In short, the Elixir is doing something completely different. And they are also not counting some new pieces like the database cluster and kafka based aggregators, etc.
- mrighele 3y ago> They rewrote, which is known to help too. Going from 30 to 15 instances is not bad but it's very likely that a Java-to-Java rewrite would have helped go down too. A java-to-java rewrite could have been as much if not more painful than a java-to-elixir rewrite, especially if the service is highly concurrent. Rewriting synchronous code as asynchronous in Java is a lot of work and not fun at all IMO. Given the recent arrival of virtual threads in Java 21 this may not be necessary any longer but at the time I think it was a perfectly reasonable choice.
- za3faran 3y agoWait a few years for a blog post outlining their rewrite in Java's virtual threads for an improvement in performance and latency ;)
- astrobe_ 3y ago> The big one however is going from 200 Python servers to 4 Erlang ones If your rewrite or refactor gives you a 10X in performance, that's not an optimization, but a bugfix. Unless you are a researcher who have just found a revolutionary algorithm.
- coldtea 3y agoOr you just used a crappy inefficient architecture or algorithsm etc. Just because they were slow doesn't make them bugs, nor does using something faster in a rewrite means it's a bugfix.
- znpy 3y agoI’m going to be cynical and say that if they went from 200 to 4 servers they were probably spawning the python code via cgi, synchronously…
- anonymoushn 3y agoWhy? Most software is at least 10x slower than necessary for reasons that don't require inventing new algorithms.
- astrobe_ 3y ago
- 0xedd 3y agoThey threw DB[s] and Kafka into the mix. Python would get them the same net gain, if not more, with less dev cost. Python's I/O workloads perform on par with Go/NodeJS (See FastAPIs 3rd party quarterly benchmarks as an example). If they rebrand Python to Metal or some other name, people would recommend it left and right. It's just suffering from bandwagon criticism. Yet it remains one of the top 3 languages for years, covering several domains.
- ebiester 3y agoPython is the second best language for everything. Even with its warts (which any language gets after 30 years), it's a very solid, defensible choice. However, it does not have a great concurrency story relative to languages that were built concurrency first (Go, Erlang), and it's fair to acknowledge that.
- klohto 3y agoThat’s exactly the thing? Why would you bother optimizing the code, looking for overallocations, leakage, tweak parameters, when you can just take a friendlier language for the same benefits.
- acdha 3y agoI think the question is whether they significantly changed the architecture at the same time. For example, reading the description of the Python migration sounds like they applied a lot of experience which would have benefited any language, and micro-optimizations like what you described would have been a rounding error on those larger changes: https://medium.com/pinterest-engineering/fighting-spam-with-guardian-a-real-time-analytics-and-rules-engine-938e7e61fa27 https://medium.com/pinterest-engineering/fighting-spam-with-...
- asabil 3y agoFaster doesn’t mean more responsive, in fact very often an increase in throughput leads to an increase in latency.
- deleted 3y ago[deleted]
- abrookewood 3y agoErlang and Elixir are famous for maintaining very low latency under load - it's due to the way the Erlang scheduler works.
- boxed 3y ago> except memory That's an enormous "except". Absolutely gigantic.
- DoesntMatter22 3y agoElixir is slower than plain PHP according to the techempower benchmarks. I'm not even sure how that's possible but it is. By like a factor of 2 iirc. I'm not sure how elixir is that slow since it's compiled.
- te_chris 3y agoPHP is fast these days raw-speed wise. Concurrency and parallelism matter more than speed as they translate directly to utilisation.
- abrookewood 3y agoYes, calling something Fast can mean many different things. Erlang/Elixir are not 'fast' at number crunching, but they are very good at concurrency.
- Cthulhu_ 3y agoIt depends on what is actually being benchmarked though; if it's the simple JSON payload, more time will be spent on HTTP parsing (done in nginx for the PHP benchmark so really really fast) and some JSON parsing (done in a C library for the PHP benchmark so really really fast). Basically, how much PHP is actually being benchmarked? Are you looking at a benchmark that compares real-world usage, or a microbenchmark like hello world or a small JSON payload?
- DoesntMatter22 3y agoThat's a reasonable point. This is Techempower as I mentioned originally. In pretty much all the benchmarks PHP smokes Elixir
- coldtea 3y ago>Elixir is slower than plain PHP according to the techempower benchmarks. I'm not even sure how that's possible but it is. You're "Not even sure how that's possible" as if implying PHP is slow? Even old pre-7 PHP was much faster than Ruby, Python, and others. And most PHP libs are just wrappers over C code.
- sph 3y agoThe advantages of Elixir are not performance-related. There is a lot of focus on raw performance on web-related services, when in reality most of their running time is spent waiting for IO. If there are two things the BEAM excels at, is IO and turning almost any problem into half a dozen processes that are scheduled and run in parallel, if not geographically-distributed, with 1/50th the effort of any other language. We live in a world with 32+ core CPUs. If your load is not spread uniformly all over those cores, you're losing a ton of performance. Handling requests over separate threads, like 99% of languages do, still isn't enough if all the business logic runs on the same thread. I'm currently writing a web crawler in Elixir, and it is easier to design it so every request is done and processed in parallel, than to write a naive sequential one you'd do in any other language in half a day.
- deleted 3y ago[deleted]
- _m8fo 3y ago[flagged]
- mcintyre1994 3y agoI think Node is still single core by default? Elixir (or rather the Beam) will handle core utilisation so if you start a load of Elixir processes they’ll be spread across multiple cores.
- _m8fo 3y ago[flagged]
- troupo 3y agoIt doesn't effectively do that. It does about 10% of that, ineffectively. Instead of running a single VM with full knowledge of how to run lightweight processes, designed to fully take advantage of modern multi-core CPUs with multiple guarantees enforced by the runtime you have multiple single-threaded VMs awkwardly communicating with each other over a bolted-on API
- Cthulhu_ 3y agoI think it's more important to look at the re-architecting than the different language. Second, I think certain architectures - like the actor model described in the article - work better and more intuitive if you use a different language. That said, I'm sure a 2x performance improvement could've been done in Java as well if they did a re-architecture. They could also have made a lateral movement and go to a different JVM language, like Scala that also has an actor concurrency model + accompanying syntax.
- zaphar 3y agoThis is an important point. Sometimes when you need to do a major rearchitecture of a system it can help to choose a language that is more appropriate to that architecture. The Elixir/Erlang ecosystem has a better story for Actor Model development so it makes sense to choose them for the new architecture. It depends on the team and the specifics of the new architecture because the devil is always in the details. It may be that the new language isn't enough of a win to justify the switch and sometimes the win from the new architecture is big enough that sticking with the current language makes sense. But a knee jerk response of: This is mostly just good because they rewrote/rearchitected it, ignores the benefits of using a language or technology that fits the new architecture better.
- OkayPhysicist 3y ago> certain architectures... work better and more intuitive if you use a different language. This is the key point that people miss when pretending that languages are interchangeable. The entire point of making a programming language is to make certain types of ways to solve problems easier to express. This constitutes a language's "pretty path". By providing such pretty paths, languages necessarily make less desirable paths, which will be painful to slog through. If you try writing a functional pipeline in Java, you're going to have a much worse time than doing the same in Elixir. If you try to do Object-Oriented class towers in Scheme, it's going to be painful. Etc, Etc. You can write a Rust program and a C program that compile to the exact same binary, but I can put a whole stack of cash on which one's going to be easier.
- coldtea 3y ago>Would be curious to know how they tried to optimise the Java stack They say "The combined effect of better architecture and Elixir". So the post is mostly useless.
- pclmulqdq 3y agoNot if Elixir enables you to use a better architecture. See all of the people who lifted linked-list structures out of C into C++/Rust vectors.
- neeleshs 3y agoYep. We got a 10x improvement on throughput for our backend runtime, which was in Java, by moving to a better architecture for performance hotspots.. using Java again. In a rewrite with a different design/architecture, that new design typically accounts for most gains, rather than language. A language may make some parts of that rewrite simpler.
- javanonymous 3y agoI've gotten 100x improvement with no code change by just adding an index in the database table. An inexperienced developer might have blamed the database and insisted on moving to NoSQL because of "web scale". If they got the chance to rewrite it, they could have pointed to the performance increase as a proof that they were right.
- no_wizard 3y agoThey really should teach benchmark training more widely in the industry. Even though I'm readily here to sing the praises of Elixir when warranted, nothing beats actually profiling end to end the workload(s) in question that need to be improved. Sometimes, it really is optimizing the database that matters most, like adding an index (or using window functions or stored procedures, as in many cases I've had in the past)
- neeleshs 3y agoYes, that too. Indexes and caching
- panick21_ 3y agoAnd of course you can just go and buy a much faster JVM from Azul rather then rewrite everything.
- no_wizard 3y agothen I have to pay Azul for the faster JVM and I may still not sufficiently cut my server costs, or the cost of the JVM paces the reduction I have in server costs. 2 million a year is several developers compensation saved every year. It also opens the door to more savings down the road, potentially, as existing workloads may discover they can use the same approaches to reduce cost / overhead.
- panick21_ 3y agoI was just adding that in addition to the truth that normal Java could just do it. But if that is really not enough, then paying for a few Azul licenses is an addition can save you time and money.
- jjav 3y ago>Would be curious to know how they tried to optimise the Java stack. Fairly safe to day not at all. The JVM is extremely fast, very efficient and very scalable (you can write java code that scales linearly with available cores). If performance or scalability is a metric to care about, it is nearly impossible to outscore java. You can, with very skilled C/C++ developers, but it's going to be difficult to find those people and it'll be a lot of work. If you need extreme performance on a reasonable budget, you can't do better than java. I know java isn't hip anymore, so this is not a popular truth, but it is.