12 ms·

Lithium ranked the highest performance web application platform

99 points by matt42 6y ago

matt42 6y ago
Link to the lithium homepage: https://github.com/matt-42/lithium https://github.com/matt-42/lithium
- Flow 6y ago
  It's Linux only apparently. Is it the use of some Linux only kernel API that enables it to be slightly faster than the other frameworks?
  - matt42 6y ago
    It is linux only because it's based on epoll. It could be easily ported to macos and windows but my focus was to get the performances right first.
    - jeremyjh 6y ago
      There are still a lot of people who develop on MacOS and deploy on linux, so a kqueue port would be valuable even if it were not as well tuned.
      - matt42 6y ago
        a kqueue port should be ok. I'll add an issue for it.
    - giancarlostoro 6y ago
      One could argue if it works on WSL then it works on Windows for development use cases to deploy on Linux.
    - hinkley 6y ago
      There are a few libraries that wrap epoll and friends to work more cross platform. How widely do you use it? Everywhere? A few spots?
      - matt42 6y ago
        The epoll code use is very small, few hundreds of LOC (but is at the very center of the framework since it runs the event loop), actually writing it from scratch was better to understand the mechanism, to optimize it and to adapt it for non blocking sql.
- fluffy87 6y ago
  Can you do an AMA? Why did you create lithium? How long have you been working on it? What do you do professionally? What are your plans for it?
  - matt42 6y ago
    Can you do an AMA? of course! Why did you create lithium? To see how much C++ can get closer to dynamic langagues typically used for web programming and at the same time make absolutely no compromise on performances. How long have you been working on it? 8 months on lithium, but I already wrote another framework (silicon, which I rewrote from scratch because of slow compilation time). What do you do professionally? I worked for several company, doing different things including, real time image processing, neural nets for photography, web programming, robotics What are your plans for it? No big plans yet, just having fun for now, but I'd like to keep implementing the features if the users ask for it, improve the doc. In the more long term, building websockets would be nice too, and other services on top of it.
    - altcognito 6y ago
      What's the real time image processing stuff you work on?
      - matt42 6y ago
        High density point tracking and optical flow was a big part of my PhD, here is a demo :) https://www.youtube.com/watch?v=GIUDAZLfYhY https://www.youtube.com/watch?v=GIUDAZLfYhY
    - ilovelearning 6y ago
      Impressive! Two questions. What’s your opinion regarding Rust versus C++? And what robotics stuff have you worked on?
      - matt42 6y ago
        For Rust vs c++: - I never used rust but I like the language and the safety it provides. - a mainstream package/dependency manager is missing in C++ so rust is clearly better on this point. This is a big problem for the C++ community where code is much less shared than in the javascript community for example. - The other big problem I have with C++ is compile time, I heard that Rust is slow too but I never compared the two. And what robotics stuff have you worked on? I'm building a offline programming tool for industrial robots: http://stara-robotics.com/ http://stara-robotics.com/
- ponytech 6y ago
  Thanks for the link. A web search for "lithium web framework" gave https://li3.me https://li3.me as the first result, which is a PHP framework. Why did you pick such a name if already taken by a similar product?
  - matt42 6y ago
    I was not aware of li3.me and when I realized it, I was too lazy to find a new name..
- yesbutt11 6y ago
  Re: metamap being zero cost I thought there were no zero cost abstractions in C++? https://youtu.be/rHIkrotSwcc https://youtu.be/rHIkrotSwcc Still, it isn’t RoR or anything JS so it has that going for it.
  - matt42 6y ago
    Yes zero cost is subjective. It has a cost at least at compile time. But you can see metamap is a like loop unrolling for objects (if you think loop unrolling is zero cost then metamap too).
- codekilla 6y ago
  This looks great. How would you compare to seastar?
  - matt42 6y ago
    Sadly seastar has no implementation of this benchmark. I had a quick look at the website and it seams that seastar is much more low level (it does not provide a http parser) and more generic that lithium, which is focused on simplifying the writing of HTTP APIs.
montroser 6y ago
On lithium's JSON parser, they describe: "Only cases where the structure of the object is known at compile time are covered." Numbers are impressive, but there are some big trade-offs that come with.
- Yoric 6y ago
  That depends exactly on what that sentence means. It is fairly unusual that you need to parse a JSON with unknown structure during any kind of speed-critical section. My personal criticism is that C++ is usually (not always, though) the wrong language for implementing web-facing applications or APIs. There is a considerable advantage in terms of speed (when using the right framework) but there are so many things that can subtly go wrong in terms of safety and security that you need to be very, very careful when picking C++. And I write this as a C++ developer.
  - regularfry 6y ago
    It's fairly unusual that today I don't know the structure of the JSON I'm parsing. It's not at all uncommon to want the server to do the right thing if I add keys to the object which it didn't previously care about, though.
- deleted 6y ago
  [deleted]
- deleted 6y ago
  [deleted]
polote 6y ago
Lithium is the fastest web framework among the frameworks that nobody use Seriously why is this reaching the front page, who really care about such benchmark ?
- melling 6y ago
  It would be interesting to filter the ones that are used?
- fluffy87 6y ago
  Those who work on these frameworks, and those interested in http framework benchmarking. Some of the ideas developed in these frameworks percolate down to the PLs that use them and to the libraries they use, which benefits everyone. These frameworks also test new language features.
  - aioprisan 6y ago
    Particularly the case with PHP and Zend from 5.0+ days. Are there other examples across other stacks?
- matt42 6y ago
  Lithium development started 8 months ago and is not yet backed by a company. So it has clearly not a lot of traction yet. Just give it some time.
  - polote 6y ago
    I have nothing against Lithium and I wish it become successful, I just don't understand the point of the benchmark
    - matt42 6y ago
      The benchmark is clearly not perfect. But it is great for the community: It pushes all the framework maintainers to optimize their code, so kind of all users of all frameworks are indirectly positively impacted.
fluffy87 6y ago
2x faster than the Actix Rust framework. Rust, C++ just said “hi” to you again, it’s on. Time to make Actix fast.
- deleted 6y ago
  [deleted]
andybak 6y ago
Is this a criteria that's important enough to warrant this much effort? I can think of several metrics that would come first - even for sites with fairly high performance requirements. How often is the web framework genuinely the bottleneck (at least in a way that's not easily fixable with caching or any other similar sledgehammer)?
- matt42 6y ago
  The benchmark show that the frameworks has a big impact on performance. While all the frameworks are using the same database server, performances varies from 1x to 100x. While the benchmark focus on more req/s, we can also see it the other way around: for the same req/s, you can use smaller hardware, i.e. save money and energy.
  - ghostwriter 6y ago
    Not every framework presented in the benchmark perform the same amount of necessary protocol verifications. For instance, here's the take from Servant perspective - https://www.reddit.com/r/haskell/comments/6b3dlt/techempower_benchmarks_14_released_with_servant/dhl500h/?utm_source=reddit&utm_medium=web2x&context=3 https://www.reddit.com/r/haskell/comments/6b3dlt/techempower...
- stagas 6y ago
  In a typical web app you probably wouldn't notice any difference but in the niche where you're handling millions of reqs/s even slight improvements there could possibly translate to a non-trivial amount of money savings. On the other hand if you've already scaled to such a degree then those savings wouldn't matter that much. So, what's left is maybe for educational or research purposes, it's good to have an idea where the upper bound limit is while still maintaining a decent API, so there's value there, it's a great feat regardless.
  - weego 6y ago
    At millions of reqs per second the thin contract of the internals of http request/response will never be the layer that will have any fruit left to pick to make gains.
ex_amazon_sde 6y ago
Most of the time the framework is not in the top 10 bottlenecks of your application. Optimize your webapp before hunting for the fastest frameworks.
- ratww 6y ago
  I know this is a popular line of thought, but as someone who really measures things, it doesn't match my experience when it comes to web server frameworks. In web applications, apart from glaring mistakes such as N+1 queries, pretty much every bottleneck I had in production was due to slow serialisation, slow database abstraction or excessive CPU/memory usage due to language/framework. Also had issues caused by initialisation time. Saving 200ms or 300ms on each request can give you better user experience. For growing startups, having a fast web framework gives you the choice to postpone doing complex horizontal-scaling for months, or even years. This saves money on the sort term and allows you to keep growing their products without having to worry about other things. The database can scale vertically in the meantime. As for large tech companies: at my previous workplace there were two migrations in the span of two years because of performance issues. Hundreds of developers collectively agreed on changing the tech twice because of framework bottlenecks. Of course you shouldn't go straight into a C++ framework when Rails becomes too slow for you, but people should know the tradeoffs of the framework they're using.
  - sitkack 6y ago
    Developer velocity matters way more than the framework as long as the framework allows for horizontal scaling. You don't have to use it, but it should be there and it should be easy to use. Depending on your goals, one should be writing a system that horizontally scales after the first major prototype. Then push the horizontal scaling tests into your CI/CD pipeline and dev locally as if your cluster scaled down to one node.
    - ratww 6y ago
      Nice of you to mention developer velocity: it is extremely important, and it's also severely impacted by things that depend on the framework having good performance, such as initialisation time, development-time performance (it's important to have a quick feedback loop), time required to run individual tests (important for TDD), time required to run the whole test suite (running it often saves developers time during refactoring), time required to compile assets, time for the CI/CD to run (important to allow quick and safe bug fixes and a fast process), etc. Once your app stops being trivial, those things start adding up and people start talking about migrating to something else, breaking the app into microservices, or developing in-house tooling. All those things cost money. It might take a few months but it happens. And I've seen over and over. Also, having horizontal scaling is definitely not easy for non-trivial apps. It costs money to the company, might require extra employees, and might put a hold on new features. And horizontal scaling doesn't help you with performance time or developer velocity. Also please keep in mind that performance and developer velocity are not at odds. Most of today's "fast" frameworks are quite good when it comes to developer velocity: ASP.NET Core for instance is #6 on the Techempower framework and is incredibly productive.
deleted 6y ago
[deleted]
deleted 6y ago
[deleted]
The_rationalist 6y ago
I had investigated what explained the huge gap between the few fastest frameworks vs the rest. The answer is a deceptive one, they didn't achieve revolutionary optimizations. The thing is: on many of those benchmarcks, the bottleneck obviously is the DB. The ability to do DB queries asynchronously and with batching is the differentiating factor. Only postrgresql can support such a feature but you need support on the postrgresql client too. The official C postrgresql client used everywhere does not support said feature except with a patch from 2016. Yes the secret of drogon (and probably of Lithium) is that they use a fork of libpq from 2016 because upstream can't agree on merging the patch and nobody is working on upstreaming it. Actix web benefited from the feature because their client tokio-postgres is a reimplementation and does not use libpq. The industry grade server ecosystem that is the JVM use the jdbc which sadly has a blocking socket thus not allowing asynchronicty. But when loom arrive every jdbc existing code will magically, automatically become truly asynchronous such spring should come on the top 4 place. There is also a wrapper of the jdbc through kotlin coroutines and there are reactive jdbc implementations such as R2DBC. It is unclear as of today if such solution enable postrgresql async queries and batch processing. It seems that nobody has tried those on TechEmpowerUp which is sad. Finally one could use libpq over JNI. Edit: I have read that the next release of pgjdbc (43) will switch from std socket to the NIO non blocking socket. What should be heuristically the fastest HTTP framework (H2O, in C) has refused to use the old libpq fork because the api is not stable and thus not production grade.
- aww_dang 6y ago
  >The industry grade server ecosystem that is the JVM use the jdbc which sadly has a blocking socket thus not allowing asynchronicty. Assuming connection pooling, can't you just use another Thread to do your queries?
  - The_rationalist 6y ago
    I'm not an expert but I believe this would not allow batching, only pipelineing and to a limited extent as spawning OS thread is "slow" and by default I doubt they spawn more threads than CPU number * 2?
paultopia 6y ago
I'm more interested in the fact that something called asp.net core is one of the five fastest, while asp.net not 'core' is the single slowest. What's the deal there? I dimly know that asp.net is a Microsoft thing, and remember it being used by a bunch of enterprise sites a decade or so ago...
- deleted 6y ago
  [deleted]
- mythz 6y ago
  ASP.NET Web Framework is the Windows only 18 yo Web Framework based on .NET Framework https://en.wikipedia.org/wiki/ASP.NET https://en.wikipedia.org/wiki/ASP.NET It could run on Mono/Linux, but did so very slowly & buggy and possibly what the slowest benchmark was run on. ASP.NET Core is the newer (complete rewrite, inc runtime), leaner cross-platform (Win,Linux,macOS) Web Framework that's built on top of the performance focused .NET Core runtime released in 2016 https://dotnet.microsoft.com/learn/aspnet/what-is-aspnet-core https://dotnet.microsoft.com/learn/aspnet/what-is-aspnet-cor... They're only the same by name which is unfortunate, because it's completely new. It's the future runtime/platform for .NET and will be renamed to .NET 5 whilst the older .NET Framework will stay at v4.x (currently at v4.8).
  - qes 6y ago
    > They're only the same by name which is unfortunate, because it's completely new. I certainly wouldn't say that. Full Framework .Net and .Net Core are _far_ more similar than they are different - they share a very large common base class library API surface in Net Standard. A huge percentage of .Net MVC 3-5 (full framework) applications could be migrated to .Net Core MVC in a matter of hours to days. You don't even have to specifically target .Net Core in a lot of cases - v2 included a compatibility shim so that libraries targeting full framework just work as long as they don't call any of the API's that were in full framework but are not in the Core runtime. The main differences are in startup configuration of your web app and how you go about implementing cross-cutting concerns that might inspect or intercept every request - but even for a lot of common things like authentication there are similar extension points to what existed previously. There are some particular technologies - like WCF Server or Linq2Sql - that are hard blockers if you were relying on them - but by and large moving from older full framework .Net to .Net Core is not that difficult and certainly doesn't require learning much new, especially fundamentals - it's mostly just some details of the web app framework.
bsaul 6y ago
has there ever been a code created just specifically for this benchmark, in assembly language with everything hard-coded, just to see what the upper bound is ?
- matt42 6y ago
  Some framework are using raw sql requests to skip the overhead of the ORM but they are still slower (lithium's ORM has no cost at runtime anyway). Coding everything in ASM would be insane and increadibly hard to ensure that this is the optimal version.
  - e12e 6y ago
    Brings to mind: "Rapid Web Application Server (in Assembler)": https://2ton.com.au/rwasa/ https://2ton.com.au/rwasa/ Discussed on hn a few times as I recall: https://news.ycombinator.com/item?id=9948749 https://news.ycombinator.com/item?id=9948749 (see also sshtalk): https://2ton.com.au/sshtalk/ https://2ton.com.au/sshtalk/ https://news.ycombinator.com/item?id=15829206 https://news.ycombinator.com/item?id=15829206
    - matt42 6y ago
      omg...
9wzYQbTYsAIc 6y ago
Interesting to see asp.net at the very bottom of the list of 100+ and asp.net core in the top 10.
- k__ 6y ago
  Well, at least MS acknowledged their mistake
- e12e 6y ago
  Indeed, quite impressive for a presumably "fully production grade" stack to score so hight. But is it me, or are there no fsharp listings in round 19?
  - Multicomp 6y ago
    Zebra is on there. Giraffe may be in there.
    - e12e 6y ago
      Strange, I can find zebra in the filter list, but not in the result list. (ff and chrome on Android).
- ThePadawan 6y ago
  From a cursory inspection, the asp.net implementation seemingly runs on Mono [0] in order to run on linux at all. That might definitely be a performance "gotcha". [0] https://github.com/TechEmpower/FrameworkBenchmarks/tree/master/frameworks/CSharp/aspnet-mono-ngx https://github.com/TechEmpower/FrameworkBenchmarks/tree/mast...
rapfaria 6y ago
Anybody else's laptop activates their fans when visiting this?
- matt42 6y ago
  Me, the benchmark data is pretty slow to process.
foolinaround 6y ago
Link to the project : https://github.com/matt-42/lithium https://github.com/matt-42/lithium not obvious from a google search
matt42 6y ago
I'm wondering why this post got suddenly removed from the front page...
- matt42 6y ago
  And back on the frontpage.. I don't understand HN ranking algorithm.
- giancarlostoro 6y ago
  People probably flagged it cause people hate benchmarks / dont trust them. I think they at least give us some indication about things on the other hand.
  - matt42 6y ago
    arf. ok. It's probably the mods who unflagged it.
    - ricardobeat 6y ago
      There is a comment by one of TechEmpower guys - this benchmark is not the official one and results should be taken with a handful of salt.
      - matt42 6y ago
        This is the official continuous techempower benchmark, run by techempower on the exact same environment than the techempower "official rounds". The only diff I'm aware of is that there is more manual review and check for rounds than for continuous benchmarks. Here is more info about it here: From https://www.techempower.com/blog/2018/06/06/framework-benchmarks-round-16/ https://www.techempower.com/blog/2018/06/06/framework-benchm...: We have already seen tremendous social adoption of the continuous benchmarking results. For selfish reasons, we want to continue creating and posting official rounds such as today's Round 16 periodically. (Mostly so that we can use the opportunity to write a blog entry and generate hype!) We ask that you humor us and treat official rounds as the super interesting and meaningful events that they are. Jokes aside, the continuous results are intended for contributors to the project. The official rounds are less-frequent snapshots suitable for everyone else who may find the data interesting.
karmakaze 6y ago
I really do like these TechEmpower Fortunes benchmarks. It has quite an extensive list of frameworks, languages, and configurations with throughput, latency (w/ SD), and errors. I always check any lesser known framework to see which ballpark it falls into. I'm always surprised to see that so many of the popular frameworks are ~10x worse than the best--although many of those 'best' don't do as much processing. I'm much more likely to pay attention to the error counts, max latency, or SD (σ).
supermatt 6y ago
Incredibly frustrating the benchmarks not linking to the actual frameworks - searching for "lithium web framework" gives me some PHP framework... Edit: Comment lower down suggests: https://github.com/matt-42/lithium https://github.com/matt-42/lithium is the correct one.
- matt42 6y ago
  I'll report this to the benchmark maintainers.
- sitkack 6y ago
  I came here to ask the same question, "Where is the page, with clickable links to the frameworks?".
  - giancarlostoro 6y ago
    They really need this and a page explaining why certain frameworks are removed over time.
- brightball 6y ago
  Yep. That's ooooooooold. Lithium was a project as CakePHP was going through a big update 10 years ago.
- bhauer 6y ago
  We plan to collect additional metadata from maintainers, including home page URL, source URL, description, and so on. Once that happens, we'll be able to provide the kinds of helpful links you want.
  - olaf 6y ago
    Until this is done, you could just give your visitors the download links you used to download the frameworks. Doesn't seem to me overly complicated.
    - 6y ago
bhauer 6y ago
Brian from TechEmpower here. I don't want to take away from these results, but I do want to provide an important piece of context. This link is to a rendering of data from a continuous run [1] that hasn't received the type of sanity checking we do for an official TFB round. You can tell from the title ("Test") and the UUID of the run visible in the gray box under the navigation. Based on this link's attention, we'll make a more prominent warning for renderings of continuous runs so that they are more obvious to readers not familiar with the project. We execute runs like this continuously to allow maintainers of test implementations to observe the results of their contributions. Given the performance seen here, it is very likely that Lithium will be well ranked in the next official round. But we feel the ranking seen in a continuous run such as this should be taken with a grain of salt until that next official round is available. [1] https://tfb-status.techempower.com/results/57b25c85-082a-4013-b572-b0939006eaff https://tfb-status.techempower.com/results/57b25c85-082a-401...
- matt42 6y ago
  Hi Brian, can't wait for round 20 then :D.
- AtlasBarfed 6y ago
  Is that why it didn't vary when I selected cloud vs physical?
  - matt42 6y ago
    Yes, this is only physical. The cloud env (Azure) only run once every 1 or 2 months I think. On the official rounds they do both.
pknerd 6y ago
Thanks to Google, The PHP Framework, Lithium will get some unwanted praise here.
tluyben2 6y ago
Offtopic: matt42 (or others); where do I learn modern optimization like this? I can optimize for cycles on embedded MCU's and older computers and I can optimize in .NET/JVM (and others) but are there any good sources for OS on modern metal optimizations? Besides reading the Lithium sourcecode ofcourse.
- matt42 6y ago
  I would say the profiler output is the best source of information. Lithium is not using black magic at all. Actually on large framework like lithium, performance problems come more from obvious errors that get hiden in the quantity of code than in code that you did not crazy optimize. Only the profiler will tell you if there is a hidden malloc that is slowing down all the rest, or a inefficient datastructure, or ... There is probably lot of non optimized code in lithium, it just has no impact (or if it has I still has to find it). That said, for other kind of software (usually more CPU bound), going down to assembly or using SIMD can be the only way to get good perfs.
thdrdt 6y ago
Wow, the difference between .Net core and .Net!
RocketSyntax 6y ago
just ahead of jooby and kooby
dathinab 6y ago
I wonder how realistic/fair the code of some benchmarks is. I just took some small peak at atix (because I happen to know it) and while first all looked fine it wasn't quite that realistic. Mainly: - It uses a fork of tokio-postgres specific for the test (which differs in that Client is no longer send and it has a Unsafely soundness hole by wrongly using Unsafe Cell, and no Issue tracking enabled on, through replacing that cell with RefCell probably yield very similar performance) - Instead of using the default web::Json responder it uses simd-json (EDIT: explicitly encoding the data into a buffer instead of returning it wrapped in web::Json) - Uses snmalloc instead of the default allocator (https://github.com/microsoft/snmalloc https://github.com/microsoft/snmalloc) Just to be clear besides the first point all of this is not unrealistic for a context where you want to highly optimize your server. But not how actix is used most of the time. --- Edit: Also just to be clear I didn't nit pick intentional on actix, it just happen to be the framework I'm more familiar with and especially using snmalloc for actix seems to be quite a reasonable idea. EDIT2: The Unsefety unsoundness isn't triggered given the way this library is used, it's still there and would prevent this change from ever been merged upstream.

99 points by matt42 6y ago

matt42 6y ago

Link to the lithium homepage: https://github.com/matt-42/lithium https://github.com/matt-42/lithium

Flow 6y ago
It's Linux only apparently. Is it the use of some Linux only kernel API that enables it to be slightly faster than the other frameworks?
- matt42 6y ago
  It is linux only because it's based on epoll. It could be easily ported to macos and windows but my focus was to get the performances right first.
  - jeremyjh 6y ago
    There are still a lot of people who develop on MacOS and deploy on linux, so a kqueue port would be valuable even if it were not as well tuned.
    - matt42 6y ago
      a kqueue port should be ok. I'll add an issue for it.
  - giancarlostoro 6y ago
    One could argue if it works on WSL then it works on Windows for development use cases to deploy on Linux.
  - hinkley 6y ago
    There are a few libraries that wrap epoll and friends to work more cross platform. How widely do you use it? Everywhere? A few spots?
    - matt42 6y ago
      The epoll code use is very small, few hundreds of LOC (but is at the very center of the framework since it runs the event loop), actually writing it from scratch was better to understand the mechanism, to optimize it and to adapt it for non blocking sql.
fluffy87 6y ago
Can you do an AMA? Why did you create lithium? How long have you been working on it? What do you do professionally? What are your plans for it?
- matt42 6y ago
  Can you do an AMA? of course! Why did you create lithium? To see how much C++ can get closer to dynamic langagues typically used for web programming and at the same time make absolutely no compromise on performances. How long have you been working on it? 8 months on lithium, but I already wrote another framework (silicon, which I rewrote from scratch because of slow compilation time). What do you do professionally? I worked for several company, doing different things including, real time image processing, neural nets for photography, web programming, robotics What are your plans for it? No big plans yet, just having fun for now, but I'd like to keep implementing the features if the users ask for it, improve the doc. In the more long term, building websockets would be nice too, and other services on top of it.
  - altcognito 6y ago
    What's the real time image processing stuff you work on?
    - matt42 6y ago
      High density point tracking and optical flow was a big part of my PhD, here is a demo :) https://www.youtube.com/watch?v=GIUDAZLfYhY https://www.youtube.com/watch?v=GIUDAZLfYhY
  - ilovelearning 6y ago
    Impressive! Two questions. What’s your opinion regarding Rust versus C++? And what robotics stuff have you worked on?
    - matt42 6y ago
      For Rust vs c++: - I never used rust but I like the language and the safety it provides. - a mainstream package/dependency manager is missing in C++ so rust is clearly better on this point. This is a big problem for the C++ community where code is much less shared than in the javascript community for example. - The other big problem I have with C++ is compile time, I heard that Rust is slow too but I never compared the two. And what robotics stuff have you worked on? I'm building a offline programming tool for industrial robots: http://stara-robotics.com/ http://stara-robotics.com/
ponytech 6y ago
Thanks for the link. A web search for "lithium web framework" gave https://li3.me https://li3.me as the first result, which is a PHP framework. Why did you pick such a name if already taken by a similar product?
- matt42 6y ago
  I was not aware of li3.me and when I realized it, I was too lazy to find a new name..
yesbutt11 6y ago
Re: metamap being zero cost I thought there were no zero cost abstractions in C++? https://youtu.be/rHIkrotSwcc https://youtu.be/rHIkrotSwcc Still, it isn’t RoR or anything JS so it has that going for it.
- matt42 6y ago
  Yes zero cost is subjective. It has a cost at least at compile time. But you can see metamap is a like loop unrolling for objects (if you think loop unrolling is zero cost then metamap too).
codekilla 6y ago
This looks great. How would you compare to seastar?
- matt42 6y ago
  Sadly seastar has no implementation of this benchmark. I had a quick look at the website and it seams that seastar is much more low level (it does not provide a http parser) and more generic that lithium, which is focused on simplifying the writing of HTTP APIs.

montroser 6y ago

On lithium's JSON parser, they describe: "Only cases where the structure of the object is known at compile time are covered." Numbers are impressive, but there are some big trade-offs that come with.

Yoric 6y ago
That depends exactly on what that sentence means. It is fairly unusual that you need to parse a JSON with unknown structure during any kind of speed-critical section. My personal criticism is that C++ is usually (not always, though) the wrong language for implementing web-facing applications or APIs. There is a considerable advantage in terms of speed (when using the right framework) but there are so many things that can subtly go wrong in terms of safety and security that you need to be very, very careful when picking C++. And I write this as a C++ developer.
- regularfry 6y ago
  It's fairly unusual that today I don't know the structure of the JSON I'm parsing. It's not at all uncommon to want the server to do the right thing if I add keys to the object which it didn't previously care about, though.
deleted 6y ago
[deleted]
deleted 6y ago
[deleted]

polote 6y ago

Lithium is the fastest web framework among the frameworks that nobody use Seriously why is this reaching the front page, who really care about such benchmark ?

melling 6y ago
It would be interesting to filter the ones that are used?
fluffy87 6y ago
Those who work on these frameworks, and those interested in http framework benchmarking. Some of the ideas developed in these frameworks percolate down to the PLs that use them and to the libraries they use, which benefits everyone. These frameworks also test new language features.
- aioprisan 6y ago
  Particularly the case with PHP and Zend from 5.0+ days. Are there other examples across other stacks?
matt42 6y ago
Lithium development started 8 months ago and is not yet backed by a company. So it has clearly not a lot of traction yet. Just give it some time.
- polote 6y ago
  I have nothing against Lithium and I wish it become successful, I just don't understand the point of the benchmark
  - matt42 6y ago
    The benchmark is clearly not perfect. But it is great for the community: It pushes all the framework maintainers to optimize their code, so kind of all users of all frameworks are indirectly positively impacted.

fluffy87 6y ago

2x faster than the Actix Rust framework. Rust, C++ just said “hi” to you again, it’s on. Time to make Actix fast.

deleted 6y ago
[deleted]

andybak 6y ago

Is this a criteria that's important enough to warrant this much effort? I can think of several metrics that would come first - even for sites with fairly high performance requirements. How often is the web framework genuinely the bottleneck (at least in a way that's not easily fixable with caching or any other similar sledgehammer)?

matt42 6y ago
The benchmark show that the frameworks has a big impact on performance. While all the frameworks are using the same database server, performances varies from 1x to 100x. While the benchmark focus on more req/s, we can also see it the other way around: for the same req/s, you can use smaller hardware, i.e. save money and energy.
- ghostwriter 6y ago
  Not every framework presented in the benchmark perform the same amount of necessary protocol verifications. For instance, here's the take from Servant perspective - https://www.reddit.com/r/haskell/comments/6b3dlt/techempower_benchmarks_14_released_with_servant/dhl500h/?utm_source=reddit&utm_medium=web2x&context=3 https://www.reddit.com/r/haskell/comments/6b3dlt/techempower...
stagas 6y ago
In a typical web app you probably wouldn't notice any difference but in the niche where you're handling millions of reqs/s even slight improvements there could possibly translate to a non-trivial amount of money savings. On the other hand if you've already scaled to such a degree then those savings wouldn't matter that much. So, what's left is maybe for educational or research purposes, it's good to have an idea where the upper bound limit is while still maintaining a decent API, so there's value there, it's a great feat regardless.
- weego 6y ago
  At millions of reqs per second the thin contract of the internals of http request/response will never be the layer that will have any fruit left to pick to make gains.

ex_amazon_sde 6y ago

Most of the time the framework is not in the top 10 bottlenecks of your application. Optimize your webapp before hunting for the fastest frameworks.

ratww 6y ago
I know this is a popular line of thought, but as someone who really measures things, it doesn't match my experience when it comes to web server frameworks. In web applications, apart from glaring mistakes such as N+1 queries, pretty much every bottleneck I had in production was due to slow serialisation, slow database abstraction or excessive CPU/memory usage due to language/framework. Also had issues caused by initialisation time. Saving 200ms or 300ms on each request can give you better user experience. For growing startups, having a fast web framework gives you the choice to postpone doing complex horizontal-scaling for months, or even years. This saves money on the sort term and allows you to keep growing their products without having to worry about other things. The database can scale vertically in the meantime. As for large tech companies: at my previous workplace there were two migrations in the span of two years because of performance issues. Hundreds of developers collectively agreed on changing the tech twice because of framework bottlenecks. Of course you shouldn't go straight into a C++ framework when Rails becomes too slow for you, but people should know the tradeoffs of the framework they're using.
- sitkack 6y ago
  Developer velocity matters way more than the framework as long as the framework allows for horizontal scaling. You don't have to use it, but it should be there and it should be easy to use. Depending on your goals, one should be writing a system that horizontally scales after the first major prototype. Then push the horizontal scaling tests into your CI/CD pipeline and dev locally as if your cluster scaled down to one node.
  - ratww 6y ago
    Nice of you to mention developer velocity: it is extremely important, and it's also severely impacted by things that depend on the framework having good performance, such as initialisation time, development-time performance (it's important to have a quick feedback loop), time required to run individual tests (important for TDD), time required to run the whole test suite (running it often saves developers time during refactoring), time required to compile assets, time for the CI/CD to run (important to allow quick and safe bug fixes and a fast process), etc. Once your app stops being trivial, those things start adding up and people start talking about migrating to something else, breaking the app into microservices, or developing in-house tooling. All those things cost money. It might take a few months but it happens. And I've seen over and over. Also, having horizontal scaling is definitely not easy for non-trivial apps. It costs money to the company, might require extra employees, and might put a hold on new features. And horizontal scaling doesn't help you with performance time or developer velocity. Also please keep in mind that performance and developer velocity are not at odds. Most of today's "fast" frameworks are quite good when it comes to developer velocity: ASP.NET Core for instance is #6 on the Techempower framework and is incredibly productive.

deleted 6y ago

[deleted]

deleted 6y ago

[deleted]

The_rationalist 6y ago

I had investigated what explained the huge gap between the few fastest frameworks vs the rest. The answer is a deceptive one, they didn't achieve revolutionary optimizations. The thing is: on many of those benchmarcks, the bottleneck obviously is the DB. The ability to do DB queries asynchronously and with batching is the differentiating factor. Only postrgresql can support such a feature but you need support on the postrgresql client too. The official C postrgresql client used everywhere does not support said feature except with a patch from 2016. Yes the secret of drogon (and probably of Lithium) is that they use a fork of libpq from 2016 because upstream can't agree on merging the patch and nobody is working on upstreaming it. Actix web benefited from the feature because their client tokio-postgres is a reimplementation and does not use libpq. The industry grade server ecosystem that is the JVM use the jdbc which sadly has a blocking socket thus not allowing asynchronicty. But when loom arrive every jdbc existing code will magically, automatically become truly asynchronous such spring should come on the top 4 place. There is also a wrapper of the jdbc through kotlin coroutines and there are reactive jdbc implementations such as R2DBC. It is unclear as of today if such solution enable postrgresql async queries and batch processing. It seems that nobody has tried those on TechEmpowerUp which is sad. Finally one could use libpq over JNI. Edit: I have read that the next release of pgjdbc (43) will switch from std socket to the NIO non blocking socket. What should be heuristically the fastest HTTP framework (H2O, in C) has refused to use the old libpq fork because the api is not stable and thus not production grade.

aww_dang 6y ago
>The industry grade server ecosystem that is the JVM use the jdbc which sadly has a blocking socket thus not allowing asynchronicty. Assuming connection pooling, can't you just use another Thread to do your queries?
- The_rationalist 6y ago
  I'm not an expert but I believe this would not allow batching, only pipelineing and to a limited extent as spawning OS thread is "slow" and by default I doubt they spawn more threads than CPU number * 2?

paultopia 6y ago

I'm more interested in the fact that something called asp.net core is one of the five fastest, while asp.net not 'core' is the single slowest. What's the deal there? I dimly know that asp.net is a Microsoft thing, and remember it being used by a bunch of enterprise sites a decade or so ago...

deleted 6y ago
[deleted]
mythz 6y ago
ASP.NET Web Framework is the Windows only 18 yo Web Framework based on .NET Framework https://en.wikipedia.org/wiki/ASP.NET https://en.wikipedia.org/wiki/ASP.NET It could run on Mono/Linux, but did so very slowly & buggy and possibly what the slowest benchmark was run on. ASP.NET Core is the newer (complete rewrite, inc runtime), leaner cross-platform (Win,Linux,macOS) Web Framework that's built on top of the performance focused .NET Core runtime released in 2016 https://dotnet.microsoft.com/learn/aspnet/what-is-aspnet-core https://dotnet.microsoft.com/learn/aspnet/what-is-aspnet-cor... They're only the same by name which is unfortunate, because it's completely new. It's the future runtime/platform for .NET and will be renamed to .NET 5 whilst the older .NET Framework will stay at v4.x (currently at v4.8).
- qes 6y ago
  > They're only the same by name which is unfortunate, because it's completely new. I certainly wouldn't say that. Full Framework .Net and .Net Core are _far_ more similar than they are different - they share a very large common base class library API surface in Net Standard. A huge percentage of .Net MVC 3-5 (full framework) applications could be migrated to .Net Core MVC in a matter of hours to days. You don't even have to specifically target .Net Core in a lot of cases - v2 included a compatibility shim so that libraries targeting full framework just work as long as they don't call any of the API's that were in full framework but are not in the Core runtime. The main differences are in startup configuration of your web app and how you go about implementing cross-cutting concerns that might inspect or intercept every request - but even for a lot of common things like authentication there are similar extension points to what existed previously. There are some particular technologies - like WCF Server or Linq2Sql - that are hard blockers if you were relying on them - but by and large moving from older full framework .Net to .Net Core is not that difficult and certainly doesn't require learning much new, especially fundamentals - it's mostly just some details of the web app framework.

bsaul 6y ago

has there ever been a code created just specifically for this benchmark, in assembly language with everything hard-coded, just to see what the upper bound is ?

matt42 6y ago
Some framework are using raw sql requests to skip the overhead of the ORM but they are still slower (lithium's ORM has no cost at runtime anyway). Coding everything in ASM would be insane and increadibly hard to ensure that this is the optimal version.
- e12e 6y ago
  Brings to mind: "Rapid Web Application Server (in Assembler)": https://2ton.com.au/rwasa/ https://2ton.com.au/rwasa/ Discussed on hn a few times as I recall: https://news.ycombinator.com/item?id=9948749 https://news.ycombinator.com/item?id=9948749 (see also sshtalk): https://2ton.com.au/sshtalk/ https://2ton.com.au/sshtalk/ https://news.ycombinator.com/item?id=15829206 https://news.ycombinator.com/item?id=15829206
  - matt42 6y ago
    omg...

9wzYQbTYsAIc 6y ago

Interesting to see asp.net at the very bottom of the list of 100+ and asp.net core in the top 10.

k__ 6y ago
Well, at least MS acknowledged their mistake
e12e 6y ago
Indeed, quite impressive for a presumably "fully production grade" stack to score so hight. But is it me, or are there no fsharp listings in round 19?
- Multicomp 6y ago
  Zebra is on there. Giraffe may be in there.
  - e12e 6y ago
    Strange, I can find zebra in the filter list, but not in the result list. (ff and chrome on Android).
ThePadawan 6y ago
From a cursory inspection, the asp.net implementation seemingly runs on Mono [0] in order to run on linux at all. That might definitely be a performance "gotcha". [0] https://github.com/TechEmpower/FrameworkBenchmarks/tree/master/frameworks/CSharp/aspnet-mono-ngx https://github.com/TechEmpower/FrameworkBenchmarks/tree/mast...

rapfaria 6y ago

Anybody else's laptop activates their fans when visiting this?

matt42 6y ago
Me, the benchmark data is pretty slow to process.

foolinaround 6y ago

Link to the project : https://github.com/matt-42/lithium https://github.com/matt-42/lithium not obvious from a google search

matt42 6y ago

I'm wondering why this post got suddenly removed from the front page...

matt42 6y ago
And back on the frontpage.. I don't understand HN ranking algorithm.
giancarlostoro 6y ago
People probably flagged it cause people hate benchmarks / dont trust them. I think they at least give us some indication about things on the other hand.
- matt42 6y ago
  arf. ok. It's probably the mods who unflagged it.
  - ricardobeat 6y ago
    There is a comment by one of TechEmpower guys - this benchmark is not the official one and results should be taken with a handful of salt.
    - matt42 6y ago
      This is the official continuous techempower benchmark, run by techempower on the exact same environment than the techempower "official rounds". The only diff I'm aware of is that there is more manual review and check for rounds than for continuous benchmarks. Here is more info about it here: From https://www.techempower.com/blog/2018/06/06/framework-benchmarks-round-16/ https://www.techempower.com/blog/2018/06/06/framework-benchm...: We have already seen tremendous social adoption of the continuous benchmarking results. For selfish reasons, we want to continue creating and posting official rounds such as today's Round 16 periodically. (Mostly so that we can use the opportunity to write a blog entry and generate hype!) We ask that you humor us and treat official rounds as the super interesting and meaningful events that they are. Jokes aside, the continuous results are intended for contributors to the project. The official rounds are less-frequent snapshots suitable for everyone else who may find the data interesting.

karmakaze 6y ago

I really do like these TechEmpower Fortunes benchmarks. It has quite an extensive list of frameworks, languages, and configurations with throughput, latency (w/ SD), and errors. I always check any lesser known framework to see which ballpark it falls into. I'm always surprised to see that so many of the popular frameworks are ~10x worse than the best--although many of those 'best' don't do as much processing. I'm much more likely to pay attention to the error counts, max latency, or SD (σ).

supermatt 6y ago

Incredibly frustrating the benchmarks not linking to the actual frameworks - searching for "lithium web framework" gives me some PHP framework... Edit: Comment lower down suggests: https://github.com/matt-42/lithium https://github.com/matt-42/lithium is the correct one.

matt42 6y ago
I'll report this to the benchmark maintainers.
sitkack 6y ago
I came here to ask the same question, "Where is the page, with clickable links to the frameworks?".
- giancarlostoro 6y ago
  They really need this and a page explaining why certain frameworks are removed over time.
brightball 6y ago
Yep. That's ooooooooold. Lithium was a project as CakePHP was going through a big update 10 years ago.
bhauer 6y ago
We plan to collect additional metadata from maintainers, including home page URL, source URL, description, and so on. Once that happens, we'll be able to provide the kinds of helpful links you want.
- olaf 6y ago
  Until this is done, you could just give your visitors the download links you used to download the frameworks. Doesn't seem to me overly complicated.
  - 6y ago

bhauer 6y ago

Brian from TechEmpower here. I don't want to take away from these results, but I do want to provide an important piece of context. This link is to a rendering of data from a continuous run [1] that hasn't received the type of sanity checking we do for an official TFB round. You can tell from the title ("Test") and the UUID of the run visible in the gray box under the navigation. Based on this link's attention, we'll make a more prominent warning for renderings of continuous runs so that they are more obvious to readers not familiar with the project. We execute runs like this continuously to allow maintainers of test implementations to observe the results of their contributions. Given the performance seen here, it is very likely that Lithium will be well ranked in the next official round. But we feel the ranking seen in a continuous run such as this should be taken with a grain of salt until that next official round is available. [1] https://tfb-status.techempower.com/results/57b25c85-082a-4013-b572-b0939006eaff https://tfb-status.techempower.com/results/57b25c85-082a-401...

matt42 6y ago
Hi Brian, can't wait for round 20 then :D.
AtlasBarfed 6y ago
Is that why it didn't vary when I selected cloud vs physical?
- matt42 6y ago
  Yes, this is only physical. The cloud env (Azure) only run once every 1 or 2 months I think. On the official rounds they do both.

pknerd 6y ago

Thanks to Google, The PHP Framework, Lithium will get some unwanted praise here.

tluyben2 6y ago

Offtopic: matt42 (or others); where do I learn modern optimization like this? I can optimize for cycles on embedded MCU's and older computers and I can optimize in .NET/JVM (and others) but are there any good sources for OS on modern metal optimizations? Besides reading the Lithium sourcecode ofcourse.

matt42 6y ago
I would say the profiler output is the best source of information. Lithium is not using black magic at all. Actually on large framework like lithium, performance problems come more from obvious errors that get hiden in the quantity of code than in code that you did not crazy optimize. Only the profiler will tell you if there is a hidden malloc that is slowing down all the rest, or a inefficient datastructure, or ... There is probably lot of non optimized code in lithium, it just has no impact (or if it has I still has to find it). That said, for other kind of software (usually more CPU bound), going down to assembly or using SIMD can be the only way to get good perfs.

thdrdt 6y ago

Wow, the difference between .Net core and .Net!

RocketSyntax 6y ago

just ahead of jooby and kooby

dathinab 6y ago

I wonder how realistic/fair the code of some benchmarks is. I just took some small peak at atix (because I happen to know it) and while first all looked fine it wasn't quite that realistic. Mainly: - It uses a fork of tokio-postgres specific for the test (which differs in that Client is no longer send and it has a Unsafely soundness hole by wrongly using Unsafe Cell, and no Issue tracking enabled on, through replacing that cell with RefCell probably yield very similar performance) - Instead of using the default web::Json responder it uses simd-json (EDIT: explicitly encoding the data into a buffer instead of returning it wrapped in web::Json) - Uses snmalloc instead of the default allocator (https://github.com/microsoft/snmalloc https://github.com/microsoft/snmalloc) Just to be clear besides the first point all of this is not unrealistic for a context where you want to highly optimize your server. But not how actix is used most of the time. --- Edit: Also just to be clear I didn't nit pick intentional on actix, it just happen to be the framework I'm more familiar with and especially using snmalloc for actix seems to be quite a reasonable idea. EDIT2: The Unsefety unsoundness isn't triggered given the way this library is used, it's still there and would prevent this change from ever been merged upstream.