14 ms·
Web Framework Benchmarks Round 4
- bhauer 13y agoThis is the most recent update to our ongoing project measuring the performance of web application platforms and frameworks. In this round we've received several more community-contributed tests in Perl, PHP, Python, Java, and JavaScript. Go is a comeback champion thanks to changes made by Brad Fitzpatrick [1] and others in the Go community. A new "Fortunes" test was also added (implemented in 17 of the frameworks) that exercises server-side templates and collections. With 57 total frameworks being tested, we have implemented some filtering to allow you to narrow your view to only those you care about. As always, we'd really like to hear your questions, suggestions, and criticisms. And we hope you enjoy this latest round of data. [1] https://code.google.com/p/go/source/detail?r=45c12efb46 https://code.google.com/p/go/source/detail?r=45c12efb46
- nopal 13y agoAny reason you didn't test ASP.NET MVC or ASP Web Forms?
- JulienSchmidt 13y agoMaybe because the server is not running Windows? Edit: yeah right Mono and w/e
- cmircea 13y agoWould it be too much to ask to also run a Windows VM? Or use Mono?
- platz 13y agohttp://www.techempower.com/benchmarks/#section=motivation http://www.techempower.com/benchmarks/#section=motivation "As with the previous question, we'd love to. We have heard tentative word from a reader/contributor that a pull request may be incoming soon that will include several .NET frameworks on Mono, which we assume will be as easy to include as any other pull request. One challenge we face is that the test infrastructure we've built assumes a Linux deployment that we can automate using ssh and Python. To do a proper .NET test on Windows Servers, we will need to work on adapting that platform to automate Windows Servers as well. Community assistance on this would be greatly appreciated."
- iknight 13y agoCan you share who the Mono contributor is?
- bhauer 13y agoSure. We now have two pending issues, one of which is a pull request. If you can weigh in and perhaps contribute some review time to the PR in particular, that would be very appreciated! https://github.com/TechEmpower/FrameworkBenchmarks/pull/239 https://github.com/TechEmpower/FrameworkBenchmarks/pull/239 https://github.com/TechEmpower/FrameworkBenchmarks/issues/158 https://github.com/TechEmpower/FrameworkBenchmarks/issues/15...
- voidlogic 13y agoLinux amd64 is the dominate internet serving platform; supporting a non-free OS for the benefit of a single test entry seems dumb. I'm all for a Mono entry however.
- diroussel 13y agoI don't think it seems dumb. It would be a very useful comparison between platforms. Stackoverlfow runs on .Net and Windows servers and they say it's very performant. So why not compare with other frameworks on the same hardware.
- bhauer 13y agoVoidlogic is right that we are waiting to get a pull request that will include some .Net frameworks [1]. If you can help, it would be greatly appreciated. We do want to include .Net. We will test on Mono to start. We also want to test on .Net's native Windows platform. But we need to work on the testing platform we've built in order to automate a Windows server in the same way we presently automate a Linux server. [1] https://github.com/TechEmpower/FrameworkBenchmarks/issues/158 https://github.com/TechEmpower/FrameworkBenchmarks/issues/15...
- AndrewGaspar 13y agoI had the same thought. ASP.NET MVC and Web Forms are both very popular frameworks, and they're free to develop for. Might be harder to set up on a non-Windows machine, which could be why they were not tested.
- voidlogic 13y agoThey were not tested because no one has submitted a pull request for them.
- pfalls 13y agoWe would like to have a .NET test running on mono. We were hoping to get a pull request for round 4, but unfortunately we have yet to receive one.
- apalmer 13y agoASP.Net kind of put themselves out of the benchmark game here: Mono Issue #1, since the vast vast majority of ASP.Net websites run on windows a Mono performance test even if accurate is going to be of dubious value. Mono Issue #2, since Mono is nowhere near as polished as the Microsoft .Net implementation the numbers wont really be meaningful. Windows Issue #1, if you do the test on a different OS than every other test implementation, the results really wont be comparable in any fair way. Microsoft Issue #1, dont know if it still holds now a days but in past official EULA for .Net prohibited publishing benchmark results. PERIOD. I am a .net developer and as much as I like ASP.Net I dont think the effort of adding a .Net implementation really would pay off.
- bsaul 13y agoI completely understand your point, but I think it's fair to say that most .Net code will run on Windows server, and that pretty much everything else will run on some kind of linux flavor. Just like you have a "keep the default framework setting" approach to help compare very different frameworks because that's how the majority of people will use them, you may very well assume that comparing frameworks on their preferred OS is fair enough. I know that i wouldn't mind switching to a windows+.Net environment if it proved to be much much faster than what i'm using right now.
- DennisP 13y agoExactly, and if there is such a performance boost, having the numbers would help you figure out whether hardware savings would be worth the licensing cost.
- voidlogic 13y agoCare to backup your Mono claims with evidence? I have a friend who went to a Mono talk at MS MIX where a Mono developer was speaking. The Mono developer said that while Mono is a little slower than .NET (and he was talking a couple percent) Mono often ends up being faster on the same hardware because the Linux system calls the runtime uses are faster. There were a few ROFLs in the audience. I also agree with you that investing in a .NET (Windows) test isn't good bang for the buck here.
- diroussel 13y agoThe requests per second is importanct, but some frameworks seem to get high average throughput but at the expense of a few slow requests. Also when measuring latency, average and std dev are only revelent if the distribution is guassian in distrition. Which is unlikely. Better to show percentile based measurements. Like 90% of all requests served in 5ms, and 99% of requests served in 15ms. See Gil Tene's talk "How not to measure latency" [1] for more info. Also be sure you are not falling into the "Coordinated Omission" trap where you end up measureing the latency wrong. [1] http://www.infoq.com/presentations/latency-pitfalls http://www.infoq.com/presentations/latency-pitfalls
- bhauer 13y agoHello diroussel, Thanks for the feedback! We started the project with WeigHTTP, then starting with Round 2 we switched to Wrk [1] at the advice of other readers. Wrk provides latency measurements consisting of average, standard deviation, and maximum. See the earlier conversation about standard deviation here: https://news.ycombinator.com/item?id=5455972 https://news.ycombinator.com/item?id=5455972 If we had distribution data available, we would aim to provide that in some form. And perhaps the author of Wrk could add that in time. However, for the time being, I consider the matter somewhat academic. Not to be dismissive--I value your opinion--but I don't believe that would measurably impact my assessment of each framework's performance. Though, it would be fascinating to be able to validate my suspicion that Onion, being written in C, does not suffer even the tiny garbage collection pauses of the Java frameworks. [1] https://github.com/wg/wrk https://github.com/wg/wrk
- diroussel 13y agoOk, I've raise an enhancement for wrk https://github.com/wg/wrk/issues/31 https://github.com/wg/wrk/issues/31 Perhaps you could upvote or something? Thanks for all the great work in these benchmarks. A useful resource.
- tlarkworthy 13y ago"average and std dev are only revelent if the distribution is Gaussian in distrition" technically not true. Knowledge of the second order moment (variance) lets you uniquely identify other distributions like Poisson, or uniform. Knowledge of even higher order moments lets you fit more complicated statistical models. Low variance is good, regardless of underlying distribution.
- laumars 13y agoThere's two main mysql drivers for Go mymysql and go-mysql-driver. I've found the concurrency performance of the former to be abysmal when doing my benchmarking. Then the moment I switched to the latter, Go's performance went through the roof.
- blissofbeing 13y agoFor those wanting the link straight to the benchmarks its here: http://www.techempower.com/benchmarks/#section=data-r4 http://www.techempower.com/benchmarks/#section=data-r4
- Goranek 13y agoWow Go is getting faster and faster
- axlerunner 13y agoGood to see Java is holding its own.
- exabrial 13y agoYah, I noticed nearly all of the top frameworks are Java or JVM based. Pretty cool! I just noticed something, the two major JSF2.0 implementations MyFaces and Mojarra are both missing.
- TheKickOff 13y agoI was just thinking the same thing. Someone should definitely contribute a JSF test and if possible it should be run on both those implementations.
- bjhoops1 13y agoReally. I've recently been getting into node/express after years of Java, and these results make me feel a whole lot less cool.
- exabrial 13y agoWell no matter what you use, you should always benchmark for that exact reason. Making things go fast takes a lot of performance engineering, and the JVM has millions of dollars worth of tuning put into it. BTW, if you like Node.js, you should probably look at Vert.x. I haven't used it, but it's a similar concept, it runs on the JVM, and it seems to spank Node.js.
- tlear 13y agoWhen you benchmark Vert.x on a multicore system vs 1 node process. Now add multiple node processes to occupy the cores and store changes completely
- aphyr 13y agoIndeed. Node's IPC costs are orders of magnitude slower than Java's volatile or atomic datatypes. I would only choose Node for stateless or trivially parallelizable problems--e.g., those where I could push the state problem into a runtime with real threads. http://aphyr.com/posts/244-context-switches-and-serialization-in-node http://aphyr.com/posts/244-context-switches-and-serializatio...
- camwest 13y agoWhat version of the programming languages are being used?There is a major difference between Ruby 1.9.x and Ruby 2.x
- krg 13y agoRuby 2.0.0-p0 http://www.techempower.com/benchmarks/#section=environment http://www.techempower.com/benchmarks/#section=environment
- hilem 13y agothough they're also using Rails 3.2.11, which not only isn't the latest 3.2.* release but doesn't take advantage of Ruby 2.0... it would be more interesting to see the latest Rails 3.2.* on Ruby 1.9.* along with the latest Rails 4 release candidate on Ruby 2.0.0-p0
- oberhamsi 13y agoI think he is too polite to say: "patch please"
- JoshGlazebrook 13y agoSo move it up 2 or 3 spots then...
- krg 13y agoI created a Github issue to address the upgrade to 3.2.13, and pfalls is working on getting this done. https://github.com/TechEmpower/FrameworkBenchmarks/issues/247 https://github.com/TechEmpower/FrameworkBenchmarks/issues/24... I encourage you to benchmark Rails 4 yourself and see if there is a measurable difference from the latest 3.2.x, to get a preview of the impact it will have. When Rails 4 is released we'll definitely want to upgrade to that. Edit: pfalls is too fast for me, and just completed the upgrade to 3.2.13 and closed the issue. The next round will use Rails 3.2.13.
- geekam 13y agoSome of the PHP frameworks do better than Python frameworks.
- artursapek 13y agoWow, I should really give Go a try.
- Goranek 13y agoI'm thinking the same. This summer vacation is reserved for Go :)
- schumacher 13y agoI also will give Go a try :-)
- swah 13y agoAgreed, but the language is sufficiently (edit) MORE verbose than Python that it is still a valid strategy to write in Python first, and then rewrite in Go for performance..
- artursapek 13y agoWow. Great to hear. My current big project is all in Python. Maybe a rewrite is in my future if I find the project successful.
- swah 13y agoI mean't more verbose, sorry.
- gtaylor 13y agoIt translates fairly directly as well, even with the differences in OO and multi-processing/threading.
- papsosouid 13y agoIf go is less verbose, then why write it in a more verbose language first? The reality is, go and python are pretty much even for verbosity, and there is absolutely no benefit to writing your app in python first and then go. You'd be better off writing it in go and then rewriting it in go if you want the benefits of learning from your first attempt.
- rustc 13y agoI wonder why the 9 lowest ranked frameworks are all in PHP...
- showerst 13y agoNot that PHP is particularly fast, but I notice that Raw PHP does about middle of the pack, but all of the ORM versions seem to do terribly. Interesting stuff. I've usually had pretty good luck with Slim, I'll have to try a version with & without redbean and see how big a difference it makes.
- cmircea 13y agoProbably because all those big frameworks have to initialize everything for every request. Parsing, connections, configuration, you name it.
- showerst 13y agoYeah, of course. With slim in particular I notice that the benchmarks list it as "Raw database connectivity" but in the code it looks like it's using RedBean ORM. I'll look more at lunch, i'm probably just misreading something. Although obviously ORM is more realistic, since if you're sophisticated enough to be using composer and a framework, you're probably using an ORM. I know the point of this benchmark is frameworks not ORMs, but it would be interesting to swap them out and see if there's a huge difference.
- notJim 13y agoRaw php also has to initialize connections, and it does really well, so I don't think that's it. To me, that points to overhead from initializing the objects as the bottleneck. I would have thought that with APC, this wouldn't be a major issue, though, so I wonder if that's still not it.
- egeozcan 13y agoI always hear this myth that php gets very slow when you write proper OO code. I neither write nor maintain any php code so I wouldn't know.
- rallison 13y agoFor those new to these benchmarks, all of the source code is up on github: https://github.com/TechEmpower/FrameworkBenchmarks https://github.com/TechEmpower/FrameworkBenchmarks We welcome all pull requests, suggestions and criticisms.
- exabrial 13y agoAny chance you could add JEE6? The two major JSF 2.1 implementations MyFaces and Mojarra are both missing. I don't expect awesome performance numbers, but since JSF2.1 is the 'official' (sigh) web framework of JEE6, it would be interesting how awful they are compared to some of these other languages. I guess though if this is only testing JSON serialization, it may not make sense. Perhaps adding JAX-RS implementations like CXF, Jersey, RESTEasy, and RESTLet would be more appropriate.
- pfalls 13y agoThe latest test that we have, dubbed "Fortunes", does server-side templating, so we (or a generous community member) could add a JSF implementation for that test.
- henk53 13y ago>it would be interesting how awful they are compared to some of these other languages. The World Wide Wait benchmark, which tested quite a lot, showed quite favorable numbers for both JSF implementations.
- ipostonthisacc 13y agoLua is impressive - the first as a non-compiled lang. Is that Lua or Luajit in OpenResty?
- saosebastiao 13y agoI love what you guys are doing. This is by far the most comprehensive (in terms of number of frameworks) comparison of web frameworks. I also am a big fan of the new filtering metadata. However, I'm starting to think that all of the advocates of various frameworks are now conspiring independently to make this comparison meaningless...any framework (except Cake for some reason) can be superoptimized towards a small set of tasks. If you do another round, could you increase the number of different tasks? Some examples could be: 1) Mixed bag of queries of various complexity 2) Static file serving 3) A few computation/memory-intense benchmarks (such as those in the Language Benchmarks Game) 4) Templating
- pfalls 13y agoThanks for the kind words. We're very interested in adding additional tests, this round even includes a new test dubbed "Fortunes" which does in fact do server-side templating. We have an open github issue[1] asking for the community's input for just this sort of thing, and we'd love to have your feedback included. [1] https://github.com/TechEmpower/FrameworkBenchmarks/issues/133 https://github.com/TechEmpower/FrameworkBenchmarks/issues/13...
- deleted 13y ago[deleted]
- bhauer 13y agoHi saosebastiao, As Pat points out, we definitely look forward to implementing some more computationally-intense request types in the future. This round does include the first server-side template test. We'd like to hear the community's opinions about more tests. That said, I feel most of the frameworks' implementations of the existing tests are not cheating. Our objective in this project is to measure every framework with realistic production-style implementation of the tests. No doubt there is temptation to trim out unnecessary functionality and focus on the benchmark's particular behavior. We have attempted to identify any such tests that remove framework features to target the benchmark as "Stripped" and those can now be filtered out from the list. In other words, our aim is that the implementation of each framework's test is idiomatic to that framework and platform. And if that's not the case for a test, we want to correct it. Your concern could be clarified by pointing out that framework authors may be tuning up their JSON serialization, database connection pools, and template processing in order to improve their position on these charts. And, to be clear, I have already seen evidence of that in my interaction with framework authors. To that concern, however, I would say: That is awesome. I want those features to be fast.
- mitchi 13y agohttp://openresty.org/ http://openresty.org/ Looks interesting but at a quick glance it looks like you are programming in configuration files? I'm not sure I like the idea.
- dataminer 13y agoYou can try Lapis http://leafo.net/lapis/ http://leafo.net/lapis/ its a new framework built on openresty.
- plorkyeran 13y agoIf you're building an actual webapp I assume you wouldn't put anything more than a require and a call to the app's entry point in the config file. Putting nontrivial amounts of Lua in the config file is more of a way to stave off the evolution into terrible turing-complete languages that config files often make.
- snaky 13y agoYes. "Nginx programming" is not a joke for a last couple of years in HighLoad world. Before ngx_lua (OpenResty is nginx plus ngx_lua module basically) you had to do it in C.
- kclay 13y agoWhy I'm still using Play! these results are disappointing once again, atleast beat node.js. Go is looking great though.
- raphaelj 13y agoPlay! gives not so bad performances if you take into account that it uses a full ORM whereas others top performers don't.
- candybar 13y agoRaw comparisons are quite misleading - ridiculous setups or slow interpreted languages aside, full stacks with ORM are going to be slower than micro stacks with raw DB access. Play! does fairly well compared to other frameworks of its kind.
- terhechte 13y agoI've dabbled a bit with Play, but found it too slow (with Scala). Have a look at Scalatra or, if you don't need a full blown framework, Spray
- room271 13y agoOr Unfiltered which performed very well in the tests too.
- gtg123 13y agowhich test are you referring to? as far as I can tell node pretty much beats play-XX in all tests
- aualin 13y agoIs it Lua or LuaJIT?
- camus 13y agoLuaJIT
- qingu 13y agoThanks for doing these extensive benchmarking tests. It would be really helpful to see a more complex example that includes user authentication. Aside from the benchmarks it's also a really good starting point to compare the code in different languages and get a first impression of a framework. On a side-note, I'd really like to know why so few start-ups seem to be using Spring. It could be just a wrong impression . But from what I have seen most start-ups use RoR or Django. My guess is that Spring is less flexible and less known outside big companies, where it is usually the default. It could also be that Spring works better with the waterfall model whereas Django or RoR are better suited for explorative programming and that fits the respective spheres better.
- sghill 13y ago> It could also be that Spring works better with the waterfall model I've used spring mvc in an agile setting a couple times now, and it has worked fine. It doesn't tend to make developers all that happy, in my experience. If you're in an enterprise full of spring, starting up the next app with it can be attractive -- there likely already exists a bunch of tooling and knowledge around spring. I wouldn't use spring directly if I were trying to build something quickly for a startup. I'd be more apt to reach for grails (which wraps spring), dropwizard, or any of the other rapid-development frameworks.
- redtuesday 13y agoMaybe that's me, but I think it's easier to learn typical web frameworks like Rails, Django etc. than spring. On top of that the xml config sucks (at least in my opinion - though I used spring the last time around 2007, maybe it's not as bad as back then).
- coldtea 13y agoIt's impressive how well PHP holds up with many queries per request (which is the most common CRUD/webapp scenario). While for no or just one query it's slower than a lot of the other frameworks (due to PHP being slow to parse, startup etc), as soon as we have a lot of DB queries, the C interface to MySQL leaves the other frameworks in the dust. The well known PHP shortcomings aside, that's a nice example of optimizing for the things that matter most, especially for it's common use cases (Wordpress, Drupal, etc).
- EGreg 13y agoIn really scalable sites, you need sharding. Unless your database itself is doing the scaling (such as with Riak), you're going to sometimes hit multiple shards. With PHP and other languages that can't do async, you're going to have to query the DB sequentially, increasing latency proportionally to the number of shards you have to hit. With Node.js and other asynchronous apps, you don't. Disclaimer: mysqli does have async capabilities, but most people such as myself use PDO for its other benefits. And mysqli only works with MySQL.
- rcoder 13y agoWhat you need for sharded queries is concurrency, not necessarily asynchronous requests. Async callbacks (ala Node.js, Twisted Python, Event Machine, etc.) give you a kind of cooperative multitasking, which is one way to have concurrent I/O-bound tasks going; multithreaded programs are another. (Ruby and Python threads are kind of in-between, due to their respective GIL limitations.) That being said, above a certain scale and complexity level, you probably want the topology of your persistent data store hidden from your web request handlers anyway. For one thing, making requests to N backend shards from M frontend web workers starts to get bad when N and M are both large; for another, introducing really complex scatter-gather query logic into your request-handling pipeline can be a maintenance and debugging nightmare. Introducing a proxy or data-abstraction service in between cuts down on the number of open connections and lets you change the data storage topology without updating frontend code.
- bhauer 13y agoSome of the fastest implementations you see in these tests are not asynchronous. With Servlet for example, a worker thread is chosen from Resin's thread pool and used to handle a request. The Servlet then executes 20 queries sequentially and returns the resulting list data structure. This is Servlet 3.0 but not using Servlet 3.0 async. Async isn't making the top performers fast. Being fast is making them fast.
- reactor 13y agoOff all the top performers, Go seems to be the only sane choice to write a web app. Moreover it is at the sweet spot; expressive, flexible, simple, super performant, good community etc. I think it is convincing enough for me to give Go a serious look for our new app.
- gtaylor 13y agoAs someone who is also really enjoying Go, I think you need to add a huge, gigantic disclaimer before making a statement like this: Go's ecosystem of web development packages is in its infancy. You're not going to find any super-well-documented, super mature/stable web frameworks (thought a few are showing great promise). and some of the individual components (for example, Gorilla) are looking very good, but still have some more cooking to do. I love the language, but let's not get too carried away until the ecosystem grows. The reality is, if you're going to use Go for web dev, you're going to need to be prepared to do a whole lot of things on your own.
- reactor 13y agoI've read about the limitation of Go in terms of third party libs, but I believe it is only temporary. I had a glance at the std libs, looks really good for a new language. Our new app doesn't require all the bells and whistles of a full fledged framework.
- voidlogic 13y ago>>I've read about the limitation of Go in terms of third party libs Must not be looking on github... Yes its new, but in a majority of cases I have not had any trouble finding third party libraries and the std ones are wide and excellent.
- mrtksn 13y agohow easy is to use GO with other frameworks, let's say PHP? Would it be possible to write an application that uses PHP for some tasks, so you can benefit from the speed of GO and the the maturity of PHP?
- cloverich 13y agoI know the following is small, but surprised Pyramid didn't make the cut. I've used it for a couple of projects outside of work, and among the several other web frameworks I've worked with (Struts, Play, Django), it seems relatively mature and well documented. http://docs.pylonsproject.org/projects/pyramid/en/1.4-branch/# http://docs.pylonsproject.org/projects/pyramid/en/1.4-branch...
- voidlogic 13y ago>>didn't make the cut Did you submit a pull request for it and it was denied?
- manulp 13y agoAny chance of seing the python frameworks behind uwsgi instead of gunicorn?
- pfalls 13y agoHere is the issue for just that: https://github.com/TechEmpower/FrameworkBenchmarks/issues/114 https://github.com/TechEmpower/FrameworkBenchmarks/issues/11... If you're interested, we'd love to have some help getting this accomplished.
- freework 13y agoI really don't like these benchmarks. Its like benchmarking Fizzbuzz or something. Frameworks don't do anything. No one chooses a framework (at least I don't) based on performance. You choose one framework over the other because you like the API and/or language. I myself am a framework author (giotto, a python framework that was not included in these benchmarks). If my framework had been included, I'm sure it would end up dead last. When I built it, I wasn't thinking about performance, I was focusing on building a framework that would result in applications that are easy to understand/debug and fast to write.
- deleted 13y ago[deleted]
- dllthomas 13y agoI agree that performance shouldn't dominate the decision, but there's no reason not to be informed by it - it can wind up mattering.
- phillmv 13y agoExcept benchmarking is really, really hard to get right, and these benchmarks aren't really testing anything that resembles a production app. For all non-trivial apps, by the time you get 100 req/sec your bottleneck is very likely going to be your database.
- mrgoldenbrown 13y agoThis is a good point. Especially if you only cared about how fast you can make your app. But if you want to also consider how cheap you can run your app, you need to consider how many app servers will it take to saturate the DB? 1 or 10? At certain scales for certain tasks, the hosting costs matter more than the development costs.
- phillmv 13y ago>But if you want to also consider how cheap you can run your app, you need to consider how many app servers will it take to saturate the DB? Moore's law has made this sorta moot. Unless you're on Heroku, for a successful small-to-medium app, the denominator in your hosting costs is doing to be the salary of the engineer or sysadmin who tends to it. (If you're on Heroku, then you start worrying about dynos because, with monitoring, you're paying $60 per "worker".) This is to say, the cost in salary to properly shard a database probably outweighs a year or two of hosting for the extra two or three boxes you're spinning up; almost no one experiences explosive growth where you need to spin up dozens of new boxes overnight.
- deleted 13y ago[deleted]
- hippich 13y agoWhenever I hear "Java" I also get association "slow". But looking into this list - java web frameworks doing incredible job!
- why-el 13y agoI always assumed the slowness is associated with build times and such. You are slow developing a Java web app, but not running it.
- mikkelewis 13y agoand GUIs.
- pjmlp 13y agoNot when using GUI designers.
- marcosdumay 13y agoAt the time Java gained its fame, the GUI applications were unresponsive, not slow. They could lock at any time, for a perceptible few hundreds of miliseconds, but had still a nice average speed.
- exabrial 13y agoYou ought to put together a benchmark for that!
- danieldk 13y agoNot really, major IDEs have had incremental compilers for ages. Usually Maven compiles of our largest projects are also done in no-time.
- peeters 13y agoJava gets that wrap from originally being slow to execute, and also having a huge up-front cost to spin up a VM. The first isn't true any more: the Java VM competes with native code on most benchmarks, and due to its ability to perform runtime optimizations, can occasionally outperform native code. The second doesn't matter at all for web servers. The cost of starting up the web server is tertiary to uptime and performance. If the thing is going to run for 4 months without going down, who cares all that much if it takes 5 seconds or 5 ms to start up?
- tlear 13y agoSince this is multi core test, how many instances of Node are running actually?
- rzimmerman 13y agoI was wondering the same thing. If they're running on a quad core (or EC2 Large) the results could look pretty poor with a single node instance.
- pfalls 13y agoWe use the cluster module to spin up multiple instances. We cap it at one instance per core.
- Xeoncross 13y agoWhat the‽ How is it possible my MicroMVC framework + ORM is faster than Native PHP + ORM?
- bhauer 13y agoMicroMVC is doing very well! It is possible that your ORM is more performant than PHP ActiveRecord, which is what we used in the PHP + ORM test.
- escaped_hn 13y agoWhen is gemini going to be released?
- niggler 13y agoHow are multiple cores handled? For example, with node.js, the standard way to scale is to run multiple instances (but according to https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/nodejs/setup.py https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast... only one instance is run). I wonder how the results would change if only one or two cores were enabled (using taskset or isolcpus)
- pfalls 13y agoWe use the cluster module to handle multiple workers (https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/nodejs/hello.js https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...), so far we've received feedback that this is a valid approach.
- dthakur 13y agoI asked for this before. Could you please add error bars to your plots?
- kainsavage 13y agoThere is a column labeled 'errors' for this, but basically only Lift and Phreeze had errors.
- shellac 13y agoOP means 'error bars' in the sense of giving and idea of the accuracy of the measured value (e.g. standard deviation, standard error).
- zix 13y agoIt's there. The latency tables show the standard deviation, as well as the maximum latency.
- ihsw 13y agoIt's very depressing to see Symfony2 at the bottom of these lists. Although there are quite a few performance optimizations that can be done to improve this, there are few excuses for such poor performance by default.
- Ecio78 13y agoAnd just to be happier, Drupal 8 will be base on Symfony2...
- Narretz 13y agoDrupal has always been slow out of the box, so it's not news that it must be heavily optimized.
- krg 13y agoIf you have improvements to the test (that would be realistic for a production deployment) then please submit a pull request. We definitely want to show each framework doing the best it can do. A few frameworks have a "stripped" version (just Django and Rails so far) to try to show the best that can be achieved when typical functionality is stripped out. Essentially optimizing for this test, which is interesting even if it isn't the point of these benchmarks. If you think Symfony2 would benefit from a separate "stripped" test then please consider a submitting a pull request with that.
- redtuesday 13y agoA stripped test is on the way it seems. One user submitted a raw test a few days ago which doesn't use doctrine.
- deleted 13y ago[deleted]
- gionn 13y agohttps://github.com/TechEmpower/FrameworkBenchmarks/issues/230 https://github.com/TechEmpower/FrameworkBenchmarks/issues/23...
- derengel 13y agoCan there be a sinatra benchmark with raw database connectivity?
- redtuesday 13y agoYeah, you could submit the test to them for example . They accept pull requests. ;o)
- codeulike 13y agoIs it linux only? Would have been nice to be able to compare c#/MVC
- iknight 13y agoI hear they are accepting pull requests... https://github.com/TechEmpower/FrameworkBenchmarks https://github.com/TechEmpower/FrameworkBenchmarks
- codeulike 13y agoAh yes, someones added aspnet_mono now, which seems to be MVC based https://github.com/TechEmpower/FrameworkBenchmarks/tree/master/aspnet-mono https://github.com/TechEmpower/FrameworkBenchmarks/tree/mast...
- seldo 13y agoFeature request: it would be nice to have permalinks (even if they were really messy URLs) to filter-sets so that I can share the slice of the benchmarks I'm looking at with people, without having to list the filters manually.
- bhauer 13y agoAgreed! I had hoped to do that in this round but simply ran out of time. I will make it a priority. I'd love to allow people to share specific comparisons with one another.
- ninjazee124 13y agoGuess Lift is not all that high performance framework that they claim it to be!
- SkyMarshal 13y agoDepends on whether any Lifters have optimized it for these tests or not, which, after skimming the github repo and commit history, does not appear to be the case. Same with any of the other frameworks - some have been optimized by their fans, others are running in default configs. Techempower should add a filter to show only frameworks that have been optimized.
- krg 13y agoAt this point, most of the tests were contributed/improved by the community. Not all have been reviewed by experts in that framework, and I agree that an "expert reviewed" marker would be nice. That said, we have tried to not run anything in the "default" configs, but rather the "production deployment" configs if we could find documentation on that. Unfortunately there is a huge variability across frameworks in how good the "production deployment" documentation is.
- SkyMarshal 13y agoHave you guys reached out directly to the mailing lists/google groups/github issues/etc of all the frameworks you're testing, asking for experts to take a look? I know some of these communities don't really frequent HN and Reddit, but they all frequent their mailing lists.
- desireco42 13y agoWhile I really enjoy results, I still would prefer to see smaller number of frameworks with specific tasks that are common in web development, logging in (assume user and pass are provided), listing of something. These two would be way more meaningful then anything else to me. Again, thank you very much for hard work. I think for some there are some revelations there, like that C framework.
- bhauer 13y agoWe aim to add more tests in time. Take a look at the new Fortunes test. That test lists rows from a database, sorts them, and then renders the list using a server-side template. It's only implemented in 17 of the 57 frameworks right now, but we hope to have better coverage on that in time as well.
- desireco42 13y agoYes something along those lines was what I was thinking. Thank you.
- pjmlp 13y agoWhat about vibe.d? While D is listed, vibe.d seems to be missing.
- bhauer 13y agoGood catch. I have "D" listed because I heard from the vibe.d community [1] that a pull request was incoming. We have not yet received it. Can you help the vibe.d guys get it submitted? [1] http://forum.dlang.org/thread/urpqdftuofgwespkcdxg@forum.dlang.org http://forum.dlang.org/thread/urpqdftuofgwespkcdxg@forum.dla...
- pjmlp 13y agoI am active on D forums and remember reading about it. I guess everyone is busy because of the DConf 2013 http://dconf.org/ http://dconf.org/ Otherwise I will remember them next week.
- bdesimone 13y agoI'd like to thank you for the effort, and genuine openness of this project. I hope you'll continue to expand these tests.
- Periodic 13y agoFirst off, I love the work you're doing, keep it up. Benchmarks like this are designed to be the starting-point of a discussion an investigation, and not as anything meaningful in their own right. Boiling it down a framework to one performance number ignores the many, many nuances of a framework. What surprises me most is the difference between different frameworks. A few years ago the mantra seemed to be "Use Rails, Django or a similar full-stack framework. Speed of deployment trumps everything!" Over the last few years I've seen a shift as people are trying to get more performance from limited hardware. Personally I'm intrigued by how a fairly innocent decision early in the project (of what language/framework) may have profound performance implications in the long run. For myself, I've been looking for a good functional-programming framework. Just looking at this gives me a good list of frameworks to start looking at. It feels to me that a framework that performs well is likely well engineered, so the ones that perform better will go at the front of my queue for investigations.
- bhauer 13y agoThanks, Periodic. It's especially rewarding to hear that people have gleaned value from the project. You're precisely right about how to put this data to use: as one point in a holistic decision making process. We address that in the Questions section of the site, in fact. That said, we are not reducing each framework to a single performance number. Our goal is to measure the performance of several key components of modern frameworks: database abstraction and connection pool performance, JSON serialization, list and collection functions, and server-side templates. We'd like to add even more computationally-intensive request types in future rounds. So, no, we're not testing your (or anyone else's) specific application on each framework. But we are testing functions that your application is likely to use. You're still better off measuring the performance of your use-case on candidate frameworks before you start work, but perhaps you can first trim the field to a manageable number. In the first round, we echoed your surprise at the spread--four orders of magnitude! I think the shifting winds of opinion come from the fact that today's high-performance languages, platforms, frameworks are not necessarily more cumbersome to use for development than the old guard. As others have pointed out elsewhere in this thread, Go is not a terribly verbose language, and yet its performance is fantastic. Has the era of sacrificing performance at the altar of developer efficiency ended? I'm not sure. But we have some data to add to the conversation.
- buster 13y agoIt'd be nice to also have a "lines of code" comparison, because i can see how some benchmarks are at the top but after looking at the code they also tend to be quite "manual".
- bhauer 13y agoWe do plan to start tackling that other dimension, efficiency. Source lines of code is one objective measure that, for all its faults, we plan to include. In a previous round, I pasted the relevant code directly into the results view. I will likely do that again soon since it's convenient for the reader. For the time being, I invite you to browse the Github repository and examine the test implementation source.
- voidlogic 13y agogzipped source size in generally considered a better measure of effort than actual LOC. The reason for this is that the languages with a lot of boiler plate usually have tooling, IDE's etc to make that pain go away, and gzipped source, while still counting repeated boiler plate end up weighing it less than unique lines of code.
- oberhamsi 13y agoI implemented the Ringo app for this benchmark and of course ran it against Node and a couple of others to see how we would perform in this neighborhood before I opened the pull request. And since that day I've been wondering: why does NodeJs (=V8 JS engine in C) talking to MongoDB have higher response times and latency than Ringo (=Rhino JS engine on JVM) talking to MySQL. The only thing where Node beats us JVM guys seems to be the JSON response test.
- ricardobeat 13y agoNode favours concurrency over raw speed; calls deferred with process.nextTick and callbacks end up costing time, in exchange for better concurrency. I think a blocking driver could leave Ringo in the dust, but it would be useless.
- oberhamsi 13y agoI will test that. Should the latency stay more constant with higher concurrency? Or what am I searching for? Looking at the res/seq we got from round4. In order of concurrency (8, 16, 32, 64, 128, 256): nodejs (mongodb raw) 12,541 22,073 26,761 26,461 28,316 28,856 ringojs (mysql raw) 13,190 23,556 27,758 30,697 31,382 31,997 both look like they got room to grow
- rdtsc 13y agoI think 256 is too low. It should at least start at 256 and go to 1024 perhaps. Then also try multiple CPUs 1 to 4 to see how it scales across.
- bhauer 13y agoHi rdtsc, Until the project includes a WebSocket-enabled test or a test with forced idle time (e.g., waiting for an external service to provide a response), concurrency higher than 256 yields very little of interest. The reason being that we are fully saturating the server's CPU cores at 256 concurrency [1]. Increasing the client-side concurrency level simply means that the front-end web server (or built-in web server's socket listener thread) needs to maintain a small queue of requests to hand off to the application server's worker threads. It doesn't make the server any faster at completing those requests. I've written some more about this at my personal blog [2]. [1] Caveat: Some frameworks appear to have locking or resource contention issues and do not saturate the CPU cores. We will attempt to capture CPU utilization stats in future rounds since this might be of interest to readers and framework maintainers. But increasing concurrency would not increase CPU utilization in these scenarios either. [2] http://tiamat.tsotech.com/rps-vs-connections http://tiamat.tsotech.com/rps-vs-connections
- Spiritus 13y agoWould love to see http://luvit.io/ http://luvit.io/ in there, to see how it holds up against Nodejs.
- minamea 13y agoI wonder if anyone can put a D lang web framework on there? Vibe.d seems to be the only viable one: http://vibed.org http://vibed.org
- just2n 13y agoBefore looking at the benchmark results, I took a glance at the Node source and I expected it to perform worse than it did previously. It does almost universally. Not only haven't the glaring perf issues remained since round 1, it's added more. In the real world, when you look at a metric that says your req/s is a bottleneck, which is what this benchmark is loosely simulating, you'd fix it. You wouldn't just say "nope, that's what this framework does, sorry boss." I still don't find these benchmarks very useful. From the looks of the comments, a lot of you don't really either (even if you don't realize it). For example, a lot of people in these comments want to correlate language speed with performance in these benchmarks, by arguing specific examples, but comparing almost any two frameworks/platforms in this "benchmark" is an apples to non-apples comparison, and the result is actually full of counter examples (faster languages performing more poorly). That should instantly tell you that this benchmark isn't telling you what you think it's telling you, and that you haven't really derived any value from it. Perhaps the biggest reason I don't find value here is that every product here does wildly different things. It's like comparing wrenches to hammers to screwdrivers to 3D printers. I also want to point out to people who say that this is a "comparison" of frameworks that it is emphatically not a comparison. What is the value of a framework? Is it speed? Atypically. And this "benchmark" tends to point at such cases as "being better" because they do better in this specific task. A framework/platform's value lies in features and abstractions. This does not compare those. I will gladly build a "framework" in NodeJS that is only capable of doing the tasks in this benchmark as fast and with as little overhead as possible. You would NEVER use it in the real world, but it would be a beast at serializing JSON and making repeated database queries in an insecure fashion. But score here is the important factor, right?
- matt2000 13y agoIn my opinion you've missed the point almost entirely: 1) If you see problems with a language you're an expert in, submit a pull request. I've never seen a benchmark done like this before, it gives everyone a chance to fix problems in their favorite framework/language. 2) It is a little bit of a unfair comparison between very low feature frameworks to higher ones, but it gives you a good idea of what you're trading off on basic performance. For example, I thought our use of play1-java wasn't far off of servlet on basic tasks, but boy was I wrong, perhaps by 10x. Should you read this list and pick the top thing on the chart? No. However, hard to argue this isn't interesting and useful information.
- zinxq 13y agoThere should be no surprise that interpreted, dynamic languages are utterly out-gunned as compared to compiled (JIT'd or otherwise) languages. It's inherent to the system - every little thing you do costs more. Many people choose Ruby and figure, given that premature optimization is the root of all evil, they'll optimize later if needed. That's like choosing between a farm tractor or a ferrari - and figuring if the tractor doesn't perform up to snuff, we'll add a spoiler (and given the 10x disparity between Java and Ruby in some of those graphs, if we throw out a 20mph top speed for a farm tractor, the ferrari analogy is actually rather spot on). There are many good reasons to choose dynamic/interpreted languages - but always know you're giving up performance in exchange.
- camus 13y agoPeople chose language X over language Y not for performance reasons. Cost , ease of use and deployment , librairies , available programmers ,etc ... Things are more complicated than just a benchmarks. Furthermore NodeJS and raw PHP are doing quite well in the benchmarks.
- zinxq 13y agoI did say exactly that - there's plenty of good reasons for choosing them - but understanding its a tradeoff is a good idea. In other words, although interesting (and exceedingly well done) these benchmarks should have "surprised" no one. Not even the disparity between languages.
- dragonwriter 13y ago> Many people choose Ruby and figure, given that premature optimization is the root of all evil, they'll optimize later if needed. True. > That's like choosing between a farm tractor or a ferrari - and figuring if the tractor doesn't perform up to snuff, we'll add a spoiler Its really not like that at all, because programming languages aren't like vehicles. Particularly, with Ruby, on typical method of optimization is finding which bits of code are bottlenecks, and then optimizing those bottlenecks, often by replacing them with C (or, if the Ruby runtime being used in JRuby, Java). Which I guess is like having your tractor turn into a Ferrari for the parts of work that involve going long distances on a road without towing something, but I think that kind of points out how bad even using the tractor/Ferrari analogy is.
- campnic 13y agoI think the overlooked part of this, once we step back from the natural desire to pick 'the best', is that people who care about the platforms are providing a vast set of starting examples for people looking to get started on each network. Its easy to do a side by side comparison of similar tasks across languages which is something that is very valuable and, in my experience, relatively novel. Thanks for all your amazing work!
- desireco42 13y agoI have one more question, I see a number of really fat frameworks on top, if rails is faster, then it must be because it is heavily optimized and people who made it are super smart, but for example codeigniter is above slim, kohana is above fuel, like twice the speed for Fortunes test. This isn't what I would expect at all, I would also expect Rails to be below php frameworks for speed alone. Did you use orm when it was available? Or just used raw query? This is in line with my previous query about more usage scenarios (and thank you for fortunes test) who would be like we use them usually. My point is, that the same reason why we use framework instead of raw language, we use orm instead of direct sql.
- oberhamsi 13y agothe source is available. as a first guess: if an ORM is the typical way to do it in an env then it was used.
- desireco42 13y agoyeah, my bad, should look into the source :)
- desireco42 13y agoOk to answer my question, it is raw query passing to db. I know this is easier for test to be made, but it is not as realistic. Having said that, I really appreciate work you are doing, let me repeat that for 100 times, I don't want to sound ungrateful, but this is still not the benchmark that can be used to compare frameworks.
- camus 13y agoIMHO, . it is stated in the benchmark when a framework test uses an ORM or "raw" pdo/whatever for db request. Things like Doctrine are elegant and smart ,but let's face it,they are so slow. PHP is not JAVA. Hibernate may be fast on JAVA but Doctrine is hardly (fast)... Symfony Laravele and Silex share the same http-kernel & event-dispatcher. Laravele and Silex however can use closures for controllers and filters/middleware, Maybe that's why there are faster. Classes are expensive in PHP , since PHP is not OO centric and classes are merely a add-on. Bootstraping Symfony means creating an insane number of objects. There are things that could be done about it. I'm sure PHP frameworks are so slow because of the abuse of class hierarchy.
- rdtsc 13y agoVery nice presentation! One thing I am wondering is "what about concurrency level"? Just because a server can handle 10x the number of requests when doing a single request a time for 1000 requests, doesn't necessarily mean it can also handle those 1000 request at 10x performance when they all come in at once or in a short time period. I saw some tests have "256 concurrency" does that mean they are sending 256 request concurrently? I want to see them play more with those numbers. Why not have 1024 or more. Then also play with the number of available CPUs and see which frameworks can auto-scale based on that. Some that can process sequential requests fast might fall face down when faced with slightly increased concurrency, in that respect these benchmarks are a bit misleading. On the other hand it is good to see latency. That is a important. Now latency vs level of concurrency would also be interesting.
- LukeHoersten 13y agoThe Haskell Snap Framework has a new http server in the works. It's completely rewritten on a new library called io-streams with performance in mind. Initial benchmark results look promising. http://snapframework.com/blog/2013/03/05/announcing-io-streams http://snapframework.com/blog/2013/03/05/announcing-io-strea...
- mercurial 13y agoOn the other hand, I'm genuinely surprised the current stable version is not doing any better.
- mightybyte 13y agoThis benchmark probably has more to do with the performance of the JSON serialization and database libraries than with web framework performance.
- mercurial 13y agoIf this was the only reason, Snap's performance would be closer to Yesod.
- how_gauche 13y agoIt's still several months away from being released.
- thomasfl 13y agopg: Do you consider a rewrite of Hacker News to something faster?
- Narretz 13y agoIs HN slow? I'd rather have some UI improvements.
- dallagi 13y agowhat about django
- dallagi 13y agowhat about django / flask + pypy?
- bilbo0s 13y agoWow... some of these tests are still pretty severely hobbled. Is there some reason that you use built in json serialization for some frameworks and not others? There is also a lot of heterogeneity in the implementation of the multiple queries test. For instance, even if I only look at... say... java frameworks, you seem to implement the exact same feature in very different ways between platforms. For instance, for servlets, you will store all of the results in a simple array... and then write them out when you are done. Like so: final World[] worlds = new World[count]; final Random random = ThreadLocalRandom.current(); try (Connection conn = source.getConnection()) { try (PreparedStatement statement = conn.prepareStatement(DB_QUERY, ResultSet.TYPE_FORWARD_ONLY, ResultSet.CONCUR_READ_ONLY)) { // Run the query the number of times requested. for (int i = 0; i < count; i++) { final int id = random.nextInt(DB_ROWS) + 1; statement.setInt(1, id); try (ResultSet results = statement.executeQuery()) { if (results.next()) { worlds[i] = new World(id, results.getInt("randomNumber")); } } } } } catch (SQLException sqlex) { System.err.println("SQL Exception: " + sqlex); } // Write JSON encoded message to the response. try { Common.MAPPER.writeValue(res.getOutputStream(), worlds); } catch (IOException ioe) { // do nothing } } But for other frameworks, like Vert.x, you use CopyOnWriteArray to store all of the results... and then write them out when you are done. Like so: private final HttpServerRequest req; private final int queries; private final List<Object> worlds = new CopyOnWriteArrayList<>(); . . . @Override public void handle(Message<JsonObject> reply) { final JsonObject body = reply.body; if ("ok".equals(body.getString("status"))) { this.worlds.add(body.getObject("result")); } if (this.worlds.size() == this.queries) { // All queries have completed; send the response. // final JsonArray arr = new JsonArray(worlds); try { final String result = mapper.writeValueAsString(worlds); final int contentLength = result.getBytes(StandardCharsets.UTF_8).length; this.req.response.putHeader("Content-Type", "application/json; charset=UTF-8"); this.req.response.putHeader("Content-Length", contentLength); this.req.response.write(result); this.req.response.end(); } catch (IOException e) { req.response.statusCode = 500; req.response.end(); } } } In other words, you literally create a new array each time you add a result to that CopyOnWriteArray. In fact, not only are you creating a new array, but you are creating new copies of the data in the array as well. Seems a little strange??? DEFINITELY inefficient. Is there a reason that is implemented differently? It seems to me that, at the least, they should both use arrays... but maybe there is something more you guys are testing??? The Onion C based code is written in an even MORE efficient manner for the multiple queries test. It actually stores it's results in json format from the outset! Like so: json_object *json=json_object_new_object(); json_object *array=json_object_new_array(); int i; for (i=0;i<queries;i++){ json_object *obj=json_object_new_object(); snprintf(query,sizeof(query), "SELECT * FROM World WHERE id = %d", 1 + (rand()%10000)); mysql_query(db, query); MYSQL_RES *sqlres = mysql_store_result(db); MYSQL_ROW row = mysql_fetch_row(sqlres); json_object_object_add(obj, "randomNumber", json_object_new_int( atoi(row[1]) )); json_object_array_add(array, obj); mysql_free_result(sqlres); } json_object_object_add(json,"json",array); const char *str=json_object_to_json_string(json); The equivalent java code would be something like: private final HttpServerRequest req; private final int queries; // INSTEAD OF: //private final List<Object> worlds = new CopyOnWriteArrayList<>(); // HAVE: private final JsonArray worlds = new JsonArray(); . . . @Override public void handle(Message<JsonObject> reply) { final JsonObject body = reply.body; if ("ok".equals(body.getString("status"))) { // INSTEAD OF: //this.worlds.add(body.getObject("result")); // HAVE: this.worlds.addObject(body.getObject("result")); } if (this.worlds.size() == this.queries) { // All queries have completed; send the response. // final JsonArray arr = new JsonArray(worlds); try { // INSTEAD OF: //final String result = mapper.writeValueAsString(worlds); // HAVE: final String result = worlds.encode(); final int contentLength = result.getBytes(StandardCharsets.UTF_8).length; this.req.response.putHeader("Content-Type", "application/json; charset=UTF-8"); this.req.response.putHeader("Content-Length", contentLength); this.req.response.write(result); this.req.response.end(); } catch (IOException e) { req.response.statusCode = 500; req.response.end(); } } } With a similar change for Servlets. According to the benchmark results, Onion comes out on top. It's the fastest. But how much of that's because it seems to be written correctly and other tests seem to be written without taking advantage of the same efficiencies. Is it the case here that some people have sent you test code optimized for their own frameworks? If that is so, you should add some tests that would not be so amenable to optimization. I'm not picking on Onion here by the way. In fact, the argument could be made that Onion is not actually 'optimized', so much as just written correctly, and the other frameworks have tests written incorrectly. But I just wanted to know if you guys actually intended to use these different implementations for some reason that I am unaware of? Do they make the tests more fair somehow???
- riprock 13y agoCompojure is looking really attractive ("modern" dynamic language with JVM performance backing it up). How is the ecosystem around Clojure web development?
- mark_l_watson 13y agoThe ecosystem around Clojure web development is in good shape. I have used Compojure and Hiccup (and Noir) for a large part of my web development in the last few years and it has been a happy experience. I have had a little less joy experimenting with both Clojurescript and Ember.js (with Clojure back end services): I eventually get things working, but at a huge time cost over writing non-rich clients just using Hiccup.
- deleted 13y ago[deleted]
- luisbebop 13y agoawesome. congratulations guys. nice job! you are defying the status quo about some frameworks with numbers instead of ego talks.
- X4 13y agoI just wanted to say: Thank you! This is highly respected work (bookmarked) and deserves all the upvotes HN can give.
- bhauer 13y agoThanks for the kind words, X4! With each round, we anticipate a spectrum of responses, but it's nevertheless a continuous surprise and honor to hear positive responses.
- cies 13y agoCould anyone explain: why is gemini sooo much faster then the others in these tests? I believe that the way these test are setup slightly advantage gemini, and more broadly java. Since they do not measure memory usage, or tasks that make memory usage critical, which is something JVM sucks at.
- bhauer 13y agoHi Cies, Gemini is our in-house framework and there are two points to consider: (a) We are obviously very familiar with Gemini and therefore know how to use it effectively. For example, we know that we prefer to deploy Gemini applications using the Caucho Resin application server because it has proven the quickest Java application server in our previous experience. Of course, the other Java Servlet-based frameworks also benefit from deployment on Resin in these tests. (b) In our design of Gemini, we do keep an eye on performance. But as the data shows, there are faster options. Although we included Gemini in these tests, we did so because we wanted to know how it stacked up against other frameworks that we routinely use on projects. See more information in response to an earlier question here: https://groups.google.com/d/msg/framework-benchmarks/p3PbUTg-Ibk/fOjQMxESEJ0J https://groups.google.com/d/msg/framework-benchmarks/p3PbUTg... Incidentally, the memory usage profile for the Gemini test is fairly compact, in terms of used heap space within the JVM, as are most of the Java tests in this project. With no need to do so, we're not trimming the memory allocation for the JVM to a minimum in our tests. But if we did, as you point out, the tests we've implemented so far don't require much memory.
- ausjke 13y agoHold on, Symphony2 is that bad?? I use Drupal, which will use Symphony2, this sucks!
- chetzit 13y agoSee, that's why these benchmarks do no good, people jump to quick conclusions. Also i suspect that it is not properly configured in OP case ( there is a ticket submitted https://github.com/TechEmpower/FrameworkBenchmarks/issues/230 https://github.com/TechEmpower/FrameworkBenchmarks/issues/23...). Also, Drupal will not use symfony2, it will use some of its components.
- fetbaffe 13y agoWould be nice if a PHP Fat-free framework was included in the benchmark. https://github.com/bcosca/fatfree https://github.com/bcosca/fatfree