3 ms·

Hello bilbo0s. Thanks for taking the time to dig in and provide some feedback. As much as possible, we want each test to be representative of idiomatic produc

by bhauer 14y ago

Hello bilbo0s. Thanks for taking the time to dig in and provide some feedback. As much as possible, we want each test to be representative of idiomatic production-grade usage of the framework or platform. Furthermore, we have solicited contributions from fans of frameworks and the frameworks' authors. A side objective is that the code double as an example of how best to use the framework or platform. All of this means we fully expect that the implementation approaches will vary significantly. The multiple query test has a client-provided count of queries, so in most Java cases, we create a fixed-size array to hold the results fetched from the database. I wrote the Servlet and Gemini tests, so I can confirm that behavior in those tests. We are not Vert.x experts and we have not yet received a community contribution for the Vert.x test. However, it is our understanding that idiomatic Vert.x usage encourages the use of asynchronous queries. The question then is: how do we collect the results into a single List in a threadsafe manner? Is your JsonArray alternative threadsafe? Admittedly, using a CopyOnWriteArrayList gave us pause, but we are not (yet) aware of a better alternative. The Onion test was contributed by a reader and admittedly its compliance with the specification we've created is perhaps a bit dubious. We want a JSON serializer to process an in-memory object into JSON. I'm not certain if the Onion implementation matches that expectation, but the test implementation nevertheless seemed sufficiently idiomatic for his platform. We're certainly open to more opinions on that matter.

bilbo0s 14y ago
"..The multiple query test has a client-provided count of queries, so in most Java cases, we create a fixed-size array to hold the results fetched from the database. I wrote the Servlet and Gemini tests, so I can confirm that behavior in those tests..." I agree, that approach would be best. I just was unsure why you didn't do it in Vert.x. "...it is our understanding that idiomatic Vert.x usage encourages the use of asynchronous queries..." Someone can correct me if I am wrong, but my understanding of Vert.x is that any query you send to the event bus is already asynchronous. There is no need for a developer to worry about threads at all when writing a vert.x handler. That handler will only ever be called from a single thread. So using a simple array is fine. Using the JsonArray is even better, because then it matches the Onion test idiomatically speaking. Which, I agree, is what you should be going for. "...The Onion test was contributed by a reader and admittedly its compliance with the specification we've created is perhaps a bit dubious. We want a JSON serializer to process an in-memory object into JSON..." Please don't misunderstand, the Onion test does what you want it to do. As well, it does it in the correct idiomatic fashion. That's exactly how I would write the Onion test. I was just wondering why the other tests went out of their way to decode Json and the reencode Json for each result. Onion only ever encodes to Json once, other tests are encoding and decoding multiple times. I only pointed out Vert.x because it was the most egregious. I mean in that case the answer from the persistor is already in Json. It is put in a non Json data structure... and then that data structure is encoded to Json??? Just seemed weird. ---- EDIT: Just verified that there is no need for thread safe code in a Vert.x handler. (Gotta say... that is pretty slick) On a connected note... man ... these tests are a VERY good way to learn more about these different frameworks! ----
- bhauer 14y ago
  You're right, this is fascinating stuff. Like I said, we had not yet received a pull request for the Vert.x test, but presumably we will get one before Round 5? :) We had not previously understood that there was no need for thread-safe behavior within a Vert.x handler. Removing that (apparently fictional) requirement allows us to use just a simple array. Out of curiosity, can you point me to where you found confirmation that handlers do not require thread safety? Thanks again for your feedback! Edit: spot checking Vert.x with a simple array does not appear to affect performance to a measurable degree.
  - bilbo0s 14y ago
    Sorry (for the tardy response)... minor emergency came up. Anyway, the relevant part of the manual is: http://vertx.io/manual.html#concurrency http://vertx.io/manual.html#concurrency Also discovered you are running the equivalent of a single threaded environment anyway. (8 workers on an 8 core machine) Any reason for that??? Maybe more confusion about Vert.x concurrency??? If so, here is the relevant documentation on worker verticles: http://vertx.io/manual.html#worker-verticles http://vertx.io/manual.html#worker-verticles Long story short, you dedicated the entire machine (8 cores) to the equivalent of database connection management, (the persistor). Very little of the machine, (whatever is context switched in effectively), is dedicated to request handling. Try something a bit more fair... like 4 workers and 4 web request handlers. Also, I was going through the Node.js tests and I had a question... do you guys do any clustering for the Node tests at all? Or are these results from the tests run on a single Node? Sorry for all the questions, just want to make all of the tests do the same thing across all of the frameworks so that when you guys run it again, we can use that data here in a more meaningful fashion. For us, it is useful.
    - bhauer 14y ago
      We used 8 Vert.x verticles on i7 because there are 8 HT cores and our understanding is that the "best practice" for Vert.x is to create a number of verticles equal to the server's core count [1]. Obviously we would be happy to hear from Vert.x experts about an more idealized configuration. Admittedly, we have not spent a great deal of time attempting to tune Vert.x and, like I said earlier, we have not received a pull request [2]. In all tests, the database test is allocating a greater amount of effort to database connection management (in totality: the handling of connections, statements, queries, and result sets) versus request handling. This is not unique to Vert.x. The reason some frameworks' database tests achieve nearly 50% as many requests per second on i7 versus the pure JSON test is simply that at the ~210k rps range for the JSON tests, we are running into a Gigabit Ethernet wall (which I have commented about elsewhere). If we had 10 GBE, the JSON test results on i7 would be even higher. (Also see comments elsewhere about our intent to normalize, to a degree, the response header requirements since the variation observed is attributable to response-headers [3].) Yes, the node.js tests are running with the cluster module [4]. Thanks for the comments. We have received great feedback in the previous rounds and this round received even more attention so there have been some more good questions. Unfortunately, there has also been some rehashing, which indicates we're not doing a great job of explaining to people how each environment is configured (linking to the repository only goes so far). That said, we also continue to receive some fantastic pull requests. Thanks to everyone who has helped out! [1] http://vertxproject.wordpress.com/2012/05/09/vert-x-vs-node-js-simple-http-benchmarks/#comment-69 http://vertxproject.wordpress.com/2012/05/09/vert-x-vs-node-... [2] https://github.com/TechEmpower/FrameworkBenchmarks/tree/master/vertx https://github.com/TechEmpower/FrameworkBenchmarks/tree/mast... [3] https://github.com/TechEmpower/FrameworkBenchmarks/issues/113 https://github.com/TechEmpower/FrameworkBenchmarks/issues/11... [4] https://news.ycombinator.com/item?id=5645598 https://news.ycombinator.com/item?id=5645598
pgsandstrom 14y ago
Concerning the CopyOnArrayList: Unless the number of reads heavily outnumbers the numbers of writes, it is better to use something like Collections.synchronizedList(new ArrayList<String>()) instead of CopyOnArrayList. The synchronized list does lock while reading or writing, but writing still becomes much faster.
- bhauer 14y ago
  Thanks! That's true. Incidentally, we spot tested the Vert.x test with a plain array in lieu of the CopyOnArrayList and there was no perceptible performance change. We're very happy to make it more correct and eek out whatever small gains can be had--plus make the code cleaner. But if there are readers looking to improve the Vert.x numbers, I think we're going to need contributions from someone with a deeper understanding of Vert.x tuning (or some more time to invest in becoming that!)

by bhauer 14y ago

bilbo0s 14y ago

"..The multiple query test has a client-provided count of queries, so in most Java cases, we create a fixed-size array to hold the results fetched from the database. I wrote the Servlet and Gemini tests, so I can confirm that behavior in those tests..." I agree, that approach would be best. I just was unsure why you didn't do it in Vert.x. "...it is our understanding that idiomatic Vert.x usage encourages the use of asynchronous queries..." Someone can correct me if I am wrong, but my understanding of Vert.x is that any query you send to the event bus is already asynchronous. There is no need for a developer to worry about threads at all when writing a vert.x handler. That handler will only ever be called from a single thread. So using a simple array is fine. Using the JsonArray is even better, because then it matches the Onion test idiomatically speaking. Which, I agree, is what you should be going for. "...The Onion test was contributed by a reader and admittedly its compliance with the specification we've created is perhaps a bit dubious. We want a JSON serializer to process an in-memory object into JSON..." Please don't misunderstand, the Onion test does what you want it to do. As well, it does it in the correct idiomatic fashion. That's exactly how I would write the Onion test. I was just wondering why the other tests went out of their way to decode Json and the reencode Json for each result. Onion only ever encodes to Json once, other tests are encoding and decoding multiple times. I only pointed out Vert.x because it was the most egregious. I mean in that case the answer from the persistor is already in Json. It is put in a non Json data structure... and then that data structure is encoded to Json??? Just seemed weird. ---- EDIT: Just verified that there is no need for thread safe code in a Vert.x handler. (Gotta say... that is pretty slick) On a connected note... man ... these tests are a VERY good way to learn more about these different frameworks! ----

bhauer 14y ago
You're right, this is fascinating stuff. Like I said, we had not yet received a pull request for the Vert.x test, but presumably we will get one before Round 5? :) We had not previously understood that there was no need for thread-safe behavior within a Vert.x handler. Removing that (apparently fictional) requirement allows us to use just a simple array. Out of curiosity, can you point me to where you found confirmation that handlers do not require thread safety? Thanks again for your feedback! Edit: spot checking Vert.x with a simple array does not appear to affect performance to a measurable degree.
- bilbo0s 14y ago
  Sorry (for the tardy response)... minor emergency came up. Anyway, the relevant part of the manual is: http://vertx.io/manual.html#concurrency http://vertx.io/manual.html#concurrency Also discovered you are running the equivalent of a single threaded environment anyway. (8 workers on an 8 core machine) Any reason for that??? Maybe more confusion about Vert.x concurrency??? If so, here is the relevant documentation on worker verticles: http://vertx.io/manual.html#worker-verticles http://vertx.io/manual.html#worker-verticles Long story short, you dedicated the entire machine (8 cores) to the equivalent of database connection management, (the persistor). Very little of the machine, (whatever is context switched in effectively), is dedicated to request handling. Try something a bit more fair... like 4 workers and 4 web request handlers. Also, I was going through the Node.js tests and I had a question... do you guys do any clustering for the Node tests at all? Or are these results from the tests run on a single Node? Sorry for all the questions, just want to make all of the tests do the same thing across all of the frameworks so that when you guys run it again, we can use that data here in a more meaningful fashion. For us, it is useful.
  - bhauer 14y ago
    We used 8 Vert.x verticles on i7 because there are 8 HT cores and our understanding is that the "best practice" for Vert.x is to create a number of verticles equal to the server's core count [1]. Obviously we would be happy to hear from Vert.x experts about an more idealized configuration. Admittedly, we have not spent a great deal of time attempting to tune Vert.x and, like I said earlier, we have not received a pull request [2]. In all tests, the database test is allocating a greater amount of effort to database connection management (in totality: the handling of connections, statements, queries, and result sets) versus request handling. This is not unique to Vert.x. The reason some frameworks' database tests achieve nearly 50% as many requests per second on i7 versus the pure JSON test is simply that at the ~210k rps range for the JSON tests, we are running into a Gigabit Ethernet wall (which I have commented about elsewhere). If we had 10 GBE, the JSON test results on i7 would be even higher. (Also see comments elsewhere about our intent to normalize, to a degree, the response header requirements since the variation observed is attributable to response-headers [3].) Yes, the node.js tests are running with the cluster module [4]. Thanks for the comments. We have received great feedback in the previous rounds and this round received even more attention so there have been some more good questions. Unfortunately, there has also been some rehashing, which indicates we're not doing a great job of explaining to people how each environment is configured (linking to the repository only goes so far). That said, we also continue to receive some fantastic pull requests. Thanks to everyone who has helped out! [1] http://vertxproject.wordpress.com/2012/05/09/vert-x-vs-node-js-simple-http-benchmarks/#comment-69 http://vertxproject.wordpress.com/2012/05/09/vert-x-vs-node-... [2] https://github.com/TechEmpower/FrameworkBenchmarks/tree/master/vertx https://github.com/TechEmpower/FrameworkBenchmarks/tree/mast... [3] https://github.com/TechEmpower/FrameworkBenchmarks/issues/113 https://github.com/TechEmpower/FrameworkBenchmarks/issues/11... [4] https://news.ycombinator.com/item?id=5645598 https://news.ycombinator.com/item?id=5645598

pgsandstrom 14y ago

Concerning the CopyOnArrayList: Unless the number of reads heavily outnumbers the numbers of writes, it is better to use something like Collections.synchronizedList(new ArrayList<String>()) instead of CopyOnArrayList. The synchronized list does lock while reading or writing, but writing still becomes much faster.

bhauer 14y ago
Thanks! That's true. Incidentally, we spot tested the Vert.x test with a plain array in lieu of the CopyOnArrayList and there was no perceptible performance change. We're very happy to make it more correct and eek out whatever small gains can be had--plus make the code cleaner. But if there are readers looking to improve the Vert.x numbers, I think we're going to need contributions from someone with a deeper understanding of Vert.x tuning (or some more time to invest in becoming that!)