3 ms·

Haha, I almost spit out my coffee when I saw how poorly we performed here! Reading the code, it's kind of incredible that Neo4j even competes - all the database

Haha, I almost spit out my coffee when I saw how poorly we performed here! Reading the code, it's kind of incredible that Neo4j even competes - all the databases benchmarked are running with 25 concurrent database connections: https://github.com/weinberger/nosql-tests/blob/master/arangodb/description.js#L5 https://github.com/weinberger/nosql-tests/blob/master/arango... Except Neo, which is given a single connection to work with :( https://github.com/weinberger/nosql-tests/blob/master/neo4j/description.js#L13 https://github.com/weinberger/nosql-tests/blob/master/neo4j/... When I modify it locally to just set {maxConnectionPoolSize: 25}, like how the other databases are set up, I get roughly an order better performance. neighbors2 went from 0.0226ms to 0.0068ms, for instance. Single write sync went from 3.36415ms to 0.77549ms avg/op.. Benchmarking is hard!

oil7abibi 9y ago
For future reference, I find the back ticks to decrease readability, at least on iOS.
- jakewins 9y ago
  Agree, I keep forgetting that those don't turn into monospace on HN. Changing that.
janemanos 9y ago
Jan from ArangoDB here: Did you read the appendix? "We used a TCP/IP connection pool of up to 25 connections, whenever the driver permitted this. All drivers seem to support this connection pooling, except Neo4j. We sent instead twenty-five requests via NodeJS to Neo4j." Just to avoid unfair conditions now, if connection pooling is possible with the neojs driver, then you would have to configure the benchmark scripts to not send 25 requests via Node as well. We tricked ourselves with such stuff when preparing the benchmark.
- maxdemarzi 9y ago
  Did you read the manual where it says the driver permits connection pooling? http://neo4j.com/docs/developer-manual/current/drivers/client-applications/#driver-config-connection-pool-management http://neo4j.com/docs/developer-manual/current/drivers/clien...
  - janemanos 9y ago
    Hi Max.. thanks also for your feedback. We worked it into an update to the benchmark, please find it here: https://news.ycombinator.com/item?id=16473117 https://news.ycombinator.com/item?id=16473117
- jakewins 9y ago
  Yep, I removed the CONCURRENCY constant from the neo4j description.js, so the numbers above are for Neo running without that async.eachLimit call, just like the rest of the databases. A single session object is backed by a single connection, so even though you "send" 25 requests via async.eachLimit, the result is that they are serialized over the single connection.
janemanos 9y ago
True, benchmarking is really not trivial. Thanks for pointing us to the bug in our benchmark suite and sorry for our mistake. We fixed it https://github.com/weinberger/nosql-tests/pull/24 https://github.com/weinberger/nosql-tests/pull/24 and reran the benchmark. We will publish an update soon and also fix the stats in the original post. Indeed, Neo4j with connection pooling improved stats in a few test cases but the overall ranking is not really affected. Thanks again for helping us to improve the benchmark and creating fair conditions for all products tested!
janemanos 9y ago
Hey Jake... thanks again for your feedback... we just published an update to the Benchmark following your feedback for Neo4j, please find it here: https://news.ycombinator.com/item?id=16473117 https://news.ycombinator.com/item?id=16473117

oil7abibi 9y ago

For future reference, I find the back ticks to decrease readability, at least on iOS.

jakewins 9y ago
Agree, I keep forgetting that those don't turn into monospace on HN. Changing that.

janemanos 9y ago

Jan from ArangoDB here: Did you read the appendix? "We used a TCP/IP connection pool of up to 25 connections, whenever the driver permitted this. All drivers seem to support this connection pooling, except Neo4j. We sent instead twenty-five requests via NodeJS to Neo4j." Just to avoid unfair conditions now, if connection pooling is possible with the neojs driver, then you would have to configure the benchmark scripts to not send 25 requests via Node as well. We tricked ourselves with such stuff when preparing the benchmark.

maxdemarzi 9y ago
Did you read the manual where it says the driver permits connection pooling? http://neo4j.com/docs/developer-manual/current/drivers/client-applications/#driver-config-connection-pool-management http://neo4j.com/docs/developer-manual/current/drivers/clien...
- janemanos 9y ago
  Hi Max.. thanks also for your feedback. We worked it into an update to the benchmark, please find it here: https://news.ycombinator.com/item?id=16473117 https://news.ycombinator.com/item?id=16473117
jakewins 9y ago
Yep, I removed the CONCURRENCY constant from the neo4j description.js, so the numbers above are for Neo running without that async.eachLimit call, just like the rest of the databases. A single session object is backed by a single connection, so even though you "send" 25 requests via async.eachLimit, the result is that they are serialized over the single connection.

janemanos 9y ago

True, benchmarking is really not trivial. Thanks for pointing us to the bug in our benchmark suite and sorry for our mistake. We fixed it https://github.com/weinberger/nosql-tests/pull/24 https://github.com/weinberger/nosql-tests/pull/24 and reran the benchmark. We will publish an update soon and also fix the stats in the original post. Indeed, Neo4j with connection pooling improved stats in a few test cases but the overall ranking is not really affected. Thanks again for helping us to improve the benchmark and creating fair conditions for all products tested!

janemanos 9y ago

Hey Jake... thanks again for your feedback... we just published an update to the Benchmark following your feedback for Neo4j, please find it here: https://news.ycombinator.com/item?id=16473117 https://news.ycombinator.com/item?id=16473117