26 ms·
Performance Benchmark 2018 – MongoDB, PostgreSQL, OrientDB, Neo4j and ArangoDB
- ahs1200 9y agoWow, that looks like a lot of work. Nice effort.
- steinerj 9y agoIt's curious that single purpose Graph DBs like Neo4j perform poorly at some typical graph queries. Or am I getting something wrong?
- mchacki 9y agoHey Michael from ArangoDB here. I was also a bit surprised. Especially for the graph case. We see that more memory is consumed but the throughput did not really increase in comparision to the benchmark we conducted 3 years ago. Nevertheless we see improvements on the shortest path queries, these challenge the database more. As we are no dedicated Neo4J experts we are intrested in every configuration that will speed things up, please share them with us.
- e12e 9y agoSee: https://news.ycombinator.com/item?id=16378157 https://news.ycombinator.com/item?id=16378157
- allandubey 9y agoMuch needed! Thank You
- anilshanbhag 9y agoI don't think these results are comparable. Query runtimes are highly dependent on how much memory they can use. Postgres for example isn't configured making it use less memory than other systems. Hence its performance is likely inferior and could be improved significantly by increasing the buffer size.
- mpv89 9y agoHi, Mark from ArangoDB here. We used the default configuration for every Database in the benchmark under the assumption they are picked reasonable. If you have any specific suggestions how to configure Postgres for the specific environment, please let us know.
- aschampion 9y agoPostgres is configured extremely conservatively by default. While it isn't reasonable to spend time tweaking each database, at least running pgtune would be reasonable for Postgres.
- jsteemann 9y agoI think it's definitely reasonable to tune important production databases. But for many people databases have become commodity applications that they install and then forget. So even in reality not all installments will be properly tuned.
- robbyt 9y agoWhich is why many apps are slow... People search for a new toy, rather than learn what they already have.
- etxm 9y agoWho does this?! I know it happens because I inherit them often. But seriously knock it off. Errrrr. Better yet don’t, job stability. Nothing to see here. Move along.
- rlpb 9y ago> But for many people databases have become commodity applications that they install and then forget. It seems unlikely to me that this group of people would reach benchmark posts.
- rsynnott 9y agoPeople using them that way probably don't care about performance very much, though.
- olavgg 9y agoI've stopped reading database benchmarks, because they are extremely vague. Instead I spend my time optimizing my current solution/stack. For example Postgresql has hundreds of knobs that you can adjust for almost every scenario you can imagine. Sometimes you have a special query and increase the work_mem just for that session. Other cases you adjust the cost settings for another query/session. You can analyze your indexes and index types. And sometimes you need to rewrite parts of a big query. Learning all this takes time, you are much better off learning more about your chosen technology stack than switching to another technology stack. Though in a few rare races, you need a different technology to solve your business problem. In most cases they complement your existing solution, like Elasticsearch/Solr for full-text search or Clickhouse for OLAP workloads.
- maxxxxx 9y agoAgreed. Switching to another system is expensive and the benefit is pretty questionable.
- emsy 9y agoUnless you hit a very specific use-case/bottleneck, which I only ever witnessed once.
- TremendousJudge 9y agoexpand, please?
- maxxxxx 9y agoI imagine something very specific like having a lot of inserts into a table and that being your main use case. Depending on your data some databases may be better than others and that should be easy to measure. In most real-world cases the requirements however are not very clear and often conflicting so it's much harder to get data that shows the performance of one system over the other.
- oneweekwonder 9y agoInteresting, thanks for doing this, and sharing it so openly. Do you maybe have the data for your graphs in csv or json? Want to plot my own graphs with it. Personally I would call it Javascript Performance Benchmark for MongoDB, PostgreSQL, OrientDB, Neo4j and ArangoDB. Which is mentioned at the end of the article under the software section. To be truly a Database Performance Benchmark you would need to interface with the c/native library. Can anybody recommend other database benchmarks?
- graetzer 9y agoI think the orkut dataset is already in a tab seperated format, to turn it into CSV you only need to add the column labels. http://snap.stanford.edu/data/com-Orkut.html http://snap.stanford.edu/data/com-Orkut.html For an example you can have a look at the import scripts: https://github.com/weinberger/nosql-tests/blob/master/arangodb/import.sh https://github.com/weinberger/nosql-tests/blob/master/arango...
- oneweekwonder 9y agoSorry I should have been more clear. I want the raw result data he used to draw the graphs from. I wanted to use high-charts to make it interactive, eg hide dbs to compare more easily. I don't want to run my own tests, but I did have a look at the github repo and dataset.
- janemanos 9y agoHere you go: https://docs.google.com/spreadsheets/d/1_unaj2x_NCVNUHcxwMUVkst9shlqLwrCZHRCMv_LDus/edit?usp=sharing https://docs.google.com/spreadsheets/d/1_unaj2x_NCVNUHcxwMUV... Let me know if you have any question or need something else
- albertgoeswoof 9y agoTLDR, just use postgres
- noah-kun 9y agoThat’s not the conclusion of this at all...
- pabl0rg 9y agoIn most benchmarks Postgres with tabular data got the best results. If you need to store json or do graph queroes and don't want to create indices youself, then consider araango
- deleted 9y ago[deleted]
- michaelbuckbee 9y agoAnd even then, should you be storing JSON? Is putting GraphQL into your datastore the right thing?
- gmueckl 9y agoThat depends. If you know that your data contains a highly irregular, non-exhaustive list of extra attributes that have no relations to other parts of the data model, a JSON column might be the right choice. It's rarely the best choice, but I can totally see applications where it is.
- rpedela 9y agoI use JSON for data that needs structure, but doesn't need to be relational. That data is then returned with the rest of the row in a SELECT. I will use JSONB for the same reason, and I also need to query that data once in awhile. JSONB is also very useful for HTTP JSON APIs because you can have the database construct the JSON response and the API server just returns it. I found it to be faster than constructing the JSON response in the API server most of the time. Of course, it depends on the use case whether you can do that.
- amorroxic 9y agoArangoDB is an incredible database with some very shy marketing imo - edge indexing over json documents is too good of a sweet spot. Not sure if still actual, however among headaches I remember race conditions in the node driver(rocksdb engine) and maybe less straightforward clustering (compared to Couchbase/Cassandra/etc at least).
- mchacki 9y agoThe race conditions have been eliminated and we greatly simplified the cluster setup. With this "new" tool shipped with arangodb https://github.com/arangodb-helper/arangodb https://github.com/arangodb-helper/arangodb it is just a single line on each machine to get the cluster up and running.
- amorroxic 9y agoProps to you guys, outstanding product really.
- drej 9y agoHas any vendor ever produced a benchmark, where their solution wasn't the clear winner?
- nathan_long 9y agoSurely! They just didn't publish it. :)
- virmundi 9y agoArango isn’t the clear winner in all of these. Unconfigured PG did pretty good. PG with JSON and no indexes did well too. It also held its own using less memory. Finally, PG has transactions where as Arango doesn’t really if you have an index constraint on a document attribute. This shows Arango is better than Neo4j and Mongo for graph resulted service and document store. It skipped PG’s recursive capability that probably allows for basic graph traversal to a known depth.
- cooervo 9y agoyou didn't even open the link. ArangoDB is not the clear winner in these results
- kbenson 9y agoIn what case does it make sense to benchmark these databases but leave MySQL/MariaDB/Percona out?
- exabrial 9y agoI was about to post the same thing, seems like a very obvious omission.
- janemanos 9y agoThink it's also a question of effort involved for testing even more stores
- don71 9y agoDisclaimer: I'm part of the ArangoDB team. As written in the post the whole benchmark is open source, and you are welcome to add other DBs. That is appreciated. It is just important that there is an official node.js driver and a GA version will be used.
- e12e 9y agoWould be nice to domain/product experts to some tweaking and host this at a third party a la "The benchmark game"[1]. A nice little graph-oriented benchmark would be great, especially avoiding things like: https://news.ycombinator.com/item?id=16378157 https://news.ycombinator.com/item?id=16378157 [ed: and: "For comparison, we used three leading single-model database systems:... and PostgreSQL for relational database." Jsonb with indexes doesn't rate as "document db", how about postgis for domain specific graph db?] Some agreement on hw (virtual or otherwise), dataset and queries might make this actually rather useful. [1] https://benchmarksgame.alioth.debian.org/ https://benchmarksgame.alioth.debian.org/
- maxdemarzi 9y agoDon’t ever trust any vendor benchmarks: https://maxdemarzi.com/2015/10/16/benchmarks-and-superchargers/ https://maxdemarzi.com/2015/10/16/benchmarks-and-supercharge...
- janemanos 9y agoIn your 2015 article, you criticized that the ArangoDB team restarted the instances after each test run. In this 2018 edition, they don't do this anymore.
- maxdemarzi 9y agoYou want to trust a vendor that used to restart the db between queries for their benchmarks?
- virmundi 9y agoYes, because they learned. Arango has honest people that are open to criticism.
- tinymollusk 9y agoBut the larger issue -- that they have direct financial incentive to unconsciously bias the results -- still exists. This isn't anything against Arango's team, I'm sure they're lovely people, but even physics experimental results have been shown to be biased when the experimenter knew the testing hypothesis. "I think I've been in the top 5% of my age cohort all my life in understanding the power of incentives, and all my life I've underestimated it. Never a year passes that I don't get some surprise that pushes my limit a little farther." -- Charlie Munger
- JackFr 9y agoOr Feynman: "The first principle is that you must not fool yourself and you are the easiest person to fool."
- etxm 9y agoArangoDB always makes for exciting benchmark posts. I could see myself there in a bowler hat with a fistful of racing chits screaming “go, Postgres, go.” I’d love to see a competition were the developers of each database got to use the same hardware and data then tune the hell out of their configs, queries, and indices. Red Bull could sponsor it. I’d buy a T-shirt.
- jsteemann 9y agoNice idea, I would take part in something like that too. By the way, there seems to be a raffle of free T-shirts (quite biased, as there's only the ArangoDB logo on them) in exchange for taking part in a survey: https://arangodb.com/community-survey/ https://arangodb.com/community-survey/
- y4mi 9y agoThat sound ridiculously good. Like esports, for DBAs
- etxm 9y agoWe could call it DB sports. On second thought that might be a bad name
- kbenson 9y agoThat doesn't sound that hard to start. Something like RealWorld[1] and the Web Framework Benchmarks[2] combined but for DB workloads. Have one dataset that includes data amenable to OLAP and OLTP, but have separate tests each consisting of OLAP queries, OLTP queries, and combined queries. Choose a low-end, mid-range and high-end set of AWS or GCE instances/configs to normalize against. Let people submit pull requests with new technologies or configs. You'd want to get some funding to run the tests (or maybe solicit Google or Amazon to see if you could get the instance time donated once a month or something. If you started small, with maybe a portion of these features, and then scaled up over time, you might actually get to the point where you had tests that emulated a power failure, or master/slave and dual master scenarios and how they handle certain common network errors (split-brain). That would be an amazing resource. Edit: It occurs to me I probably should have read more of the article, since this is sort of what they are doing already... 1: https://github.com/gothinkster/realworld https://github.com/gothinkster/realworld 2: https://www.techempower.com/benchmarks/ https://www.techempower.com/benchmarks/
- zilchers 9y agoBig missing piece of this was clustered / HA configurations. To run a production level application, you’re going to be running with some sort of clustering or replication, which is going to add a pretty significant latency depending on implementation. But, interesting article, I’m always impressed some of these smaller database companies are still kicking.
- HugoDaniel 9y agoDidn't know about this project. Their homepage made me want to try it out. Is AQL an invention they made ? reading their [examples] made me feel that this is a very welcome update on SQL. Congrats. [examples] https://www.arangodb.com/why-arangodb/sql-aql-comparison/ https://www.arangodb.com/why-arangodb/sql-aql-comparison/
- maxpert 9y agoHahahaha this makes me use PostgreSQL even more :D
- cldellow 9y agoThe table of times notes: > The shortest path query was not tested for MongoDB or PostgreSQL since those queries would have had to be implemented completely on the client side for those database systems. I can't speak for Mongo, but I imagine the shortest path query could be implemented as a recursive CTE or a sproc in Postgres, which would execute server side. It might be a bear to write, and it might be slow, but it should be possible.
- janemanos 9y agoIf you know how to write the shortest path query for Postgres, team ArangoDB would be happy to include it into the next update
- lebotski 9y agohttps://stackoverflow.com/questions/6873772/sql-postgres-shortest-path-in-graph-recursion https://stackoverflow.com/questions/6873772/sql-postgres-sho...
- tankenmate 9y agoNot exactly what you are asking for but this sits on top of PostgresQL and uses non-standard indexing to improve graph queries. https://github.com/bitnine-oss/agens-graph https://github.com/bitnine-oss/agens-graph
- devbug 9y agoI recently used RCTEs in a production environment to operate on reasonably large graphs. We eventually moved to an in memory solution built on top of ETS for various reasons, but performance was not a deciding factor (at least as much as you would think); Postgres handled our queries well beyond expectation, even with heavy factoring into functions. It's definitely not slow. The largest burden was on development and maintenance. Iterating algorithms and queries powered by RCTEs (and functions) on a changing schema is not scaleable (in effort), at least with the traditional migration scripts approach to upgrades and rollbacks. If you do want go this route, you need to change how you ship those functions and queries. We theorized that shipping them as part of the application and performing necessary migrations behind-the-scenes would have drastically reduced the maintenance burden. We also considered composing queries completely dynamically and fully leveraging our language (Elixir) to provide a toolbag of functions that aren't functions as far as Postgres was concerned. In either case, the hassle of versioning functions and queries would no longer get in the way iteration.
- speby 9y agoA benchmark made by a database vendor that shows that the database vendor performs well, if not the best, overall. Whether it is actually true or not, the result being published is entirely unsurprising.
- don71 9y agoDisclaimer: I'm part of the ArangoDB team. As written in the post the whole benchmark is open source. the idea is that you can run it on your own. Take the chance and get your own impression.
- ChicagoDave 9y agoWhy is Neo4j in this comparison at all? It's a graph database and its use-case is very specifically different than relational or document databases. Please compare apples to apples.
- janemanos 9y agoWhich DB operations included into the test is not something you also might have to perform with neo4j?
- kbd 9y agoArangoDB is also a graph database. Many of the benchmarks (shortest path, neighbors, etc.) are specifically geared towards graph databases. Neo4j is totally appropriate in this comparison.
- lobo_tuerto 9y agoFrom the article, PostgreSQL still looks like the best option in general.
- riku_iki 9y agoUnless you need easy clustering..
- lobo_tuerto 9y agoIn that case, you can go for CockroachDB. ;) You can use the same drivers for PostgreSQL to talk to CockroachDB. As specified here: https://www.cockroachlabs.com/docs/stable/frequently-asked-questions.html#what-languages-can-i-use-to-work-with-cockroachdb https://www.cockroachlabs.com/docs/stable/frequently-asked-q... "CockroachDB supports the PostgreSQL wire protocol, so you can use any available PostgreSQL client drivers."
- maxdemarzi 9y agoI hate these types of posts. They do nothing to help the graph database space. How good are they at bidirectional traversals, what about deep traversals in complex networks with more than 50 relationship types? What about x? Every vendor is going to be better at one thing than another, there is always a trade off, but it is irrelevant. Last week we took a very deep and very wide sales hierarchy query from 10 seconds in Oracle to 4ms in Neo4j. If Arango could it in 3ms, or Orient in 5ms doesn't really matter. The point is graph databases are much better at some queries than relational. They should be blogging about use cases and customer successes, not benchmarks.
- ifcologne 9y ago>They should be blogging about use cases and customer successes, not benchmarks. They do, just recently the cases of AskBlue (Location aware recommendations) and Thomson Reuters (Fast & Secure Single-View of Everything) https://www.arangodb.com/why-arangodb/case-studies/ https://www.arangodb.com/why-arangodb/case-studies/ I am very interested in Graph-Cases and appreciate the support of Neo4J for projects like the Panama- or Paradise Papers that show how connected graphs help to understand former hidden relationships. Graphs are very powerful and a great addition to relational or document approaches. (full disclosure: I worked for ArangoDB 2 years ago.) And, I've used Neo successfully for a fraud detection PoC recently...
- CodeSheikh 9y agoIs it fair to compare JSONB format for PostgreSQL in this benchmark as it is not the main highlight of PostgreSQL but more like an added feature on top of SQL?
- Simran-B 9y agoUsers asked for a JSONB comparison after the previous benchmark, so we included it. The tabular format is also there. Thus, I don't see how it would be unfair.
- rambossa 9y agoAny particular reason why https://github.com/dgraph-io/dgraph https://github.com/dgraph-io/dgraph missed the benchmark? -- It seems to be highly capable (horizontal-scale/acid) & "performant". Reference vs Neo4j: https://blog.dgraph.io/post/benchmark-neo4j/ https://blog.dgraph.io/post/benchmark-neo4j/
- ChicagoDave 9y agoIn-house performance stats are the least trustworthy thing on the planet. Are there any independent studies?
- Simran-B 9y agoNo, there aren't even standard benchmarks that cover all types of databases to compare performance etc. The ArangoDB benchmark is open source however. Feel free to test it yourself and tune as many parameters as you want. You can also add scripts for other database systems.
- vijaybritto 9y agoI don't know if its only me but the graphs are very hard to understand. Im viewing in a large desktop monitor and I cant imagine it would be in a mobile phone!
- robbiemitchell 9y agoCompare to a similar post from a couple years ago: https://news.ycombinator.com/item?id=9699102 https://news.ycombinator.com/item?id=9699102
- jakewins 9y agoHaha, I almost spit out my coffee when I saw how poorly we performed here! Reading the code, it's kind of incredible that Neo4j even competes - all the databases benchmarked are running with 25 concurrent database connections: https://github.com/weinberger/nosql-tests/blob/master/arangodb/description.js#L5 https://github.com/weinberger/nosql-tests/blob/master/arango... Except Neo, which is given a single connection to work with :( https://github.com/weinberger/nosql-tests/blob/master/neo4j/description.js#L13 https://github.com/weinberger/nosql-tests/blob/master/neo4j/... When I modify it locally to just set {maxConnectionPoolSize: 25}, like how the other databases are set up, I get roughly an order better performance. neighbors2 went from 0.0226ms to 0.0068ms, for instance. Single write sync went from 3.36415ms to 0.77549ms avg/op.. Benchmarking is hard!
- oil7abibi 9y agoFor future reference, I find the back ticks to decrease readability, at least on iOS.
- jakewins 9y agoAgree, I keep forgetting that those don't turn into monospace on HN. Changing that.
- janemanos 9y agoJan from ArangoDB here: Did you read the appendix? "We used a TCP/IP connection pool of up to 25 connections, whenever the driver permitted this. All drivers seem to support this connection pooling, except Neo4j. We sent instead twenty-five requests via NodeJS to Neo4j." Just to avoid unfair conditions now, if connection pooling is possible with the neojs driver, then you would have to configure the benchmark scripts to not send 25 requests via Node as well. We tricked ourselves with such stuff when preparing the benchmark.
- maxdemarzi 9y agoDid you read the manual where it says the driver permits connection pooling? http://neo4j.com/docs/developer-manual/current/drivers/client-applications/#driver-config-connection-pool-management http://neo4j.com/docs/developer-manual/current/drivers/clien...
- ahachete 9y agoMy feedback: - 1.6M documents. This is a tiny dataset. I bet it's much smaller than RAM size (122GB), which means all data is in memory. Interesting benchmark, but not what I'd typically expect for a database (data size >> RAM). - All databases are allowed either to use all memory or capped at 10GB. PostgreSQL is capped at 128MB, the default shared_buffers value on PostgreSQL 10! This means PostgreSQL is running off of disk, whereas the other databases have all data in memory. - Just to insist on the previous bullet: I see the point (even though I don't consider it interesting, compared to a minimum tuning of every db) of using default config, but this puts PG on a very unfair position (it's limited to 1/1000th of RAM!). Just set shared_buffers to 32GB (or 10GB to cap all db to 10GB!) and re-run the tests. I bet PG results will improve significantly. - Other basic tuning parameters (like work_mem or min_wal_size or random_page_cost) may also affect significantly the performance. This could be left as an exercise to do another benchmark with properly tuned databases. If interested, here's a recent presentation I did with many other PostgreSQL tuning recommendations: https://speakerdeck.com/ongres/postgresql-configuration-for-humans https://speakerdeck.com/ongres/postgresql-configuration-for-... Other than this, congratulations for the work on ArangoDB. Building a database is a really brave, hard work.
- janemanos 9y agoHi @ahachete and thanks for your feedback to our benchmark 2018. We worked it into an update, please find it here: https://news.ycombinator.com/item?id=16473117 https://news.ycombinator.com/item?id=16473117