4 ms·
NoSQL Benchmark Compares PostgreSQL, MongoDB, Neo4j, OrientDB and ArangoDB
- gegtik 11y agoLooking around, it seems that different graph engines pull ahead depending on the use case. http://www.slideshare.net/sympapadopoulos/adbis2014-presentation http://www.slideshare.net/sympapadopoulos/adbis2014-presenta...
- cwyers 11y agoI really, really distrust these kinds of evaluations when they come from someone whose product is included in the comparison. Even if everything is above-board, they're not going to publish if it shows their product just completely sucks at it. That kind of publication bias makes these kinds of results a lot less trustworthy than independent benchmarks even if you assume the best of intentions from the people putting them out there.
- creshal 11y agoThe particular benchmark seems bullshit, too. For postgres they seem to intentionally use the less performant json column type instead of jsonb. Not that I can verify it, because the code in the linked public "No magic, no tricks – check the code and make your own tests!" repository doesn't match the published results and doesn't even work at all with postgres… EDIT: Okay, they pushed a new version containing the Postgres data now. They ARE using the cripplingly slow json columns, not jsonb columns recommended by the documentation.
- andy_ppp 11y agoJust to be a pedant, the JSONB format as I understand is marginally slower at inserts and orders of magnitude faster at everything else...
- robconery 11y agoIt's fractionally slower, true, because of the serialization hit (string to binary). The real juice comes from the GIN index - and if you apply it to specific columns instead of a complete document, you have a rocket ship on read.
- creshal 11y ago> It's fractionally slower, true, because of the serialization hit (string to binary). And with JSON columns you have to serialize on accesses, which is a lot slower in the read-mostly tests.
- SilasX 11y agoStupid question: isn't that the typical result of indexing? Slower writes for (much) faster lookups?
- jpgvm 11y agoAnd despite that Postgres destroys all the other solutions at everything other than the non-sync (lol) single write case and graph traversal. If anything it just proves even after almost a decade of these "NoSQL" solutions being around they still can't compete even on basic queries with Postgres which is a fairly conservative SQL solution.
- BlackFingolfin 11y agoI think this is a classical case of "use the right tool for the right purpose". If you want to do "classic database" stuff, use a classic SQL database. But if e.g. graph traversal is a cruicial operation for your application, then looking at NoSQL solution seems to be quite interesting. In other words: I wouldn't call a screw driver a bad tool, just because it's not as good at driving nails into wood as a hammer.
- KingMob 11y agoI think the issue is how many NoSQL solutions over recent years have billed themselves as the default start-with-us data store, and SQL as the more complicated niche product...when it's really the other way around.
- deleted 11y ago[deleted]
- qaqy 11y agoWTF with VCs investing into all this crap? There are very few scenarios were you would be better off with NoSQL solution and there are established players serving those niches already.
- ifcologne 11y agoIngo from ArangoDB here. I agree that vendor tests are always biased, of course you want to show that your product is competitive. But as there is no independent institution that compared our product and as we want to know where we stand with ArangoDB, Claudius did his own tests. And as the work is already done, why not share it. We tried our best to do it as open as possible. PostgreSQL performed very well and we have a problem with memory consumption - have a look at the charts, we will try to improve there. - Every database configuration is public - All test scripts are available on Github - We publish updates if we get pull-requests or comments with suggestions for improvements We did that before and after the last test, some database vendors sent us improved snapshots of their databases which found their way into the latest products (OrientDB and Neo4j). If you have suggestions for improvements, please let us know.
- deleted 11y ago[deleted]
- creshal 11y ago> PostgreSQL performed very well Despite the fact that you crippled it by not using jsonb columns.
- BlackFingolfin 11y agoWhy so hostile? Personally, I assume this is just an instance of them not being PostgeSQL experts. So, before jumping to conclusions (namely that they deliberatly skewed their test, despite being very open in the way they described the setup), I'll instead wait and see -- perhaps they'll explain why they used json instead of jsonb, or perhaps (better!) they can updat the test to also include jsonb.
- swasheck 11y ago> I assume this is just an instance of them not being PostgeSQL experts. this illuminates the need for "experts" in the different components in the stack. it's intellectually dishonest to claim that a technology is not up to the task when it's not been properly treated in the first place.
- amirouche 11y agothis is more constructive that flooding the communications channels with hidden facts like Neo4j does.
- jbverschoor 11y agoand now a 10-node cluster
- exo762 11y agoHugged to death. https://archive.is/cMWCQ https://archive.is/cMWCQ
- ifcologne 11y agoNo, running on XXXXX Cloud. :( We currently look into it. Thank's for the mirrored page.
- howdoipython 11y ago>Error establishing a database connection
- n72 11y agoClicking the link got me "Error establishing a database connection." :/
- baldfat 11y agoI was kind of shocked how good PostgesSQL did. I still think PostgresSQL and MariaDB are a better tool for most jobs considered big data.
- yahliwharton 11y agoThis is not a cluster test. NoSQL databases in general are optimized for scaling horizontally on commodity hardware. That's more tricky in RDBMS.
- collyw 11y agoMost people I have spoken to are using NoSQL on single nodes.
- brightball 11y agoI think a lot of people jump on NoSQL solutions when scale out isn't needed because of the permeating "do everything in the application / client" mentality these days.
- JamesMcMinn 11y agoPostgres was actually somewhat crippled in these tests since they used json rather than jsonb for storage, which stores the json in a binary format which doesn't need to be serialised on reads.
- lobster_johnson 11y agoThat's not quite correct. The jsonb requires that reads deserialize jsonb into textual JSON, whereas the json type can be sent directly to the client with no processing. jsonb is superior when: 1. You want to use any of the built-in JSON functions, e.g. for extracting fields from the document. 2. You want to index the JSON (either the entire thing via GIN, or individual fields via ordinary B-tree indexes). 3. You want to save space; jsonb strips whitespace. jsonb incurs an overhead on both reads and writes since it must serialize to/from textual JSON.
- hardwaresofton 11y agoNo RethinkDB?
- nevi-me 11y agoLike others mention here, I'm skeptical of these types of comparisons. If I compare myself to my competitors, I won't publish results if they're better than me. I tried ArangoDB about a year ago, I think I still have the branch that I tried it on. After spending a weekend porting some stuff from MongoDB to Arango, I ended up regretting doing that by Sunday evening. It'd be nice to fire things up, update the branch's code and see how it performs.
- curiousjorge 11y agoI have looked at ArangoDB and really hope it takes off, it has some pretty nifty features I think just at this point the lack of integration with frameworks like Meteor.js is holding me back.
- ilaksh 11y agoWhy not include redis or rethinkdb?
- amirouche 11y agoredis and rethinkdb are not ACID across documents. So it's not the same usecase at all.
- merlincorey 11y agoAre you implying that MongoDB and friends are ACID across documents?
- acjohnson55 11y agoComparison of X1, X2, ... , Xn, Y, written by Y => suspicion
- covi 11y agoThe graph dataset is too small in size. It makes little sense for real-world usage.
- ifcologne 11y agoIngo from ArangoDB: Despite it's the whole dataset of a real-world use case. :) https://snap.stanford.edu/data/soc-pokec.html https://snap.stanford.edu/data/soc-pokec.html But of course, you need to test and decide on basis of your individual requirements and use cases.
- covi 11y agoIngo - SNAP has a bunch of other "real-world use case" graphs available for free, many of which larger than this 1M-node, 30M-edge toy. I've done a bunch of related benchmarkings, and the smallest real-world dataset I've used is the largest one on SNAP: orkut.
- Mindstormy 11y agoWould love to see the results for CouchDB in comparison to these.
- kbenson 11y agoI wonder why there's not the equivalent to the Frameworks Benchmark[1] for databases. It seems we could all really benefit from that. Ideally it would get to a place where they would be able to simulate real-world worst case scenarios and test for problems. Each database would likely want multiple entries with different configs, but if you have some engineered failure scenarios and tests in the results it becomes obvious what the trade-off is. Sure, a specific setting may reduce consistency in the event of a failure for speed, but sometimes that's what you might want, and if the failure cases clearly show the problem, at least you aren't going in blind. 1: https://www.techempower.com/benchmarks/ https://www.techempower.com/benchmarks/
- crudbug 11y agoHaving benchmarks for different storage models : Relational/Document/Graph/Object/XML, would be a better solution.
- don71 11y agoI'm Claudius, author of the tests. I've been asked to include a lot of different databases into the test runs. The most requested databases were Postgres/JSON and RethinkDB. I started with Postgres. The Postgres manual states that JSONB might be faster, but some StackOverflow answers indicate that it takes more space than JSON, while JSON might be slightly more compatible with legacy code. I've shown the queries and setup to some local Postgres users. They did not point that JSONB will be much faster for the kinds of requests used in the test setup. For instance, we do not use special indexes apart from the primary one by choice. I wanted to move on to RethinkDB next, but I see your point that a comparison between the different JSON formats of Postgres can also be very enlightening. This should replace guessing with hard facts. As always I will update the blog post and add this tests as well - as we did in the past, see https://www.arangodb.com/nosql-performance-blog-series/ https://www.arangodb.com/nosql-performance-blog-series/. If you have any improvements concerning the configuration of Postgres or SQL queries, I'm will be more than happy to include them as well in the update. I will push the used configuration to GITHUB as well.
- chucky_z 11y agoPlease refer to the #postgresql channel on irc.freenode.net for any postgres inquiries, you will receive an answer from experts and core developers on the correct processes within minutes for almost any question. It is a very active channel full of knowledgeable folks.
- oralhistory 11y ago> For instance, we do not use special indexes apart from the primary one by choice. For instance, we didn't use the index that makes the database go fast to make our own database look good.
- jerven 11y agoI am just going to say: have a try with the LDBC social benchmark http://ldbcouncil.org/ http://ldbcouncil.org/ and http://ldbcouncil.org/benchmarks http://ldbcouncil.org/benchmarks. Where you can even have audited results. These are also graph database benchmarks that are synthetic, designed to look like real data and are quite hard to do well on. As someone responsible for a public free to use deployment of a graph database with more than 2 billion nodes and 15 billion edges (sparql.uniprot.org) I must say this looks like a SPARQL benchmark from 10 years ago.
- crudbug 11y agoWould like to see - Titan with Cassandra backend here.
- lobster_johnson 11y agoOut of interest, which version of Titan are you on? I see that 1.0 was released recently, with little apparent fanfare.
- pella 11y agoor with http://www.scylladb.com/ http://www.scylladb.com/ backend. "ScyllaDB: world's fastest NoSQL column store database; Fully compatible with Apache Cassandra at 10x the throughput and jaw dropping low latency"