5 ms·

I really, really distrust these kinds of evaluations when they come from someone whose product is included in the comparison. Even if everything is above-board,

by cwyers 11y ago

I really, really distrust these kinds of evaluations when they come from someone whose product is included in the comparison. Even if everything is above-board, they're not going to publish if it shows their product just completely sucks at it. That kind of publication bias makes these kinds of results a lot less trustworthy than independent benchmarks even if you assume the best of intentions from the people putting them out there.

creshal 11y ago
The particular benchmark seems bullshit, too. For postgres they seem to intentionally use the less performant json column type instead of jsonb. Not that I can verify it, because the code in the linked public "No magic, no tricks – check the code and make your own tests!" repository doesn't match the published results and doesn't even work at all with postgres… EDIT: Okay, they pushed a new version containing the Postgres data now. They ARE using the cripplingly slow json columns, not jsonb columns recommended by the documentation.
- andy_ppp 11y ago
  Just to be a pedant, the JSONB format as I understand is marginally slower at inserts and orders of magnitude faster at everything else...
  - robconery 11y ago
    It's fractionally slower, true, because of the serialization hit (string to binary). The real juice comes from the GIN index - and if you apply it to specific columns instead of a complete document, you have a rocket ship on read.
    - creshal 11y ago
      > It's fractionally slower, true, because of the serialization hit (string to binary). And with JSON columns you have to serialize on accesses, which is a lot slower in the read-mostly tests.
  - SilasX 11y ago
    Stupid question: isn't that the typical result of indexing? Slower writes for (much) faster lookups?
- jpgvm 11y ago
  And despite that Postgres destroys all the other solutions at everything other than the non-sync (lol) single write case and graph traversal. If anything it just proves even after almost a decade of these "NoSQL" solutions being around they still can't compete even on basic queries with Postgres which is a fairly conservative SQL solution.
  - BlackFingolfin 11y ago
    I think this is a classical case of "use the right tool for the right purpose". If you want to do "classic database" stuff, use a classic SQL database. But if e.g. graph traversal is a cruicial operation for your application, then looking at NoSQL solution seems to be quite interesting. In other words: I wouldn't call a screw driver a bad tool, just because it's not as good at driving nails into wood as a hammer.
    - KingMob 11y ago
      I think the issue is how many NoSQL solutions over recent years have billed themselves as the default start-with-us data store, and SQL as the more complicated niche product...when it's really the other way around.
deleted 11y ago
[deleted]
qaqy 11y ago
WTF with VCs investing into all this crap? There are very few scenarios were you would be better off with NoSQL solution and there are established players serving those niches already.
ifcologne 11y ago
Ingo from ArangoDB here. I agree that vendor tests are always biased, of course you want to show that your product is competitive. But as there is no independent institution that compared our product and as we want to know where we stand with ArangoDB, Claudius did his own tests. And as the work is already done, why not share it. We tried our best to do it as open as possible. PostgreSQL performed very well and we have a problem with memory consumption - have a look at the charts, we will try to improve there. - Every database configuration is public - All test scripts are available on Github - We publish updates if we get pull-requests or comments with suggestions for improvements We did that before and after the last test, some database vendors sent us improved snapshots of their databases which found their way into the latest products (OrientDB and Neo4j). If you have suggestions for improvements, please let us know.
- deleted 11y ago
  [deleted]
- creshal 11y ago
  > PostgreSQL performed very well Despite the fact that you crippled it by not using jsonb columns.
  - BlackFingolfin 11y ago
    Why so hostile? Personally, I assume this is just an instance of them not being PostgeSQL experts. So, before jumping to conclusions (namely that they deliberatly skewed their test, despite being very open in the way they described the setup), I'll instead wait and see -- perhaps they'll explain why they used json instead of jsonb, or perhaps (better!) they can updat the test to also include jsonb.
    - swasheck 11y ago
      > I assume this is just an instance of them not being PostgeSQL experts. this illuminates the need for "experts" in the different components in the stack. it's intellectually dishonest to claim that a technology is not up to the task when it's not been properly treated in the first place.
amirouche 11y ago
this is more constructive that flooding the communications channels with hidden facts like Neo4j does.

by cwyers 11y ago

creshal 11y ago

The particular benchmark seems bullshit, too. For postgres they seem to intentionally use the less performant json column type instead of jsonb. Not that I can verify it, because the code in the linked public "No magic, no tricks – check the code and make your own tests!" repository doesn't match the published results and doesn't even work at all with postgres… EDIT: Okay, they pushed a new version containing the Postgres data now. They ARE using the cripplingly slow json columns, not jsonb columns recommended by the documentation.

andy_ppp 11y ago
Just to be a pedant, the JSONB format as I understand is marginally slower at inserts and orders of magnitude faster at everything else...
- robconery 11y ago
  It's fractionally slower, true, because of the serialization hit (string to binary). The real juice comes from the GIN index - and if you apply it to specific columns instead of a complete document, you have a rocket ship on read.
  - creshal 11y ago
    > It's fractionally slower, true, because of the serialization hit (string to binary). And with JSON columns you have to serialize on accesses, which is a lot slower in the read-mostly tests.
- SilasX 11y ago
  Stupid question: isn't that the typical result of indexing? Slower writes for (much) faster lookups?
jpgvm 11y ago
And despite that Postgres destroys all the other solutions at everything other than the non-sync (lol) single write case and graph traversal. If anything it just proves even after almost a decade of these "NoSQL" solutions being around they still can't compete even on basic queries with Postgres which is a fairly conservative SQL solution.
- BlackFingolfin 11y ago
  I think this is a classical case of "use the right tool for the right purpose". If you want to do "classic database" stuff, use a classic SQL database. But if e.g. graph traversal is a cruicial operation for your application, then looking at NoSQL solution seems to be quite interesting. In other words: I wouldn't call a screw driver a bad tool, just because it's not as good at driving nails into wood as a hammer.
  - KingMob 11y ago
    I think the issue is how many NoSQL solutions over recent years have billed themselves as the default start-with-us data store, and SQL as the more complicated niche product...when it's really the other way around.

deleted 11y ago

[deleted]

qaqy 11y ago

WTF with VCs investing into all this crap? There are very few scenarios were you would be better off with NoSQL solution and there are established players serving those niches already.

ifcologne 11y ago

Ingo from ArangoDB here. I agree that vendor tests are always biased, of course you want to show that your product is competitive. But as there is no independent institution that compared our product and as we want to know where we stand with ArangoDB, Claudius did his own tests. And as the work is already done, why not share it. We tried our best to do it as open as possible. PostgreSQL performed very well and we have a problem with memory consumption - have a look at the charts, we will try to improve there. - Every database configuration is public - All test scripts are available on Github - We publish updates if we get pull-requests or comments with suggestions for improvements We did that before and after the last test, some database vendors sent us improved snapshots of their databases which found their way into the latest products (OrientDB and Neo4j). If you have suggestions for improvements, please let us know.

deleted 11y ago
[deleted]
creshal 11y ago
> PostgreSQL performed very well Despite the fact that you crippled it by not using jsonb columns.
- BlackFingolfin 11y ago
  Why so hostile? Personally, I assume this is just an instance of them not being PostgeSQL experts. So, before jumping to conclusions (namely that they deliberatly skewed their test, despite being very open in the way they described the setup), I'll instead wait and see -- perhaps they'll explain why they used json instead of jsonb, or perhaps (better!) they can updat the test to also include jsonb.
  - swasheck 11y ago
    > I assume this is just an instance of them not being PostgeSQL experts. this illuminates the need for "experts" in the different components in the stack. it's intellectually dishonest to claim that a technology is not up to the task when it's not been properly treated in the first place.

amirouche 11y ago

this is more constructive that flooding the communications channels with hidden facts like Neo4j does.