6 ms·

I get that this is trying to point out that neo4j shouldn't be that far behind, but why are the i7/gatling test numbers being directly compared to memgraphs g6

by mhio 4y ago

I get that this is trying to point out that neo4j shouldn't be that far behind, but why are the i7/gatling test numbers being directly compared to memgraphs g6 test results? The conclusion is a bit premature without the other half of the test... What performance does memgraph have on the newer, single socket hardware?

junon 4y ago
Yeah that was strange, it's my understanding that you can't compare benchmarks between different machines, especially if they're not 1:1 identical hardware. If you're referring to this line, then it struct me as very odd. > Instead of 112 queries per second, I get 531q/s. Instead of a p99 latency of 94.49ms, I get 28ms with a min, mean, p50, p75 and p95 of 14ms to 18ms. Alright, what about query 2? Same story. Otherwise, the article holds up.
- robertlagrant 4y ago
  Author is just stating the differences between the benchmarketing hardware and his own. Not comparing new hardware and one DB with old hardware and other DB.
  - roetlich 4y ago
    Seems like he does in his conclusion: > It looks like Neo4j is faster than Memgraph in the Aggregate queries by about 3 times.
    - robertlagrant 4y ago
      Having re-read it, I now can't decide. It would be a little silly if the author is completely different devices, so I'm going to stick to that interpretation.
  - mhio 4y ago
    The bottom table contains the memgraph mgBench (G6 2x Xeon X5650) "hot run, medium, isolated" throughput results: https://memgraph.com/benchgraph/base?condition=hot&datasetSize=medium&workloadType=mixed&querySelection=%5B%7B%22index%22:0,%22queries%22:%5B0,1,2,3%5D%7D,%7B%22index%22:1,%22queries%22:%5B0,1,2,3,4,5,6,7,8,9,10,11,12,13,14%5D%7D,%7B%22index%22:2,%22queries%22:%5B0%5D%7D%5D&tab=Global%20Results https://memgraph.com/benchgraph/base?condition=hot&datasetSi... The "By" columns compares those results to the new test suite on the ~10y newer cpu.
  - junon 4y ago
    But those are needless comparisons that make no sense to even mention. Of course the performance profile is different. That was the original point, if I understand correctly. EDIT: GP followed up, I did not in fact understand correctly.
- roetlich 4y ago
  I think that part is still fine, because there he's only saying that he got different results for the same test on his hardware, which might help to set a baseline. Really weird is the table after "Let’s see the breakdown". It's not super clearly labeled so I'm not fully sure which data is which, but it looks he's comparing neo4j on his machine to memgraph on their older hardware, that would be very silly. Looking at the source for the benchmark, that also seems to hold.
- mhio 4y ago
  It was... > It looks like Neo4j is faster than Memgraph in the Aggregate queries by about 3 times. Memgraph is faster than Neo4j for the queries they selected by about 2-3x except... Unless that's meant to be a joke? Maybe they were dunking on the "bullshit" benchmark with a worse comparison.
  - junon 4y ago
    Oh, then I'm not sure what you mean. That line makes sense - the final benchmarks were performed on the author's machine. That's where the conclusion comes from. In theory, the spread between two benchmarked programs is not going to be hugely different between machines unless one is taking advantage of the hardware of one of the machines where the other doesn't (e.g. new syscall mechanisms such as io_uring, SIMD support, multithreading in some cases, etc). 2-3x is a much more reasonable spread than 100x. If they really did have 100x speedups, then the culprit may be the fact they're using obscenely old hardware, which would be disingenuous given that people are not typically running graph databases on such infrastructure anymore.
    - roetlich 4y ago
      > the final benchmarks were performed on the author's machine No, the author didn't run any benchmarks for Memgraph AFAICT, only for Neo4j. The numbers for mempgraph at the end are from the old benchmark, so on the old hardware.
      - junon 4y ago
        I don't think that's right. That's not how I understood the article.
        aeyes 4y ago
        The author published the code and I only see adapters which interface with Neo4J: https://github.com/maxdemarzi/memgraph_benchmark/tree/main/src/test/scala https://github.com/maxdemarzi/memgraph_benchmark/tree/main/s... The numbers for Memgraph match what is shown on their benchmark website.

by mhio 4y ago

junon 4y ago

Yeah that was strange, it's my understanding that you can't compare benchmarks between different machines, especially if they're not 1:1 identical hardware. If you're referring to this line, then it struct me as very odd. > Instead of 112 queries per second, I get 531q/s. Instead of a p99 latency of 94.49ms, I get 28ms with a min, mean, p50, p75 and p95 of 14ms to 18ms. Alright, what about query 2? Same story. Otherwise, the article holds up.

robertlagrant 4y ago
Author is just stating the differences between the benchmarketing hardware and his own. Not comparing new hardware and one DB with old hardware and other DB.
- roetlich 4y ago
  Seems like he does in his conclusion: > It looks like Neo4j is faster than Memgraph in the Aggregate queries by about 3 times.
  - robertlagrant 4y ago
    Having re-read it, I now can't decide. It would be a little silly if the author is completely different devices, so I'm going to stick to that interpretation.
- mhio 4y ago
  The bottom table contains the memgraph mgBench (G6 2x Xeon X5650) "hot run, medium, isolated" throughput results: https://memgraph.com/benchgraph/base?condition=hot&datasetSize=medium&workloadType=mixed&querySelection=%5B%7B%22index%22:0,%22queries%22:%5B0,1,2,3%5D%7D,%7B%22index%22:1,%22queries%22:%5B0,1,2,3,4,5,6,7,8,9,10,11,12,13,14%5D%7D,%7B%22index%22:2,%22queries%22:%5B0%5D%7D%5D&tab=Global%20Results https://memgraph.com/benchgraph/base?condition=hot&datasetSi... The "By" columns compares those results to the new test suite on the ~10y newer cpu.
- junon 4y ago
  But those are needless comparisons that make no sense to even mention. Of course the performance profile is different. That was the original point, if I understand correctly. EDIT: GP followed up, I did not in fact understand correctly.
roetlich 4y ago
I think that part is still fine, because there he's only saying that he got different results for the same test on his hardware, which might help to set a baseline. Really weird is the table after "Let’s see the breakdown". It's not super clearly labeled so I'm not fully sure which data is which, but it looks he's comparing neo4j on his machine to memgraph on their older hardware, that would be very silly. Looking at the source for the benchmark, that also seems to hold.
mhio 4y ago
It was... > It looks like Neo4j is faster than Memgraph in the Aggregate queries by about 3 times. Memgraph is faster than Neo4j for the queries they selected by about 2-3x except... Unless that's meant to be a joke? Maybe they were dunking on the "bullshit" benchmark with a worse comparison.
- junon 4y ago
  Oh, then I'm not sure what you mean. That line makes sense - the final benchmarks were performed on the author's machine. That's where the conclusion comes from. In theory, the spread between two benchmarked programs is not going to be hugely different between machines unless one is taking advantage of the hardware of one of the machines where the other doesn't (e.g. new syscall mechanisms such as io_uring, SIMD support, multithreading in some cases, etc). 2-3x is a much more reasonable spread than 100x. If they really did have 100x speedups, then the culprit may be the fact they're using obscenely old hardware, which would be disingenuous given that people are not typically running graph databases on such infrastructure anymore.
  - roetlich 4y ago
    > the final benchmarks were performed on the author's machine No, the author didn't run any benchmarks for Memgraph AFAICT, only for Neo4j. The numbers for mempgraph at the end are from the old benchmark, so on the old hardware.
    - junon 4y ago
      I don't think that's right. That's not how I understood the article.
      - aeyes 4y ago
        The author published the code and I only see adapters which interface with Neo4J: https://github.com/maxdemarzi/memgraph_benchmark/tree/main/src/test/scala https://github.com/maxdemarzi/memgraph_benchmark/tree/main/s... The numbers for Memgraph match what is shown on their benchmark website.