4 ms·

I don't think these results are comparable. Query runtimes are highly dependent on how much memory they can use. Postgres for example isn't configured making it

I don't think these results are comparable. Query runtimes are highly dependent on how much memory they can use. Postgres for example isn't configured making it use less memory than other systems. Hence its performance is likely inferior and could be improved significantly by increasing the buffer size.

mpv89 9y ago
Hi, Mark from ArangoDB here. We used the default configuration for every Database in the benchmark under the assumption they are picked reasonable. If you have any specific suggestions how to configure Postgres for the specific environment, please let us know.
- aschampion 9y ago
  Postgres is configured extremely conservatively by default. While it isn't reasonable to spend time tweaking each database, at least running pgtune would be reasonable for Postgres.
  - jsteemann 9y ago
    I think it's definitely reasonable to tune important production databases. But for many people databases have become commodity applications that they install and then forget. So even in reality not all installments will be properly tuned.
    - robbyt 9y ago
      Which is why many apps are slow... People search for a new toy, rather than learn what they already have.
    - etxm 9y ago
      Who does this?! I know it happens because I inherit them often. But seriously knock it off. Errrrr. Better yet don’t, job stability. Nothing to see here. Move along.
    - rlpb 9y ago
      > But for many people databases have become commodity applications that they install and then forget. It seems unlikely to me that this group of people would reach benchmark posts.
    - rsynnott 9y ago
      People using them that way probably don't care about performance very much, though.
- jpalomaki 9y ago
  I think the first step would be to use the automatic configuration generation tool [1]. You give few inputs (like the amount of server memory and number of connections), and get the config. [1] https://www.pgconfig.org/#/tuning https://www.pgconfig.org/#/tuning
  - fceller 9y ago
    Thanks a lot for the suggestion. We have used http://pgtune.leopard.in.ua http://pgtune.leopard.in.ua I have appended the resulting config. The result is that the default config is already very good for our benchmark. There is no visible difference between the old and new config when running the benchmark. We will publish an update to the blog post and show the numbers using the tuned config. best Frank DBVersion: 10 Linux, Type: "Mixed type of Applications" 122GB RAM 25 Connections SSD Storage => max_connections = 25 shared_buffers = 31232MB effective_cache_size = 93696MB work_mem = 639631kB maintenance_work_mem = 2GB min_wal_size = 1GB max_wal_size = 2GB checkpoint_completion_target = 0.9 wal_buffers = 16MB default_statistics_target = 100 random_page_cost = 1.1
- jug 9y ago
  But which default database configuration has a reasonable tuning to run a benchmark?
- eloff 9y ago
  Lies, damn lies, and benchmarks. Using default configuration invalidates the result at best, at worst it's knowingly misleading. Why? Because nobody who cares about performance runs their database with default configuration. Plus some databases, like PostgreSQL are well known to have very conservative settings. So benchmarking with the defaults like that is knowingly trying to make your product look better than it is. You didn't know? Then what are you doing benchmarking databases?!? Seriously, this is shady as hell.
- bsg75 9y ago
  That is a terrible approach. No database platform is tuned by default. You may have to run a utility, or edit a config file, but a new system must be configured to fully utilize available hardware, and for the expected workload. A benchmark run on default config is at best misleading - it smells like marketing spam. You actually dampen interest in a forum like this with such an approach.
  - rpedela 9y ago
    > No database platform is tuned by default. Not really true any more. Many of the newer databases (e.g. Mongo) and database-like systems (e.g. Elasticsearch) try to give a good default experience.
    - bsg75 9y ago
      Since I don't know about current versions of MongoDB, and am genuinely interested, how does Mongo know how much memory to use as an example? Typically with PostgreSQL, you have to expand the defaults when running on modern server class hardware, so the engine uses enough RAM. Simply letting it use all RAM available can be problematic, since the DB engine and OS have to fight for cache, etc. Java based systems like Elastic might still need JVM tuning, especially with a distributed platform like ES, which might be run in a large number of small machines, a smaller number of large machines, or containers.
      - dijit 9y ago
        You are exactly right. You must adjust the java heap size on Elasticsearch.
  - don71 9y ago
    Disclaimer: I'm part of the ArangoDB team. As written in the post the whole benchmark is open source. The idea is that you can run it on your own. Also, pull requests are welcome. If you think it's marketing spam, take the chance and improve the configuration. We will publish an update of the post.

mpv89 9y ago

Hi, Mark from ArangoDB here. We used the default configuration for every Database in the benchmark under the assumption they are picked reasonable. If you have any specific suggestions how to configure Postgres for the specific environment, please let us know.

aschampion 9y ago
Postgres is configured extremely conservatively by default. While it isn't reasonable to spend time tweaking each database, at least running pgtune would be reasonable for Postgres.
- jsteemann 9y ago
  I think it's definitely reasonable to tune important production databases. But for many people databases have become commodity applications that they install and then forget. So even in reality not all installments will be properly tuned.
  - robbyt 9y ago
    Which is why many apps are slow... People search for a new toy, rather than learn what they already have.
  - etxm 9y ago
    Who does this?! I know it happens because I inherit them often. But seriously knock it off. Errrrr. Better yet don’t, job stability. Nothing to see here. Move along.
  - rlpb 9y ago
    > But for many people databases have become commodity applications that they install and then forget. It seems unlikely to me that this group of people would reach benchmark posts.
  - rsynnott 9y ago
    People using them that way probably don't care about performance very much, though.
jpalomaki 9y ago
I think the first step would be to use the automatic configuration generation tool [1]. You give few inputs (like the amount of server memory and number of connections), and get the config. [1] https://www.pgconfig.org/#/tuning https://www.pgconfig.org/#/tuning
- fceller 9y ago
  Thanks a lot for the suggestion. We have used http://pgtune.leopard.in.ua http://pgtune.leopard.in.ua I have appended the resulting config. The result is that the default config is already very good for our benchmark. There is no visible difference between the old and new config when running the benchmark. We will publish an update to the blog post and show the numbers using the tuned config. best Frank DBVersion: 10 Linux, Type: "Mixed type of Applications" 122GB RAM 25 Connections SSD Storage => max_connections = 25 shared_buffers = 31232MB effective_cache_size = 93696MB work_mem = 639631kB maintenance_work_mem = 2GB min_wal_size = 1GB max_wal_size = 2GB checkpoint_completion_target = 0.9 wal_buffers = 16MB default_statistics_target = 100 random_page_cost = 1.1
jug 9y ago
But which default database configuration has a reasonable tuning to run a benchmark?
eloff 9y ago
Lies, damn lies, and benchmarks. Using default configuration invalidates the result at best, at worst it's knowingly misleading. Why? Because nobody who cares about performance runs their database with default configuration. Plus some databases, like PostgreSQL are well known to have very conservative settings. So benchmarking with the defaults like that is knowingly trying to make your product look better than it is. You didn't know? Then what are you doing benchmarking databases?!? Seriously, this is shady as hell.
bsg75 9y ago
That is a terrible approach. No database platform is tuned by default. You may have to run a utility, or edit a config file, but a new system must be configured to fully utilize available hardware, and for the expected workload. A benchmark run on default config is at best misleading - it smells like marketing spam. You actually dampen interest in a forum like this with such an approach.
- rpedela 9y ago
  > No database platform is tuned by default. Not really true any more. Many of the newer databases (e.g. Mongo) and database-like systems (e.g. Elasticsearch) try to give a good default experience.
  - bsg75 9y ago
    Since I don't know about current versions of MongoDB, and am genuinely interested, how does Mongo know how much memory to use as an example? Typically with PostgreSQL, you have to expand the defaults when running on modern server class hardware, so the engine uses enough RAM. Simply letting it use all RAM available can be problematic, since the DB engine and OS have to fight for cache, etc. Java based systems like Elastic might still need JVM tuning, especially with a distributed platform like ES, which might be run in a large number of small machines, a smaller number of large machines, or containers.
    - dijit 9y ago
      You are exactly right. You must adjust the java heap size on Elasticsearch.
- don71 9y ago
  Disclaimer: I'm part of the ArangoDB team. As written in the post the whole benchmark is open source. The idea is that you can run it on your own. Also, pull requests are welcome. If you think it's marketing spam, take the chance and improve the configuration. We will publish an update of the post.