5 ms·

Whenever one sees a claim about new database technology outperforming standbys like MySQL and Postgres by factors of 10x or more, it's a good idea to suspect th

Whenever one sees a claim about new database technology outperforming standbys like MySQL and Postgres by factors of 10x or more, it's a good idea to suspect the out-of-the-box configuration isn't really durable. What kills write performance in databases is the time it takes to write to disk, not the software overhead.

gouranga 14y ago
Or in the case of a lot of vendors we won't mention - blatant lies.
- 16s 14y ago
  Vendors lie. That is the sole truth of IT purchasing. When testing network gear, some of our network engineers would say, "Vendors lie, but packets don't".
  - cluda01 14y ago
    Nice quote! More generally when considering any solution you should always test it in your environment according to your performance needs BEFORE you buy it. Any vendor worth their salt will let you do this.
    - gouranga 14y ago
      From experience, that's pretty much no vendor these days. I will say positive things for Microsoft: you can always trial their software before you buy it.
      - cluda01 14y ago
        We tested Azul Systems Zing JVM (pauseless GC) inhouse. They sent out support engineers and gave us great presentations to boot. Good company.
- dredmorbius 14y ago
  Are you specifically not mentioning Oracle by any chance?
  - gouranga 14y ago
    I'm specifically making no comment on that ;-)
    - jacques_chester 14y ago
      And according to the standard Oracle contract ... you can't.
      - gouranga 14y ago
        They didn't say you can't make no comment though!
    - deleted 14y ago
      [deleted]
wheaties 14y ago
Or if they use a global write lock.
endersshadow 14y ago
For what it's worth SQL Server doesn't meet his definition of "durable." It writes a transaction log and then writes those transactions to disk on a checkpoint [1]. Typically, that checkpoint automagically happens, but you can also force it to clear the buffer with a "checkpoint" command. InnoDB also does this, too [2]. I don't know if MemSQL actually has checkpointing or what, but it's just worth noting. [1]: http://msdn.microsoft.com/en-us/library/ms189573.aspx http://msdn.microsoft.com/en-us/library/ms189573.aspx [2]: http://dev.mysql.com/doc/refman/5.0/en/innodb-checkpoints.html http://dev.mysql.com/doc/refman/5.0/en/innodb-checkpoints.ht...
- spudlyo 14y ago
  I think you may be missing the point. Most databases use transaction/redo logs, it has to do with how often the writes to the transaction log are flushed. With proper durability, the transaction log is flushed with every commit, not when buffers become full or a background thread gets around to it.
- __dumbideas 14y ago
  MySQL's InnoDB has configurable durability: http://dev.mysql.com/doc/refman/4.1/en/innodb-parameters.html#sysvar_innodb_flush_log_at_trx_commit http://dev.mysql.com/doc/refman/4.1/en/innodb-parameters.htm...
- jstclair 14y ago
  This is a little misleading; an internal checkpoint happens upon transaction commit[1]. The comment makes it sound as if one or more transactions can commit before a checkpoint writes them to disk. [1]: http://msdn.microsoft.com/en-us/library/ms186259(v=sql.105).aspx http://msdn.microsoft.com/en-us/library/ms186259(v=sql.105)....
  - spudlyo 14y ago
    The transaction is actually written to disk twice. Once when it is written to the transaction log, and then again when a checkpoint happens and it's written to the underlying data blocks. Transaction log writes are sequential and fast, writes to the underlying data blocks are random and slow. The whole point of the checkpoint is to convert this sequential i/o to random i/o in an efficient manner. Still, that's not what we're talking about. We're talking about the difference between writing data to the transaction log, and writing data to the transaction log and then flushing it. Once you call fsync() and the data is flushed you know for a fact (provided the hardware isn't lying to you) the bytes are on the damn platter, and not in some OS buffer cache.
rdtsc 14y ago
Very good observation. Established DBs already do a decent job optimizing the path from network to disk, at least in the most basic one client, one server test. So say they are hitting the 80MB/s throughput. If someone's benchmark shows 800MB/s throughput on same hardware -- you know they are making stuff up.

gouranga 14y ago

Or in the case of a lot of vendors we won't mention - blatant lies.

16s 14y ago
Vendors lie. That is the sole truth of IT purchasing. When testing network gear, some of our network engineers would say, "Vendors lie, but packets don't".
- cluda01 14y ago
  Nice quote! More generally when considering any solution you should always test it in your environment according to your performance needs BEFORE you buy it. Any vendor worth their salt will let you do this.
  - gouranga 14y ago
    From experience, that's pretty much no vendor these days. I will say positive things for Microsoft: you can always trial their software before you buy it.
    - cluda01 14y ago
      We tested Azul Systems Zing JVM (pauseless GC) inhouse. They sent out support engineers and gave us great presentations to boot. Good company.
dredmorbius 14y ago
Are you specifically not mentioning Oracle by any chance?
- gouranga 14y ago
  I'm specifically making no comment on that ;-)
  - jacques_chester 14y ago
    And according to the standard Oracle contract ... you can't.
    - gouranga 14y ago
      They didn't say you can't make no comment though!
  - deleted 14y ago
    [deleted]

wheaties 14y ago

Or if they use a global write lock.

endersshadow 14y ago

For what it's worth SQL Server doesn't meet his definition of "durable." It writes a transaction log and then writes those transactions to disk on a checkpoint [1]. Typically, that checkpoint automagically happens, but you can also force it to clear the buffer with a "checkpoint" command. InnoDB also does this, too [2]. I don't know if MemSQL actually has checkpointing or what, but it's just worth noting. [1]: http://msdn.microsoft.com/en-us/library/ms189573.aspx http://msdn.microsoft.com/en-us/library/ms189573.aspx [2]: http://dev.mysql.com/doc/refman/5.0/en/innodb-checkpoints.html http://dev.mysql.com/doc/refman/5.0/en/innodb-checkpoints.ht...

spudlyo 14y ago
I think you may be missing the point. Most databases use transaction/redo logs, it has to do with how often the writes to the transaction log are flushed. With proper durability, the transaction log is flushed with every commit, not when buffers become full or a background thread gets around to it.
__dumbideas 14y ago
MySQL's InnoDB has configurable durability: http://dev.mysql.com/doc/refman/4.1/en/innodb-parameters.html#sysvar_innodb_flush_log_at_trx_commit http://dev.mysql.com/doc/refman/4.1/en/innodb-parameters.htm...
jstclair 14y ago
This is a little misleading; an internal checkpoint happens upon transaction commit[1]. The comment makes it sound as if one or more transactions can commit before a checkpoint writes them to disk. [1]: http://msdn.microsoft.com/en-us/library/ms186259(v=sql.105).aspx http://msdn.microsoft.com/en-us/library/ms186259(v=sql.105)....
- spudlyo 14y ago
  The transaction is actually written to disk twice. Once when it is written to the transaction log, and then again when a checkpoint happens and it's written to the underlying data blocks. Transaction log writes are sequential and fast, writes to the underlying data blocks are random and slow. The whole point of the checkpoint is to convert this sequential i/o to random i/o in an efficient manner. Still, that's not what we're talking about. We're talking about the difference between writing data to the transaction log, and writing data to the transaction log and then flushing it. Once you call fsync() and the data is flushed you know for a fact (provided the hardware isn't lying to you) the bytes are on the damn platter, and not in some OS buffer cache.

rdtsc 14y ago

Very good observation. Established DBs already do a decent job optimizing the path from network to disk, at least in the most basic one client, one server test. So say they are hitting the 80MB/s throughput. If someone's benchmark shows 800MB/s throughput on same hardware -- you know they are making stuff up.