4 ms·

I know of no database products that write 128-byte blocks. Most are at least 2KB, if not larger (SQL Server and pgsql are 8KB blocks). Yes, you could conceivabl

I know of no database products that write 128-byte blocks. Most are at least 2KB, if not larger (SQL Server and pgsql are 8KB blocks). Yes, you could conceivably imagine up some hypothetical situation that would be bad for an SSD, but it is incredibly unlikely. And when I say 100GB of writes, I mean, literally, 100GB of writes, which in actual source data is generally much, much, much, much smaller.

dillondf 12y ago
You do understand that flash erase blocks are around 128KB now, right? Random writes to a SSD can only be write-combined to a point. Once the SSD has to actually start erasing blocks and doing collections of those randomly combined sectors things can get dicey real fast in the face of further random writes. It doesn't have a magic wand. The point is that the SSD has no way to determine ahead of time what the rewrite order is going to be as random blocks continue to get written. You can rationalize it all you want, the write amplification is going to happen everywhere in the chain to one degree or another. For that matter, filesystems are ganging blocks together now too (and have been for a long time). It is not uncommon to see 16KB-64KB filesystem block sizes. Nobody gives a shit about a mere 100GB in physical writes to a storage subsystem in a day. A single consumer 512GB crucial can handle a rate like that for almost 30 years. Write amplification effects are not to be underestimated, but fortunately there are only a very few situations where it's an issue. And as I said, the situations can be largely mitigated by database writers getting off their butts and fixing their backends to push the complexity to the indexes and away from trying to optimize the linear layout of the database by seek-writing into it. -Matt
- engendered 12y ago
  You do understand that flash erase blocks are around 128KB now, right? How is that relevant to your wrong claim about database 128-byte writes? You word that as if you're correcting me. I don't even know what point you're trying to make anymore, but you're trying to buttress your argument with what can best be described as diversions. Most people write far less than they think. Databases aren't particularly magical, any more than the web server log file that is written to a line at a time. Yes, people "give a shit" about a "mere" 100GB of writes, because the vast majority of real projects, including at major corporations, write far less than that per day. So are we just talking about dick measuring now?
  - dillondf 12y ago
    I'm sorry, engendered... but are you an idiot? Do you even understand the context of the conversation? -Matt
    - dang 12y ago
      > are you an idiot? Do you even understand This comment breaks the HN guidelines. We'd appreciate it if you'd read the site rules and follow them: https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newswelcome.html https://news.ycombinator.com/newswelcome.html
  - AlisdairO 12y ago
    A block might be 8KB, but the actual update you're making to the block might be much smaller. I imagine by '128 byte write' he's talking about a lot of random row updates, where each row is 128 bytes. Now, if you're not too unlucky, many updates will be combined on the same page per checkpoint, but that's not a given. On the other hand, it's reasonably likely that several updates will be combined per erase block per checkpoint. A heavily indexed table can exhibit some pretty random write patterns, though. Additionally, the WAL has to be synced to disk every commit (unlike a web server log file), and WAL records can be very small. WAL is of course append-only, so you'd hope that a good SSD with a battery/cap backup would cache the writes and flush on the SSD erase block filling up.
    - engendered 12y ago
      A block might be 8KB, but the actual update you're making to the block might be much smaller. All major databases will only deal in the 8KB increments (or whatever their block size is, whether larger or smaller, but never as small as originally claimed) though. They don't write less. Indeed, it's worth noting that most (every single major one) database systems actually write to a sequential transaction log (which they do not have to checkpoint every n-bytes), and only on commit do they actually then make a strategy for changing those source pages and extents, unrolling it from the log and checkpointing it, which by default includes coalescing and combining strategies. The idea that databases are randomly writing small bits of data all over the place is simply wrong, but is the entire foundation of almost every comment in this thread. https://technet.microsoft.com/en-us/library/aa337560%28v=sql.105%29.aspx https://technet.microsoft.com/en-us/library/aa337560%28v=sql... As one example. Oracle, pgsql, mysql, and others do the exact same thing. They aren't randomly changing an int here and a bool there. I worked on a financial system where we wrote just absurd amounts of data a day. We ran it on a FusionIO SLC device (with a mirror backup), and churned the data essentially around the clock. After three years the FusionIO little lifespan hadn't even moved. tldr; people grossly overestimate the "magical" nature of databases.
- pkaye 12y ago
  4KB random writes should have a WAF of 2-7 depending on over provisioning and assuming random writes. But real workload are not purely random so it will be better than that.
- deleted 12y ago
  [deleted]

dillondf 12y ago

You do understand that flash erase blocks are around 128KB now, right? Random writes to a SSD can only be write-combined to a point. Once the SSD has to actually start erasing blocks and doing collections of those randomly combined sectors things can get dicey real fast in the face of further random writes. It doesn't have a magic wand. The point is that the SSD has no way to determine ahead of time what the rewrite order is going to be as random blocks continue to get written. You can rationalize it all you want, the write amplification is going to happen everywhere in the chain to one degree or another. For that matter, filesystems are ganging blocks together now too (and have been for a long time). It is not uncommon to see 16KB-64KB filesystem block sizes. Nobody gives a shit about a mere 100GB in physical writes to a storage subsystem in a day. A single consumer 512GB crucial can handle a rate like that for almost 30 years. Write amplification effects are not to be underestimated, but fortunately there are only a very few situations where it's an issue. And as I said, the situations can be largely mitigated by database writers getting off their butts and fixing their backends to push the complexity to the indexes and away from trying to optimize the linear layout of the database by seek-writing into it. -Matt

engendered 12y ago
You do understand that flash erase blocks are around 128KB now, right? How is that relevant to your wrong claim about database 128-byte writes? You word that as if you're correcting me. I don't even know what point you're trying to make anymore, but you're trying to buttress your argument with what can best be described as diversions. Most people write far less than they think. Databases aren't particularly magical, any more than the web server log file that is written to a line at a time. Yes, people "give a shit" about a "mere" 100GB of writes, because the vast majority of real projects, including at major corporations, write far less than that per day. So are we just talking about dick measuring now?
- dillondf 12y ago
  I'm sorry, engendered... but are you an idiot? Do you even understand the context of the conversation? -Matt
  - dang 12y ago
    > are you an idiot? Do you even understand This comment breaks the HN guidelines. We'd appreciate it if you'd read the site rules and follow them: https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newswelcome.html https://news.ycombinator.com/newswelcome.html
- AlisdairO 12y ago
  A block might be 8KB, but the actual update you're making to the block might be much smaller. I imagine by '128 byte write' he's talking about a lot of random row updates, where each row is 128 bytes. Now, if you're not too unlucky, many updates will be combined on the same page per checkpoint, but that's not a given. On the other hand, it's reasonably likely that several updates will be combined per erase block per checkpoint. A heavily indexed table can exhibit some pretty random write patterns, though. Additionally, the WAL has to be synced to disk every commit (unlike a web server log file), and WAL records can be very small. WAL is of course append-only, so you'd hope that a good SSD with a battery/cap backup would cache the writes and flush on the SSD erase block filling up.
  - engendered 12y ago
    A block might be 8KB, but the actual update you're making to the block might be much smaller. All major databases will only deal in the 8KB increments (or whatever their block size is, whether larger or smaller, but never as small as originally claimed) though. They don't write less. Indeed, it's worth noting that most (every single major one) database systems actually write to a sequential transaction log (which they do not have to checkpoint every n-bytes), and only on commit do they actually then make a strategy for changing those source pages and extents, unrolling it from the log and checkpointing it, which by default includes coalescing and combining strategies. The idea that databases are randomly writing small bits of data all over the place is simply wrong, but is the entire foundation of almost every comment in this thread. https://technet.microsoft.com/en-us/library/aa337560%28v=sql.105%29.aspx https://technet.microsoft.com/en-us/library/aa337560%28v=sql... As one example. Oracle, pgsql, mysql, and others do the exact same thing. They aren't randomly changing an int here and a bool there. I worked on a financial system where we wrote just absurd amounts of data a day. We ran it on a FusionIO SLC device (with a mirror backup), and churned the data essentially around the clock. After three years the FusionIO little lifespan hadn't even moved. tldr; people grossly overestimate the "magical" nature of databases.
pkaye 12y ago
4KB random writes should have a WAF of 2-7 depending on over provisioning and assuming random writes. But real workload are not purely random so it will be better than that.
deleted 12y ago
[deleted]