5 ms·

Paul Buchheit: The secret to making things easy: avoid hard problems

33 points by paul 20y ago

brlewis 20y ago
An even better secret to making things easy: only try to solve problems that exist. This is a nonexistent problem: "The problem? Most databases are still storing my name using the same techniques that they did 15 or more years ago -- they seek the disk head to some specific location and then read or write my name to that location. These databases rely on the one operation that DID NOT dramatically improve in the past 15 years! They still perform like it's 1991." I don't think PostgreSQL is the only database that uses write-ahead logging.
- paul 20y ago
  Post some numbers. How many updates per second can you do?
  - brlewis 20y ago
    I don't know. If that becomes a problem I'll put the write-ahead log on a faster disk to buy time to look into it.
    - mattculbreth 20y ago
      Why can't you just succumb to peer pressure and get rid of that database? :)
      - brlewis 20y ago
        I'm not sure. First I'm going to get rid of that operating system. It has lots of features that aren't relevant to my web app.
        paul 20y ago
        It's somewhat amusing, but also mildly depressing, to see these remarks despite the fact that I explicitly stated (in bold, even), that "I'm not advising you to ditch your database!" And yet somehow that gets translated into "get rid of that database."
        mattculbreth 20y ago
        Don't be depressed. You started something that's spawned (as of now) 28 comments. Great thread.
        brlewis 20y ago
        Sorry it's mildly depressing. Unfortunately, any thoughts you bring up related to a polarizing issue are going to end up being framed in terms of the polarization. That's just the way it is; some things will never change. On the up side, you are going to learn a lot about essay writing, and about how concepts do/don't get through to people. These are very valuable skills.
    - paul 20y ago
      But wait, didn't you say it was a non-existent problem?
      - brlewis 20y ago
        Someday updates per second might be a problem. The nonexistent problem is "they seek the disk head to some specific location and then read or write my name to that location." I may have missed it, but I haven't seen you admit yet that you were wrong. You've been informed now that reads use a memory cache and writes use a write-ahead log, right?
        paul 20y ago
        I'm probably at least partially wrong about everything.. It's true that PostgreSQL uses a write-ahead log, but it's still going to do that seek when it checkpoints, correct? I don't think that it will be able to sustain high transaction rates (100,000 tps for small transactions should be very doable). If you find that this isn't true and that someone has demonstrated these rates, let me know -- it would certainly be good news.
        brlewis 20y ago
        Yes, it will still do that seek when it checkpoints, but I seriously doubt it will do one seek pert transaction at that checkpoint. Still, I doubt 100,000 tps can be achieved with PostgreSQL. As for being wrong, that's one of the privileges you lose with celebrity. Everything you write in your blog is going to be preceded by "The guy who wrote gmail says..." When you say something that looks like it might become a popular misconception, guys like me will jump up and down, scream, and tear at our hair. Or maybe we'll just post terse, cold, unfriendly corrections. We still like you, though. My main issue with your post is the way it comes across as saying databases are overengineering for 99% of web sites, but mainly deals with a performance issue that won't matter for 99% of web sites. I would have no problem if you framed it as high tps vs low tps for certain classes of application, not as simplicity vs overengineering. Using a relational database is a fine default choice for web apps today, just as you expect to have an OS on your server by default.
    - paul 20y ago
    - dfranke 20y ago
      Fast disks are only faster by a factor of two.
      - brlewis 20y ago
        I think I could address any such performance issues within a factor of two growth of updates per second.
  - rtm 20y ago
    167 small transactions per second on MySQL 4.1 / InnoDB / FreeBSD / FFS / SCSI, which looks like one per rotation. Presumably the time required to write a block at the end of the write-ahead log. 5000 per second with --innodb_flush_log_at_trx_commit=0, which only writes the log to disk once per second. The MySQL documentation claims that this configuration does crash-recovery correctly, though you may lose the last second's worth of transactions.
    - paul 20y ago
      BTW, in case anyone is thinking, "why would I need more than 167 transactions per second", imagine that you have a web site that displays everyone's "last active" time (for chat maybe). If users are performing some activity once every 30 sec, then you will be pretty much be at your limit with about 5000 simultaneous users. Robert, did you test MySQL with a parallel load? To give it the best chance at maximizing tps, it will need to be able to combine multiple pending transactions in a single write. Also, can you tell if it's updating it's b-trees? (if not, it may not be able to sustain these rates)
      - rtm 20y ago
        I ran the test with 1,000,000 INSERTs (taking 200 seconds), with the innodb cache size set to only 200,000 bytes. So there's a fair chance it updated the b-trees on disk.
mattjaynes 20y ago
Loved this point: "The thing is, a lot of those difficult problems are irrelevant for 99% of products. For example, "real" databases can handle transactions that are too large to fit in memory. That was probably a really important feature in 1980. Today, you can buy a computer with 32GB of memory for around $5000. How many GB transactions do you suppose Twitter performs? My guess is zero -- I suspect that their average transaction size is closer to 0.0000002 GB (messages are limited to 140 characters)."
Sam_Odio 20y ago
Want an example of "simple but less reliable"? Here's one (in one or two easy steps): 1. All updates go in to memcached, but not the database 2. (optional) A background process occasionally copies entries from memcached to the db. Without this, the values will be completely lost when memcached restarts. Paul - here's an interesting take on your recommended solution: Use solid state ram and forget about step two. You get fast a read/write, without the risk of data loss if the machine restarts. Here's a pretty cool board that will take DDR SDRAM, and supply it with a constant source of power (imitating solid-state memory): http://techreport.com/reviews/2006q1/gigabyte-iram/index.x?pg=1 http://techreport.com/reviews/2006q1/gigabyte-iram/index.x?pg=1 Those who insist on a database, could even store the database on this board. Best of both worlds. Here's an insane graph showing the database transaction rate for the board vs. a conventional hard drive: http://techreport.com/reviews/2006q1/gigabyte-iram/index.x?pg=7 http://techreport.com/reviews/2006q1/gigabyte-iram/index.x?pg=7
- aristus 20y ago
  Solid-state drives greater than 128GB are in OEM testing now. Expect server economics to turn upside-down in about 12 months, when persistent storage seek times get 100X faster.
  - paul 20y ago
    Yeah, solid-state storage is clearly the future for most apps. Flash is over 10x cheaper than DRAM though, so it'd be nice to see some high-performance flash drives as well. It's also nice to not have to worry about losing data during a prolonged power outage.
    - aristus 20y ago
      No need to wait, really. Addonics' SATA/CF adapters for your $80/8GB flash cards cost $35. Even cheap compact flash does random seeks 40x faster than platters. Throughput is a little slow but what the hell, for 115 bucks? If I were building a server farm right now I'd try those things out.

33 points by paul 20y ago

An even better secret to making things easy: only try to solve problems that exist. This is a nonexistent problem: "The problem? Most databases are still storing my name using the same techniques that they did 15 or more years ago -- they seek the disk head to some specific location and then read or write my name to that location. These databases rely on the one operation that DID NOT dramatically improve in the past 15 years! They still perform like it's 1991." I don't think PostgreSQL is the only database that uses write-ahead logging.

paul 20y ago
Post some numbers. How many updates per second can you do?
- brlewis 20y ago
  I don't know. If that becomes a problem I'll put the write-ahead log on a faster disk to buy time to look into it.
  - mattculbreth 20y ago
    Why can't you just succumb to peer pressure and get rid of that database? :)
    - brlewis 20y ago
      I'm not sure. First I'm going to get rid of that operating system. It has lots of features that aren't relevant to my web app.
      - paul 20y ago
        It's somewhat amusing, but also mildly depressing, to see these remarks despite the fact that I explicitly stated (in bold, even), that "I'm not advising you to ditch your database!" And yet somehow that gets translated into "get rid of that database."
        mattculbreth 20y ago
        Don't be depressed. You started something that's spawned (as of now) 28 comments. Great thread.
        brlewis 20y ago
        Sorry it's mildly depressing. Unfortunately, any thoughts you bring up related to a polarizing issue are going to end up being framed in terms of the polarization. That's just the way it is; some things will never change. On the up side, you are going to learn a lot about essay writing, and about how concepts do/don't get through to people. These are very valuable skills.
  - paul 20y ago
    But wait, didn't you say it was a non-existent problem?
    - brlewis 20y ago
      Someday updates per second might be a problem. The nonexistent problem is "they seek the disk head to some specific location and then read or write my name to that location." I may have missed it, but I haven't seen you admit yet that you were wrong. You've been informed now that reads use a memory cache and writes use a write-ahead log, right?
      - paul 20y ago
        I'm probably at least partially wrong about everything.. It's true that PostgreSQL uses a write-ahead log, but it's still going to do that seek when it checkpoints, correct? I don't think that it will be able to sustain high transaction rates (100,000 tps for small transactions should be very doable). If you find that this isn't true and that someone has demonstrated these rates, let me know -- it would certainly be good news.
        brlewis 20y ago
        Yes, it will still do that seek when it checkpoints, but I seriously doubt it will do one seek pert transaction at that checkpoint. Still, I doubt 100,000 tps can be achieved with PostgreSQL. As for being wrong, that's one of the privileges you lose with celebrity. Everything you write in your blog is going to be preceded by "The guy who wrote gmail says..." When you say something that looks like it might become a popular misconception, guys like me will jump up and down, scream, and tear at our hair. Or maybe we'll just post terse, cold, unfriendly corrections. We still like you, though. My main issue with your post is the way it comes across as saying databases are overengineering for 99% of web sites, but mainly deals with a performance issue that won't matter for 99% of web sites. I would have no problem if you framed it as high tps vs low tps for certain classes of application, not as simplicity vs overengineering. Using a relational database is a fine default choice for web apps today, just as you expect to have an OS on your server by default.
  - paul 20y ago
  - dfranke 20y ago
    Fast disks are only faster by a factor of two.
    - brlewis 20y ago
      I think I could address any such performance issues within a factor of two growth of updates per second.
- rtm 20y ago
  167 small transactions per second on MySQL 4.1 / InnoDB / FreeBSD / FFS / SCSI, which looks like one per rotation. Presumably the time required to write a block at the end of the write-ahead log. 5000 per second with --innodb_flush_log_at_trx_commit=0, which only writes the log to disk once per second. The MySQL documentation claims that this configuration does crash-recovery correctly, though you may lose the last second's worth of transactions.
  - paul 20y ago
    BTW, in case anyone is thinking, "why would I need more than 167 transactions per second", imagine that you have a web site that displays everyone's "last active" time (for chat maybe). If users are performing some activity once every 30 sec, then you will be pretty much be at your limit with about 5000 simultaneous users. Robert, did you test MySQL with a parallel load? To give it the best chance at maximizing tps, it will need to be able to combine multiple pending transactions in a single write. Also, can you tell if it's updating it's b-trees? (if not, it may not be able to sustain these rates)
    - rtm 20y ago
      I ran the test with 1,000,000 INSERTs (taking 200 seconds), with the innodb cache size set to only 200,000 bytes. So there's a fair chance it updated the b-trees on disk.

mattjaynes 20y ago

Loved this point: "The thing is, a lot of those difficult problems are irrelevant for 99% of products. For example, "real" databases can handle transactions that are too large to fit in memory. That was probably a really important feature in 1980. Today, you can buy a computer with 32GB of memory for around $5000. How many GB transactions do you suppose Twitter performs? My guess is zero -- I suspect that their average transaction size is closer to 0.0000002 GB (messages are limited to 140 characters)."

Sam_Odio 20y ago

Want an example of "simple but less reliable"? Here's one (in one or two easy steps): 1. All updates go in to memcached, but not the database 2. (optional) A background process occasionally copies entries from memcached to the db. Without this, the values will be completely lost when memcached restarts. Paul - here's an interesting take on your recommended solution: Use solid state ram and forget about step two. You get fast a read/write, without the risk of data loss if the machine restarts. Here's a pretty cool board that will take DDR SDRAM, and supply it with a constant source of power (imitating solid-state memory): http://techreport.com/reviews/2006q1/gigabyte-iram/index.x?pg=1 http://techreport.com/reviews/2006q1/gigabyte-iram/index.x?pg=1 Those who insist on a database, could even store the database on this board. Best of both worlds. Here's an insane graph showing the database transaction rate for the board vs. a conventional hard drive: http://techreport.com/reviews/2006q1/gigabyte-iram/index.x?pg=7 http://techreport.com/reviews/2006q1/gigabyte-iram/index.x?pg=7

aristus 20y ago
Solid-state drives greater than 128GB are in OEM testing now. Expect server economics to turn upside-down in about 12 months, when persistent storage seek times get 100X faster.
- paul 20y ago
  Yeah, solid-state storage is clearly the future for most apps. Flash is over 10x cheaper than DRAM though, so it'd be nice to see some high-performance flash drives as well. It's also nice to not have to worry about losing data during a prolonged power outage.
  - aristus 20y ago
    No need to wait, really. Addonics' SATA/CF adapters for your $80/8GB flash cards cost $35. Even cheap compact flash does random seeks 40x faster than platters. Throughput is a little slow but what the hell, for 115 bucks? If I were building a server farm right now I'd try those things out.