3 ms·
This benchmark is probably accurate, but not necessarily fair. Here's why: If all the systems have to do a table scan (complete scan of the data), then you're
by justinsb 15y ago
This benchmark is probably accurate, but not necessarily fair. Here's why:
If all the systems have to do a table scan (complete scan of the data), then you're (almost always) I/O bound. Because we're relying on sequential I/O, a good hard disk is as good as a consumer-grade SSD. Postgresql doesn't do compression (if rows fit on a single page), and MySQL doesn't by default either (though you can enable InnoDB page compression), so if you're using compression with r17 you can get more rows per MB transferred, so you win.
However, this only works if you're doing table scans on highly compressible data. The minute you compete against an index, I suspect you're going to lose heavily (e.g. selecting the number of hits by second over a one minute interval should be very fast on the relational databases, as they should be able to answer from the index)
MySQL and PostgreSQL are really OLTP databases, but you're running an OLAP workload here (multi-minute queries and table scans are essentially the definition of OLAP). As you say, a comparison against Hadoop is probably more relevant (as long as you compress the data). For this workload, an OLAP database is the tool to beat, and Hadoop is probably the leading contender in the open-source OLAP space.
Though I'd normally bet on PostgreSQL, I think the easiest fix is to turn on Innodb compression in MySQL: you should see query times that are much more comparable. You'd probably still win (because MySQL is doing a lot more), but the margin should be much smaller.
I think it's also possible to denormalize your tables and improve your indexing, so that we can avoid table scans and then PostgreSQL/MySQL should be much faster, but I don't think this is really the point of your benchmark.
Edit: Just spotted a (much bigger) problem - if your data is 54GB raw, when compressed that should be within the memory size of your machine (12GB), but if MySQL and PostgreSQL aren't compressing their data won't fit into memory (particularly with all those unused indexes). So in fact you're comparing RAM speeds to SSD speeds, which really isn't apples to apples.
- matthewnourse 15y agoThanks for taking the time to look over the tests in such detail! MySQL's EXPLAIN output says that it's using the indexes. PostgreSQL's EXPLAIN output says that it's _not_ (and from what I can tell so far, this is a deliberate design decision by PostgreSQL developers). r17 "loses" to MySQL on one query and "wins" on the other. So I agree that r17 has a much harder job competing against indexed data...but with r17 you didn't have to wait to create the index....not such a big deal for OLTP, a much bigger deal with OLAP. I would like to redo the bakeoff(s) with compressed InnoDB tables but as they can take several days to run and life is short, first I should focus on Hadoop as we both agree that is more relevant. Re the data size issue: the 54GB raw data set compresses to 28GB. The data generator I use creates data that's more random than the "real world" and so doesn't compress very well...makes life harder for r17, which is what I want. The smaller data set for the SSD test compresses to about 13GB, which isn't ideal I agree...I should have bought a larger SSD, I didn't think that MySQL would make so many large temporary files :). To mitigate this issue I ensured that the data set was _not_ cached before I ran the r17 script. I am very keen to find out the truth about r17's usefulness, thanks for your part in that. For the next bakeoff I'll provide more details about methods and machine behavior.