4 ms·

When someone writes on their intro "... and now in return I want to waste your time a bit." it's a sign that someone isn't being serious. Domas' main criticism

When someone writes on their intro "... and now in return I want to waste your time a bit." it's a sign that someone isn't being serious. Domas' main criticism is that our video, http://vimeo.com/44087431 http://vimeo.com/44087431, uses a MySQL with standard defaults. Since tuning a database is by definition a custom process, we wanted to demonstrate what performance you'd get "out of the box." The video speaks for itself. MemSQL can push 80k/sec vs 3.5k/sec for MySQL on EC2. Domas actually demonstrates MemSQL's robust durability controls: he bounds MemSQL to disk. If you want to throw an anchor behind a speed boat, you can do it, though we don't recommend it. Even Domas agrees his article isn't serious: "I do agree, that its execution speeds for table scans are not bad – 8M rows scanned a second (for SELECT COUNT(*) query) from single thread is sure a great achievement." If that's the highest praise I can get from Domas, I can live with it.

spudlyo 14y ago
For me, the take away from this article is that MemSQL's ACID compliance is half-assed at best, which is a much more damning criticism than using a flawed benchmarking methodology.
peeters 14y ago
> Since tuning a database is by definition a custom process, we wanted to demonstrate what performance you'd get "out of the box." Yes, and Domas was criticizing that as a strawman; as comparing apples to oranges. Since they tailor their defaults to different criteria (e.g. different standards of durability), of course they will perform differently. The only really valid benchmarks are when you get an expert in each to tune their respective products to optimize for a specific scenario, and see how outperforms who. And then make public exactly what configuration was done to get there.
mikeash 14y ago
I thought his main criticism was that MemSQL isn't durable by default, and that when you enable that it becomes incredibly slow. That's a pretty big deal, and IMO running database benchmarks on an unsafe configuration like this is extremely dishonest.
- ericfrenkiel 14y ago
  That's a bit heavy-handed; additional benchmarks will be published against a variety of configurations and other databases. Companies use MemSQL because it uses memory as the primary locus of data. If you have a fast data problem, you couldn't use a disk-based system. At high speeds, you'd want to deploy any database in an active-active mode.
  - mikeash 14y ago
    How is it heavy handed? As I understand the article, MemSQL by default will often say that a transaction has been committed even though it only exists in memory, not on disk. Comparing that scenario with a MySQL setup which only says a transaction has been committed once it's on non-volatile storage is just wrong.
  - deleted 14y ago
    [deleted]
  - shin_lao 14y ago
    Then you cannot claim you're ACID, since the D stands for durable. Something is wrong with your marketing.
  - merlincorey 14y ago
    I just want to echo mike above with a slightly different tact that perhaps will get across my perception of our feelings. It's dishonest, at the very least, to claim ACID compliance and then do benchmarks against a known ACID compliant competitor without having the same level of ACIDity enabled in your program. If you want to prove how fast MemSQL is against MongoDB, that's one thing (I understand it also can be "fully durable" but it is not by default due to speed concerns). To prove how fast MemSQL is against MySQL, you have to prove it on the same playing field, which means it has to, necessarily, be just as ACIDic and respond to "real" queries on large datasets with high concurrency. tl;dr Do some benchmarks of apples-to-apples ACID, and do something about the common case of a head of a table, then you will be honestly faster (or not) than MySQL
    - 14y ago
caladri 14y ago
How do you feel about the reported performance of "SELECT * FROM table ORDER BY id DESC LIMIT 5;"? Is this something that needs to be improved there, or are you happy with that being one of the built-in anchors? Additionally, I'm not sure that the implementation of the hard reliable case sounds very mature or even well thought-out. Having a syncer running at 20hz and waiting for it even in the synchronous path seems a bad idea. That's kind of like adding an anchor to your anchor so your speedboat can crawl while it crawls. Or was that a misrepresentation? And who cares if the tone is serious? The criticisms seem to be valid, and ones that anyone releasing software making bold claims should expect. Statements about performance need to be qualified. Out-of-the-box behavior is not a reasonable point of comparison if (1) that's not what people do in the real world (2) the tests you use are ones that one piece of software is tuned for and the other is tuned against. Some kind of sorted index by field is necessary for some very common and desirable queries. Some of the performance gains come at an unacknowledged cost. The durability just isn't there (and for trivial reasons that could be fixed by making a few more parameters tunable, it sounds like), but is implicitly lauded by claiming to be ACID. I don't know if you need to go looking for praise in that article. It really seems like there are some criticisms it would be beneficial to take seriously, even if you object to the author's tone or lack of praise. Even-handedness is not a marker of honesty, and honest criticism is worth the headaches it causes and the growth it spurs if you're willing to take it seriously.
- tintor 14y ago
  "How do you feel about the reported performance of "SELECT * FROM table ORDER BY id DESC LIMIT 5;"?" Index was missing on ID column in MemSQL's case, so MemSQL was running entire table scan instead of using the index as MySQL.
cbsmith 14y ago
On this basis though, why develop a new database system when instead you could just develop a new default configuration for MySQL?
deleted 14y ago
[deleted]
rmc 14y ago
So you've basically critizised their tone, and repeated your original claims, without really addressing what they said.
jhugg 14y ago
"If you want to throw an anchor behind a speed boat, you can do it, though we don't recommend it." There's a weird implication here that once you need disk persistence, you might as well give up. Group commit and well tuned, append-focused IO can do wonders if leveraged correctly. For example, we've benchmarked VoltDB using synchronous disk writes (group commit every 2ms) at half a million SQL statements/sec and 100k txns/sec on a single box using a magnetic media and disk controller with a BBU. This same config on EC2/EBS will run dramatically slower, but would still be considered respectable compared to MySQL.
steve8918 14y ago
Personally, I think out-of-the-box comparisons are fairly useless, because anyone who doesn't tune their database probably doesn't have enough data to make a difference. I have no opinions either way on your product, but have you tried benchmarking against an out-of-the-box MySQL installed entirely on a ramdisk, so that it doesn't touch disk at all as well? That would make the comparisons more valid, since both installations wouldn't touch a physical disk.
timaelliott 14y ago
> If you want to throw an anchor behind a speed boat, you can do it, though we don't recommend it. So, what your video is saying, is that your speedboat is 80,000 faster than MySQL with an anchor attached. Seems like a fair comparison.
- slurgfest 14y ago
  I have no interest in MemSQL. I think it is reasonably fair to test MySQL's default, recommended configuration which writes to disk frequently, particularly if the other configuration is not worth using. If it is worth using, then the real question is what you give up by using MemSQL instead of MySQL in the alternate configuration. If you mean to ding MemSQL on durability then you can ding MySQL on speed. If you aren't going to ding MemSQL on durability (because you are comparing it to less disk-intensive MySQL configs) then presenting a case for MySQL means you have to come up with something that you are giving up other than durability based on frequent disk writes.
mikeryan 14y ago
When someone writes on their intro "... and now in return I want to waste your time a bit." it's a sign that someone isn't being serious. Perhaps the vimeo video needs the same disclaimer. Since tuning a database is by definition a custom process, we wanted to demonstrate what performance you'd get "out of the box." Since you're picking the tests and setting what the "out of the box" defaults are the whole thing is kind of a waste of time, which is really what Domas is pointing out.
midom 14y ago
No, that is not the highest praise you can get from me. It definitely is very interesting technology, and what it does well, it does really well - though I didn't get to benchmark parallel performance, as I had work to do :-) I wouldn't have written that post the way it is if your marketing would be honest and with lots of actual technical merit. There're plenty other companies in the community that have invested much more into decent technical communication. I agree that in some cases pure durability isn't needed and the best-effort with enforced maximum slippage is good enough, but still, pure durability implementation is extremely naive, and I can show that it hits benchmarks really badly :-) Your optimizer does not optimize very simple things, and though you can claim that in-memory performance hides that, still, it is nowhere close a drop-in replacement of any kind. I understand what you've done from technology perspective, and it is an interesting approach (albeit I'm not trying to evaluate applicability). I just cannot approve the way you did initial publicity - unfair comparisons are unfair comparisons, and if you're using them that way, I can come up with my unfair comparison :)
willvarfar 14y ago
(Ooooh damage control; aren't we your potential customers?)
Devilboy 14y ago
You're claiming 'full durability' and comparing it to InnoDB's durability which is complete bullshit as the article explains.

spudlyo 14y ago

For me, the take away from this article is that MemSQL's ACID compliance is half-assed at best, which is a much more damning criticism than using a flawed benchmarking methodology.

peeters 14y ago

> Since tuning a database is by definition a custom process, we wanted to demonstrate what performance you'd get "out of the box." Yes, and Domas was criticizing that as a strawman; as comparing apples to oranges. Since they tailor their defaults to different criteria (e.g. different standards of durability), of course they will perform differently. The only really valid benchmarks are when you get an expert in each to tune their respective products to optimize for a specific scenario, and see how outperforms who. And then make public exactly what configuration was done to get there.

mikeash 14y ago

I thought his main criticism was that MemSQL isn't durable by default, and that when you enable that it becomes incredibly slow. That's a pretty big deal, and IMO running database benchmarks on an unsafe configuration like this is extremely dishonest.

ericfrenkiel 14y ago
That's a bit heavy-handed; additional benchmarks will be published against a variety of configurations and other databases. Companies use MemSQL because it uses memory as the primary locus of data. If you have a fast data problem, you couldn't use a disk-based system. At high speeds, you'd want to deploy any database in an active-active mode.
- mikeash 14y ago
  How is it heavy handed? As I understand the article, MemSQL by default will often say that a transaction has been committed even though it only exists in memory, not on disk. Comparing that scenario with a MySQL setup which only says a transaction has been committed once it's on non-volatile storage is just wrong.
- deleted 14y ago
  [deleted]
- shin_lao 14y ago
  Then you cannot claim you're ACID, since the D stands for durable. Something is wrong with your marketing.
- merlincorey 14y ago
  I just want to echo mike above with a slightly different tact that perhaps will get across my perception of our feelings. It's dishonest, at the very least, to claim ACID compliance and then do benchmarks against a known ACID compliant competitor without having the same level of ACIDity enabled in your program. If you want to prove how fast MemSQL is against MongoDB, that's one thing (I understand it also can be "fully durable" but it is not by default due to speed concerns). To prove how fast MemSQL is against MySQL, you have to prove it on the same playing field, which means it has to, necessarily, be just as ACIDic and respond to "real" queries on large datasets with high concurrency. tl;dr Do some benchmarks of apples-to-apples ACID, and do something about the common case of a head of a table, then you will be honestly faster (or not) than MySQL
  - 14y ago

caladri 14y ago

How do you feel about the reported performance of "SELECT * FROM table ORDER BY id DESC LIMIT 5;"? Is this something that needs to be improved there, or are you happy with that being one of the built-in anchors? Additionally, I'm not sure that the implementation of the hard reliable case sounds very mature or even well thought-out. Having a syncer running at 20hz and waiting for it even in the synchronous path seems a bad idea. That's kind of like adding an anchor to your anchor so your speedboat can crawl while it crawls. Or was that a misrepresentation? And who cares if the tone is serious? The criticisms seem to be valid, and ones that anyone releasing software making bold claims should expect. Statements about performance need to be qualified. Out-of-the-box behavior is not a reasonable point of comparison if (1) that's not what people do in the real world (2) the tests you use are ones that one piece of software is tuned for and the other is tuned against. Some kind of sorted index by field is necessary for some very common and desirable queries. Some of the performance gains come at an unacknowledged cost. The durability just isn't there (and for trivial reasons that could be fixed by making a few more parameters tunable, it sounds like), but is implicitly lauded by claiming to be ACID. I don't know if you need to go looking for praise in that article. It really seems like there are some criticisms it would be beneficial to take seriously, even if you object to the author's tone or lack of praise. Even-handedness is not a marker of honesty, and honest criticism is worth the headaches it causes and the growth it spurs if you're willing to take it seriously.

tintor 14y ago
"How do you feel about the reported performance of "SELECT * FROM table ORDER BY id DESC LIMIT 5;"?" Index was missing on ID column in MemSQL's case, so MemSQL was running entire table scan instead of using the index as MySQL.

cbsmith 14y ago

On this basis though, why develop a new database system when instead you could just develop a new default configuration for MySQL?

deleted 14y ago

[deleted]

rmc 14y ago

So you've basically critizised their tone, and repeated your original claims, without really addressing what they said.

jhugg 14y ago

"If you want to throw an anchor behind a speed boat, you can do it, though we don't recommend it." There's a weird implication here that once you need disk persistence, you might as well give up. Group commit and well tuned, append-focused IO can do wonders if leveraged correctly. For example, we've benchmarked VoltDB using synchronous disk writes (group commit every 2ms) at half a million SQL statements/sec and 100k txns/sec on a single box using a magnetic media and disk controller with a BBU. This same config on EC2/EBS will run dramatically slower, but would still be considered respectable compared to MySQL.

steve8918 14y ago

Personally, I think out-of-the-box comparisons are fairly useless, because anyone who doesn't tune their database probably doesn't have enough data to make a difference. I have no opinions either way on your product, but have you tried benchmarking against an out-of-the-box MySQL installed entirely on a ramdisk, so that it doesn't touch disk at all as well? That would make the comparisons more valid, since both installations wouldn't touch a physical disk.

timaelliott 14y ago

> If you want to throw an anchor behind a speed boat, you can do it, though we don't recommend it. So, what your video is saying, is that your speedboat is 80,000 faster than MySQL with an anchor attached. Seems like a fair comparison.

slurgfest 14y ago
I have no interest in MemSQL. I think it is reasonably fair to test MySQL's default, recommended configuration which writes to disk frequently, particularly if the other configuration is not worth using. If it is worth using, then the real question is what you give up by using MemSQL instead of MySQL in the alternate configuration. If you mean to ding MemSQL on durability then you can ding MySQL on speed. If you aren't going to ding MemSQL on durability (because you are comparing it to less disk-intensive MySQL configs) then presenting a case for MySQL means you have to come up with something that you are giving up other than durability based on frequent disk writes.

mikeryan 14y ago

When someone writes on their intro "... and now in return I want to waste your time a bit." it's a sign that someone isn't being serious. Perhaps the vimeo video needs the same disclaimer. Since tuning a database is by definition a custom process, we wanted to demonstrate what performance you'd get "out of the box." Since you're picking the tests and setting what the "out of the box" defaults are the whole thing is kind of a waste of time, which is really what Domas is pointing out.

midom 14y ago

No, that is not the highest praise you can get from me. It definitely is very interesting technology, and what it does well, it does really well - though I didn't get to benchmark parallel performance, as I had work to do :-) I wouldn't have written that post the way it is if your marketing would be honest and with lots of actual technical merit. There're plenty other companies in the community that have invested much more into decent technical communication. I agree that in some cases pure durability isn't needed and the best-effort with enforced maximum slippage is good enough, but still, pure durability implementation is extremely naive, and I can show that it hits benchmarks really badly :-) Your optimizer does not optimize very simple things, and though you can claim that in-memory performance hides that, still, it is nowhere close a drop-in replacement of any kind. I understand what you've done from technology perspective, and it is an interesting approach (albeit I'm not trying to evaluate applicability). I just cannot approve the way you did initial publicity - unfair comparisons are unfair comparisons, and if you're using them that way, I can come up with my unfair comparison :)

willvarfar 14y ago

(Ooooh damage control; aren't we your potential customers?)

Devilboy 14y ago

You're claiming 'full durability' and comparing it to InnoDB's durability which is complete bullshit as the article explains.