6 ms·

100ms for this query is good compared to PostgreSQL (~4s when running the query as-is, ~275ms when using a window function) considering that MapD doesn't seem t

100ms for this query is good compared to PostgreSQL (~4s when running the query as-is, ~275ms when using a window function) considering that MapD doesn't seem to utilize an index for it. That being said a 146MB CSV with 5 million rows is not a challenge for vanilla PostgreSQL either. If you're willing to add an index and use window functions (which MapD doesn't seem to support), you can run this query in ~34ms: https://gist.github.com/felixge/fffef4bc41a5e87e26f62c2a5c8ed5bd https://gist.github.com/felixge/fffef4bc41a5e87e26f62c2a5c8e... I have no experience with MapD, but people often significantly underestimate the performance of relational databases and reach for the big/expensive toys too soon.

randyzwitch 9y ago
You are right, as a demonstration there is a trade-off between using a real dataset and making the data large enough to demonstrate the performance gains. You are proving the other part of my argument though, in that you don't need to add an index (though, the comparison between using a window function in Postgres is fair, since that would be the canonical way to write that query) As far as the "big/expensive toys", I'm using the Community Edition, which is FOSS.
- felixge 9y ago
  I guess you're just trying to show how to use MapD rather than showing when it makes sense to use it, which is cool. For the latter, I found the results over here fairly interesting: http://tech.marksblogg.com/benchmarks.html http://tech.marksblogg.com/benchmarks.html As far as "big/expensive toys" go, I was referring to total cost of ownership of putting something like MapD in production vs PostgreSQL. The latter can run on any machine (no GPU needed), there are plenty of hosted services, and there is a large pool of people familiar with it. If this is just about running some analysis on a local machine, that argument doesn't apply so much :).
  - tmostak 9y ago
    Note that MapD can run on CPU as well and is generally quite fast. I haven’t benchmarked but on a dataset of this size the numbers might be similar. I’d also say that while this is a great example of getting up and running with MapD the system really shines when doing interactive queries over datasets from the 100M to 100B record range without requiring indexes.

randyzwitch 9y ago

You are right, as a demonstration there is a trade-off between using a real dataset and making the data large enough to demonstrate the performance gains. You are proving the other part of my argument though, in that you don't need to add an index (though, the comparison between using a window function in Postgres is fair, since that would be the canonical way to write that query) As far as the "big/expensive toys", I'm using the Community Edition, which is FOSS.

felixge 9y ago
I guess you're just trying to show how to use MapD rather than showing when it makes sense to use it, which is cool. For the latter, I found the results over here fairly interesting: http://tech.marksblogg.com/benchmarks.html http://tech.marksblogg.com/benchmarks.html As far as "big/expensive toys" go, I was referring to total cost of ownership of putting something like MapD in production vs PostgreSQL. The latter can run on any machine (no GPU needed), there are plenty of hosted services, and there is a large pool of people familiar with it. If this is just about running some analysis on a local machine, that argument doesn't apply so much :).
- tmostak 9y ago
  Note that MapD can run on CPU as well and is generally quite fast. I haven’t benchmarked but on a dataset of this size the numbers might be similar. I’d also say that while this is a great example of getting up and running with MapD the system really shines when doing interactive queries over datasets from the 100M to 100B record range without requiring indexes.