4 ms·
The architecture boils down to: - A framework for running a query across a lot of machines (a pretty much solved problem) - Throw a metric fuckton of hardw
by dsl 16y ago
The architecture boils down to:
- A framework for running a query across a lot of machines (a pretty much solved problem)
- Throw a metric fuckton of hardware at the problem
1b rows/sec isn't all that impressive when you find out they have 1.3 Tb of RAM and a $30,000 a month EC2 bill [1].
1. "Our 40-instance (m2.2xlarge) cluster can scan, filter, and aggregate 1 billion rows in 950 milliseconds."
- rbranson 16y agoFurther extrapolating, m2.2xlarge has 8 pretty powerful 2.66GHz Nehalem Xeon cores, so that's 320 cores, or roughly 3.2 million "rows" (whatever that is) per second per core. Nothing to laugh at, but certainly not a revelation.
- deleted 16y ago[deleted]
- deleted 16y ago[deleted]
- shazow 16y agoFor comparison, tineye.com (a reverse image search engine that I worked on a few years ago) queries well over 1 billion images (and thus rows of metadata) in under 0.5s (our goal was sub-250ms). I also know that the size of the cluster is a fraction of 40 instances (especially if you don't include the subcluster of web crawlers). By comparison, it doesn't seem super impressive. Tineye's metadata is stored in a sharded PostgreSQL database, the image index itself is a separate proprietary engine (also sharded). Then again, we probably dealt with a smaller set of query types so we could optimize those scenarios much more heavily. Nonetheless, from my experience it feels like they could do better with any of the options they've tried. It's important to be aware where the bottleneck is. In our case, the IO of the EC2 storage disks was a bottleneck (which we mitigated by sharding). Once we moved the cluster off of AWS and shoved some nice SSD drives into the machines, the performance increase was dramatic (half the shards for twice the speed, pretty much).
- stavros 16y agoThe difference is that you're scanning, they're aggregating. Scanning is O(logn), aggregating is O(n).
- barrkel 16y agoIt depends on the commutitvity and associativity of your aggregation operator. With sufficient parallelism, aggregation may also be logarithmic.
- StavrosK 16y agoAre you saying you can somehow sum numbers without reading all of them?
- jchrisa 16y agoCouchDB does this. It stores precomputed sub-aggregates in the inner-nodes of its B-Tree, so that at query time, it is only summing a handful of numbers instead of all of them.
- StavrosK 16y agoIt can do this because the queries are known in advance, though...
- barrkel 16y agoI'm saying that in terms of real time, if you have enough parallelism, you don't have to wait for n units, but rather log n units. As a function of CPU time, sure it's linear, but ignoring what parallelism gives you is just not being realistic, IMHO. As to the sibling comment, that's another good point. You say it's because it knows about the queries ahead of time, but you don't get excellent real-world performance without what an academic might consider cheating. Special-case your data structures to expected queries, where those queries are either empirically discovered and adjusted for manually, or learned automatically over time. A closed-minded approach that simply shuts down creative thinking by parroting out "linear time, no can do better, sorry" is poor form.
- n_are_q 16y agoAren't metrics such as "seconds on an ec2 instance" not particularly meaningful because you get highly variable performance per instance based on who else is using the actual hardware? Am I correct to assume that m2.2xlarge instances are shared like other instance types?
- benologist 16y agoThere was a great blog post the other week that explained the multi-tenancy stuff really well: http://perfcap.blogspot.com/2011/03/understanding-and-using-amazon-ebs.html http://perfcap.blogspot.com/2011/03/understanding-and-using-... Somewhere there's an HN discussion for it too.