3 ms·
I don't see how these results provide much useful information in terms of being able to say x is faster than y. The hardware doesn't seem consistent across dif
by WhitneyLand 10y ago
I don't see how these results provide much useful information in terms of being able to say x is faster than y.
The hardware doesn't seem consistent across different benchmarks. He says it's fast for a "cpu system", but for practical purposes Phi competes more with GPGPUs.
Would this be just as fast with one redis system with 512GB ram? I don't know too many apples to oranges here.
- lorenzhs 10y agoA single machine doesn't have the required memory bandwidth to do this in the same time.
- WhitneyLand 10y agoWhy not? Sounds like the initial data load and indexing are done up front. Once you get past that to run the benchmark it's not clear that a quad channel ddr4 system would be saturated.
- lorenzhs 10y agoA current-generation Xeon E7 has a memory bandwidth of up to 102 GB/s. The article says 90, which is probably a realistic estimate of achievable throughput. But the data is 500GB. So the problem is not with loading it from disk (that can be done beforehand), but getting it from CPU to RAM fast enough.
- throwawayish 10y agoInaccurate; the very point of columnar stores is that if I'm only interested in a column I only expend the memory bandwidth required for that very column and nothing else. Hence typical queries to a columnar store would never stream the whole 500 GB for processing.
- lorenzhs 10y agoRight, a columnar store would touch only a couple (1.1bn * three numerical columns) of those 500 GB. But the question was about Redis :) The author has an overview of the benchmarks with various systems at http://tech.marksblogg.com/benchmarks.html http://tech.marksblogg.com/benchmarks.html – but one can't just compare rows as many factors vary (esp. hardware).
- sologub 10y agoThe author isn't testing just the software but combinations of various software/hardware systems, including for example PostgreSQL on an i5 CPU, 16GB RAM and 850 SSD hardware: http://tech.marksblogg.com/billion-nyc-taxi-rides-postgresql.html http://tech.marksblogg.com/billion-nyc-taxi-rides-postgresql...
- WhitneyLand 10y agoGreat, and btw I think it's cool and I enjoyed reading it. But what useful conclusions can be drawn from it?
- sologub 10y agoAt least that Xeon Phi is a great piece of engineering! :)
- LeifCarrotson 10y agoAll conclusions are only valid for similar workloads, but each of MapD and GPUs, Q/kdb+ and Xeon Phi, Redshift, Athena, Big Query, Presto, and Elasticsearch claim to be fast, inexpensive, easy to work with, and otherwise great for Big Data. Which ones really are fast? How fast is fast? How much is this going to cost? Do I need 5 nodes or 50? A few examples of some useful conclusions: - Just because a relatively well-optimized PostgreSQL database on a regular workstation takes 5 minutes to run a query doesn't mean you can't get special hardware to run that query faster than you can type. - Spark + S3 + Amazon Elastic Map Reduce look like an ideal tool to work with large data, but they're pretty slow compared to better tools, and even compared to plain PostgreSQL. - HDFS really is a lot faster than S3. - Performance of an Xeon Phi 64-core CPU is within an order of magnitude to an NVidia Titan X. - Loading 104 GB of compressed data into Q/kdb+ expands to 125 GB with and takes about 30 minutes, but on Redshift expands to 2 TB and takes many hours to upload on a normal connection, plus 4 hours to actually import! - It might cost $5000 to custom-build a GPU-based supercomputer that can do these queries in under a second, but you can run similar queries if you're willing to wait for 5 minutes each by spinning up instances for a few dollars an hour plus a few more dollars an hour for storage, or by just running PostgreSQL on your workstation. Also, not a conclusion, but it's incredibly useful to have a simple example exactly how to configure the tool and import some CSV data