3 ms·

>If you have a single table containing many terabytes of data, Hadoop might be a good option for running full table scans on it. The author mentions this at t

by jasode 11y ago

>If you have a single table containing many terabytes of data, Hadoop might be a good option for running full table scans on it. The author mentions this at the end almost as a footnote but in my experience, this is usually the motivating reason for IT to use Hadoop. It's about processing throughput and not just absolute data size. A lot of processing jobs can't use indexes (except for lookup tables) so a PostgreSQL 4TB db with a schema for OLTP workloads doesn't solve the issue. A 4tb harddrive might be cheap for $150 but at 100MB/sec, it takes 11+ hours to read. However, a big data cluster with 10 nodes can each scan their local 400GB shard and the batch job gets done in 1 hour. Even newer 10TB mechanical harddrives with 150MB/sec (and multi disks RAID to saturate a SATA3 interface) isn't going to fundamentally alter this throughput equation. On the other hand, a potential 4TB SSD with 1GB/sec might be the sweet spot if the processing is not cpu-constrained. But then again, new technology enables the goal posts to move[1], and people will invent new workloads that require a 100-node cluster each having a 10TB SSD to solve bigger problems. [1]http://en.wikipedia.org/wiki/Induced_demand http://en.wikipedia.org/wiki/Induced_demand

mark_l_watson 11y ago
Great point. I don't have applications needing Hadoop right now, but in cases where (for example) I had a ton of data in S3 and needed to make some passes through it then Elastic MapReduce was appropriate and inexpensive. Also, there are good machine learning libraries available for Hadoop and also Spark, and the overhead of using Hadoop and/or Spark might make sense to have head room for larger data sets without recoding.
bsg75 11y ago
> A lot of processing jobs can't use indexes (except for lookup tables) so a PostgreSQL 4TB db with a schema for OLTP workloads doesn't solve the issue. What are the cases where an RDBMS can't use indexes on fact tables?
- sokoloff 11y ago
  Searching within fields is one example. WHERE street_address LIKE "%MAIN%ST%" WHERE xml_blob LIKE "%<deprecrated_feature_node_name%" to find documents that need to be updated when you remove that feature, etc. "Just use Mongo" or "just use the latest pgsql and convert to JSON" or "just do XYZ" is often a less practical answer than "just query/analyze the database you already have".
  - usetrigram 11y ago
    If you're using PostgreSQL you can just use a trigram index for these kinds of searches. - http://www.postgresql.org/docs/current/interactive/pgtrgm.html#AEN150325 http://www.postgresql.org/docs/current/interactive/pgtrgm.ht... - https://swtch.com/~rsc/regexp/regexp4.html https://swtch.com/~rsc/regexp/regexp4.html
    - sokoloff 11y ago
      LOL at the new account name, but thank you very much for the pointer to an index type I didn't know existed!

by jasode 11y ago

mark_l_watson 11y ago

Great point. I don't have applications needing Hadoop right now, but in cases where (for example) I had a ton of data in S3 and needed to make some passes through it then Elastic MapReduce was appropriate and inexpensive. Also, there are good machine learning libraries available for Hadoop and also Spark, and the overhead of using Hadoop and/or Spark might make sense to have head room for larger data sets without recoding.

bsg75 11y ago

> A lot of processing jobs can't use indexes (except for lookup tables) so a PostgreSQL 4TB db with a schema for OLTP workloads doesn't solve the issue. What are the cases where an RDBMS can't use indexes on fact tables?

sokoloff 11y ago
Searching within fields is one example. WHERE street_address LIKE "%MAIN%ST%" WHERE xml_blob LIKE "%<deprecrated_feature_node_name%" to find documents that need to be updated when you remove that feature, etc. "Just use Mongo" or "just use the latest pgsql and convert to JSON" or "just do XYZ" is often a less practical answer than "just query/analyze the database you already have".
- usetrigram 11y ago
  If you're using PostgreSQL you can just use a trigram index for these kinds of searches. - http://www.postgresql.org/docs/current/interactive/pgtrgm.html#AEN150325 http://www.postgresql.org/docs/current/interactive/pgtrgm.ht... - https://swtch.com/~rsc/regexp/regexp4.html https://swtch.com/~rsc/regexp/regexp4.html
  - sokoloff 11y ago
    LOL at the new account name, but thank you very much for the pointer to an index type I didn't know existed!