3 ms·

Yeah but can you store same amount of data on one ssd as on 10 disks?

by supergirl 10y ago

snovv_crash 10y ago
This. If you can fit it on your soldered-in SSD, I'm not sure it really counts as "Big Data".
- RBerenguel 10y ago
  Well, O(100GB) is where I start thinking of "big data". Processing 100 GB is slow in a single "normal" machine (depending on what, sure). Between 10 and 100 I usually do it in a cluster since it is less hassle (even if I could process it locally with some tweaks or patience). Less than 10 is usually locally run unless I'm already computing something else
  - matteuan 10y ago
    100GB can be handled without problems by mysql or postgre on a single node (yeah maybe not a laptop). You need to tune your database and take the right design decisions, but it's still less work than setting up a distributed system.
    - EdwardDiego 10y ago
      Depends on how you're setting it up. aws s3 cp --recursive bunch-o-data s3://some-bucket/ spark-ec2 --region eu-west-1 --identity-file s.pem --key-pair=spark --instance-type m3.2xlarge --slaves 40 -v 1.5.2 launch my-cluster Is significantly easier than making PG work easily at the 100GB scale in my experience. spark-ec2 is a script that ships with Spark to make it easy to set up a cluster in AWS.
      - nchammas 10y ago
        As of Spark 2.0 spark-ec2 doesn't ship with Spark anymore [0]. It's been moved here [1]. [0] http://spark.apache.org/releases/spark-release-2-0-0.html#removals http://spark.apache.org/releases/spark-release-2-0-0.html#re... [1] https://github.com/amplab/spark-ec2 https://github.com/amplab/spark-ec2
      - RBerenguel 10y ago
        I have a custom EMR launcher to get a cluster with Spark, Zeppelin and Hive. So yes, much faster for ad-hoc than tuning Postgres, basically it is a matter of if I need ad-hoc reporting/analysis or need it for normal operations. For 100 GB, regular, I'd just drop it into our running Redshift and let it eat it.
- cle 10y ago
  Not necessarily, there are other benefits to distributed processing than just parallel data processing itself. Resiliency, redundancy, much higher network throughput, etc. If you need low processing latency, as your disk size gets bigger, your bottleneck quickly becomes network throughput, not disk throughput. And if that one box fails...
  - matteuan 10y ago
    True, but you can have more nodes with modern 'traditional' databases, replicate your data and add redundancy. Hadoop and other big data tools should be used because we have a huge amount of data, not to add redundancy or resiliency. The characteristic you listed were well know way before 'big data' became a thing.
Kirth 10y ago
No, but you also won't be using all that data at the same time. On Hadoop (and Spark on HDFS) clusters you'll find that most disks are either not that big or are heavily under-used. Our HBase cluster had 6 * 2Tb disks: about 8.5Tb of usable storage (the other 3.5Tb accounts for data replication/duplication) per host in the cluster. However, you need about 200 bytes in memory per kb on disk and should assign only 32Gb of heap to HBase. That's 2.5Tb wasted, per host. Couldn't just plug those disks out and use them somewhere else: you need all the disks in parallel to overcome the IO/bandwith bottleneck.

Yeah but can you store same amount of data on one ssd as on 10 disks?

snovv_crash 10y ago

This. If you can fit it on your soldered-in SSD, I'm not sure it really counts as "Big Data".

RBerenguel 10y ago
Well, O(100GB) is where I start thinking of "big data". Processing 100 GB is slow in a single "normal" machine (depending on what, sure). Between 10 and 100 I usually do it in a cluster since it is less hassle (even if I could process it locally with some tweaks or patience). Less than 10 is usually locally run unless I'm already computing something else
- matteuan 10y ago
  100GB can be handled without problems by mysql or postgre on a single node (yeah maybe not a laptop). You need to tune your database and take the right design decisions, but it's still less work than setting up a distributed system.
  - EdwardDiego 10y ago
    Depends on how you're setting it up. aws s3 cp --recursive bunch-o-data s3://some-bucket/ spark-ec2 --region eu-west-1 --identity-file s.pem --key-pair=spark --instance-type m3.2xlarge --slaves 40 -v 1.5.2 launch my-cluster Is significantly easier than making PG work easily at the 100GB scale in my experience. spark-ec2 is a script that ships with Spark to make it easy to set up a cluster in AWS.
    - nchammas 10y ago
      As of Spark 2.0 spark-ec2 doesn't ship with Spark anymore [0]. It's been moved here [1]. [0] http://spark.apache.org/releases/spark-release-2-0-0.html#removals http://spark.apache.org/releases/spark-release-2-0-0.html#re... [1] https://github.com/amplab/spark-ec2 https://github.com/amplab/spark-ec2
    - RBerenguel 10y ago
      I have a custom EMR launcher to get a cluster with Spark, Zeppelin and Hive. So yes, much faster for ad-hoc than tuning Postgres, basically it is a matter of if I need ad-hoc reporting/analysis or need it for normal operations. For 100 GB, regular, I'd just drop it into our running Redshift and let it eat it.
cle 10y ago
Not necessarily, there are other benefits to distributed processing than just parallel data processing itself. Resiliency, redundancy, much higher network throughput, etc. If you need low processing latency, as your disk size gets bigger, your bottleneck quickly becomes network throughput, not disk throughput. And if that one box fails...
- matteuan 10y ago
  True, but you can have more nodes with modern 'traditional' databases, replicate your data and add redundancy. Hadoop and other big data tools should be used because we have a huge amount of data, not to add redundancy or resiliency. The characteristic you listed were well know way before 'big data' became a thing.

Kirth 10y ago

No, but you also won't be using all that data at the same time. On Hadoop (and Spark on HDFS) clusters you'll find that most disks are either not that big or are heavily under-used. Our HBase cluster had 6 * 2Tb disks: about 8.5Tb of usable storage (the other 3.5Tb accounts for data replication/duplication) per host in the cluster. However, you need about 200 bytes in memory per kb on disk and should assign only 32Gb of heap to HBase. That's 2.5Tb wasted, per host. Couldn't just plug those disks out and use them somewhere else: you need all the disks in parallel to overcome the IO/bandwith bottleneck.