5 ms·

This. If you can fit it on your soldered-in SSD, I'm not sure it really counts as "Big Data".

RBerenguel 10y ago
Well, O(100GB) is where I start thinking of "big data". Processing 100 GB is slow in a single "normal" machine (depending on what, sure). Between 10 and 100 I usually do it in a cluster since it is less hassle (even if I could process it locally with some tweaks or patience). Less than 10 is usually locally run unless I'm already computing something else
- matteuan 10y ago
  100GB can be handled without problems by mysql or postgre on a single node (yeah maybe not a laptop). You need to tune your database and take the right design decisions, but it's still less work than setting up a distributed system.
  - EdwardDiego 10y ago
    Depends on how you're setting it up. aws s3 cp --recursive bunch-o-data s3://some-bucket/ spark-ec2 --region eu-west-1 --identity-file s.pem --key-pair=spark --instance-type m3.2xlarge --slaves 40 -v 1.5.2 launch my-cluster Is significantly easier than making PG work easily at the 100GB scale in my experience. spark-ec2 is a script that ships with Spark to make it easy to set up a cluster in AWS.
    - nchammas 10y ago
      As of Spark 2.0 spark-ec2 doesn't ship with Spark anymore [0]. It's been moved here [1]. [0] http://spark.apache.org/releases/spark-release-2-0-0.html#removals http://spark.apache.org/releases/spark-release-2-0-0.html#re... [1] https://github.com/amplab/spark-ec2 https://github.com/amplab/spark-ec2
    - RBerenguel 10y ago
      I have a custom EMR launcher to get a cluster with Spark, Zeppelin and Hive. So yes, much faster for ad-hoc than tuning Postgres, basically it is a matter of if I need ad-hoc reporting/analysis or need it for normal operations. For 100 GB, regular, I'd just drop it into our running Redshift and let it eat it.
cle 10y ago
Not necessarily, there are other benefits to distributed processing than just parallel data processing itself. Resiliency, redundancy, much higher network throughput, etc. If you need low processing latency, as your disk size gets bigger, your bottleneck quickly becomes network throughput, not disk throughput. And if that one box fails...
- matteuan 10y ago
  True, but you can have more nodes with modern 'traditional' databases, replicate your data and add redundancy. Hadoop and other big data tools should be used because we have a huge amount of data, not to add redundancy or resiliency. The characteristic you listed were well know way before 'big data' became a thing.

Well, O(100GB) is where I start thinking of "big data". Processing 100 GB is slow in a single "normal" machine (depending on what, sure). Between 10 and 100 I usually do it in a cluster since it is less hassle (even if I could process it locally with some tweaks or patience). Less than 10 is usually locally run unless I'm already computing something else

matteuan 10y ago
100GB can be handled without problems by mysql or postgre on a single node (yeah maybe not a laptop). You need to tune your database and take the right design decisions, but it's still less work than setting up a distributed system.
- EdwardDiego 10y ago
  Depends on how you're setting it up. aws s3 cp --recursive bunch-o-data s3://some-bucket/ spark-ec2 --region eu-west-1 --identity-file s.pem --key-pair=spark --instance-type m3.2xlarge --slaves 40 -v 1.5.2 launch my-cluster Is significantly easier than making PG work easily at the 100GB scale in my experience. spark-ec2 is a script that ships with Spark to make it easy to set up a cluster in AWS.
  - nchammas 10y ago
    As of Spark 2.0 spark-ec2 doesn't ship with Spark anymore [0]. It's been moved here [1]. [0] http://spark.apache.org/releases/spark-release-2-0-0.html#removals http://spark.apache.org/releases/spark-release-2-0-0.html#re... [1] https://github.com/amplab/spark-ec2 https://github.com/amplab/spark-ec2
  - RBerenguel 10y ago
    I have a custom EMR launcher to get a cluster with Spark, Zeppelin and Hive. So yes, much faster for ad-hoc than tuning Postgres, basically it is a matter of if I need ad-hoc reporting/analysis or need it for normal operations. For 100 GB, regular, I'd just drop it into our running Redshift and let it eat it.

cle 10y ago

Not necessarily, there are other benefits to distributed processing than just parallel data processing itself. Resiliency, redundancy, much higher network throughput, etc. If you need low processing latency, as your disk size gets bigger, your bottleneck quickly becomes network throughput, not disk throughput. And if that one box fails...

matteuan 10y ago
True, but you can have more nodes with modern 'traditional' databases, replicate your data and add redundancy. Hadoop and other big data tools should be used because we have a huge amount of data, not to add redundancy or resiliency. The characteristic you listed were well know way before 'big data' became a thing.