10 ms·

Apache Spark: A Unified Engine for Big Data Processing

162 points by akashtndn 10y ago

mpweiher 10y ago
Funky: 10 to 20 local disks, for approximately 1GB/s to 2GB/s of disk bandwidth The new MacBook Pros apparently have north of 2GB/s of disk bandwidth for their internal SSD. https://9to5mac.com/2016/11/01/the-late-2016-entry-level-13-macbook-pro-has-a-ridiculously-fast-ssd/ https://9to5mac.com/2016/11/01/the-late-2016-entry-level-13-...
- vidarh 10y ago
  You can pick up a wide variety of PCIe SSDs with that kind of bandwidth as well. It's worth keeping in mind, because it means that a whole lot of workloads that people think they need a cluster for end up being far faster and cheaper to put on a single server these days, given the cost and overheads of sufficiently high speed network interconnects to move the data around at a fast enough to compete. Of course there are plenty of workloads where you still need a cluster, but people often don't realise just how much you can put into a single commodity server today at a reasonable cost, nor what kind of overheads going from a single server to a cluster tends to add.
  - mistermann 10y ago
    If you could fit on one box, would you be better off using a sql database (assuming you knew you didn't need to scale beyond one box), or is a single node spark box faster? Or does it depend on the scenario?
    - mitchell_h 10y ago
      Spark and a SQL database have different use cases. Spark is a n analytics engine to do processing on data. SQL databases are for storage and retrieval of data(they just so happen to be able to do various analytical type stuff).
    - bsg75 10y ago
      Depends on where your bottleneck is. If its disk, maybe a one box approach is suitable. If CPU, where many current (F)OSS RDBMS are limited to one-core-per-query, a cluster can help. Additionally a cluster can manage a larger working data set in RAM than a single box can do.
      - vidarh 10y ago
        > If CPU, where many current (F)OSS RDBMS are limited to one-core-per-query, a cluster can help. If you're going to partition the data because of limitations like that, it may still be faster to run a "cluster" on a single server, depending on data volume. > Additionally a cluster can manage a larger working data set in RAM than a single box can do. That's true in theory. In practice I've never seen anyone max out the amount of RAM possible in off the shelf x86 servers. I'm sure it happens, but it's not common, and it's fairly unusual for people to even approach the point where adding ore RAM starts getting more expensive than adding more boxes. You can currently fit at least 8TB of RAM in the biggest x86 boxes; possibly more by now, and boxes that can fit 1TB-2TB are still relatively cheap. Unless you plan to exceed at least 1TB, it's likely going to be cheaper to stuff more memory in a box than add enough extra boxes to compensate for the communications overhead. Of course there are plenty of datasets where you would exceed 1TB for your dataset, but most people don't get anywhere close to that. I would say that before a hardware cluster (as opposed to dirty "hacks" like sharding on a single box) makes sense if you are buying new hardware (the maths of course looks different if one option is to make do with existing servers), you will need to have requirements that exceeds one or more of: - 48-64 x86 cores. - 1TB-2TB of RAM. - 2GB/sec aggregate disk bandwidth. For all of these, it's worth noting that you can't compare like for like, as the moment you go to a cluster you have the according overheads in both CPU, RAM and disk IO of having to spread the work out, write data to disk more places, duplicate data in memory etc.. Note also that you can do better than that on plain x86 hardware, but the above is the point where the incremental cost of increases starts to really hurt and eat into any savings vs. a cluster.
        bsg75 10y ago
        > That's true in theory. In practice I've never seen anyone max out the amount of RAM possible in off the shelf x86 servers. I'm currently experimenting with a 4 node cluster running Apache Drill (on top of MapR's Hadoop distro), with 500GB RAM, 96 cores, and 28TB of SSD. For some analytic queries that span multiple date partitions, its actually easy to see 70% RAM capacity being utilized, and 80% of the CPU. The size of the data set is currently about 7TB, but that replicates for both data locality and redundancy. We are a Postgres shop, and have tested with some of the options that run a "single node cluster" but one of the limiting factors is that RAM is divided between those DB processes, and performance is impacted. Plus we then need to implement some form of replication to obtain redundancy, fault tolerance, or HA. Quite simply we are not using a cluster because our data won't fit, but because we can get better performance and reliability on 4 nodes. We likely won't have the need for 400. There are cases where for non-interactive data processing we definitely don't need a cluster. At its simplest, multiple copies of Awk scripts can be run using GNU Parallel, Python scripts can be parallelized with the Multiprocessing module, and Go has been useful for concurrency (especially to us Python programmers).
- supergirl 10y ago
  Yeah but can you store same amount of data on one ssd as on 10 disks?
  - snovv_crash 10y ago
    This. If you can fit it on your soldered-in SSD, I'm not sure it really counts as "Big Data".
    - RBerenguel 10y ago
      Well, O(100GB) is where I start thinking of "big data". Processing 100 GB is slow in a single "normal" machine (depending on what, sure). Between 10 and 100 I usually do it in a cluster since it is less hassle (even if I could process it locally with some tweaks or patience). Less than 10 is usually locally run unless I'm already computing something else
      - matteuan 10y ago
        100GB can be handled without problems by mysql or postgre on a single node (yeah maybe not a laptop). You need to tune your database and take the right design decisions, but it's still less work than setting up a distributed system.
        EdwardDiego 10y ago
        Depends on how you're setting it up. aws s3 cp --recursive bunch-o-data s3://some-bucket/ spark-ec2 --region eu-west-1 --identity-file s.pem --key-pair=spark --instance-type m3.2xlarge --slaves 40 -v 1.5.2 launch my-cluster Is significantly easier than making PG work easily at the 100GB scale in my experience. spark-ec2 is a script that ships with Spark to make it easy to set up a cluster in AWS.
        nchammas 10y ago
        As of Spark 2.0 spark-ec2 doesn't ship with Spark anymore [0]. It's been moved here [1]. [0] http://spark.apache.org/releases/spark-release-2-0-0.html#removals http://spark.apache.org/releases/spark-release-2-0-0.html#re... [1] https://github.com/amplab/spark-ec2 https://github.com/amplab/spark-ec2
techno_modus 10y ago
> The RDD programming model provides only distributed collections of object It is ok for a non-technical high level overview but RDD should probably be referred to as a data model (how data is represented and managed) while map-reduce is a programming model (how data is being processed).
- kod 10y ago
  No, the rdd interface has methods on it that directly relate to how data is processed.
  - techno_modus 10y ago
    You are right, but what I meant is that delta between Spark and Hadoop is RDD (RDD - Resilient Distributed Dataset) which is data structure first of all. Thus Spark and Hadoop have the same data processing model implemented over different data representation models (and hence the performance gain).
    - marmaduke 10y ago
      > delta why can't we use the word difference? delta, mathematically, suggest some space and measure.
mydpy 10y ago
If you follow the Spark community closely, you won't find any new information here. However, as a software consultant, I find this to be a useful nontechnical overview to send to partners and clients.
sandGorgon 10y ago
The new programming model of spark is around Graphframes...which are a Pandas/Dataframes-esque mental model. Much nicer IMHO https://databricks.com/blog/2016/03/03/introducing-graphframes.html https://databricks.com/blog/2016/03/03/introducing-graphfram...
- mziel 10y ago
  GraphFrames are a graph-based model. Spark DataFrames is the model directly inspired by R/Pandas.
- taeric 10y ago
  I'm curious just how beneficial a giant all inclusive framework is for this. My gut is it will be a huge boon for most of us, but you will still find very valid use cases for rolling your own. That is, I see this as akin to linked lists. Something that people typically do not do on their own anymore, but those that do typically do so for a very good reason.
- ap22213 10y ago
  I was really looking forward to using Datasets. unfortunately, I've had much better results sticking with RDDs. I migrated to 2.0 a few months ago and have been banging my head ever since. RDDs are very straight forward. You get a distributed collection, and you apply functions to them. It's (almost) very explicit about what's going on. I've been trying to do similar things with Datasets that are super simple with RDDs, but the Dataset / SQL interface hides too many details. I really tried to give them a fair shot. But, I ended up falling back to the RDD interface. However, in my case, I'm dealing with 50+ TiB of data, so understanding how memory and processing are being used is very important. It's probably less important for casual users.
  - mziel 10y ago
    Do you have a write-up of your experience by any chance? We're primarily using DataFrames (since we're doing lots of ML processing) and Pipelines API, but I was looking forward to switching to Dataset API.
lobster_johnson 10y ago
Question for those of you who use Spark: Is it possible to use with Go, Rust, Nim or C++? I'm not a fan of those other languages, and I'm also trying to reduce the amount of context switching these days.
- perturbation 10y ago
  It would be possible in principle (especially if you can write a Python module bridge that can wrap your C++ / Rust / etc. code), or something that you can load in Java. But, all of the RDD and SparkContext APIs are going to be available only in Java / Scala / Python without a significant amount of work. It's probably not worth it to use another language and wrap it in that way. I would recommend using Scala (for the static typing) or Python (numpy and other libraries are very useful with Spark). They're not hard to pick up, especially when using pyspark + ipython for prototyping locally.

162 points by akashtndn 10y ago

Funky: 10 to 20 local disks, for approximately 1GB/s to 2GB/s of disk bandwidth The new MacBook Pros apparently have north of 2GB/s of disk bandwidth for their internal SSD. https://9to5mac.com/2016/11/01/the-late-2016-entry-level-13-macbook-pro-has-a-ridiculously-fast-ssd/ https://9to5mac.com/2016/11/01/the-late-2016-entry-level-13-...

vidarh 10y ago
You can pick up a wide variety of PCIe SSDs with that kind of bandwidth as well. It's worth keeping in mind, because it means that a whole lot of workloads that people think they need a cluster for end up being far faster and cheaper to put on a single server these days, given the cost and overheads of sufficiently high speed network interconnects to move the data around at a fast enough to compete. Of course there are plenty of workloads where you still need a cluster, but people often don't realise just how much you can put into a single commodity server today at a reasonable cost, nor what kind of overheads going from a single server to a cluster tends to add.
- mistermann 10y ago
  If you could fit on one box, would you be better off using a sql database (assuming you knew you didn't need to scale beyond one box), or is a single node spark box faster? Or does it depend on the scenario?
  - mitchell_h 10y ago
    Spark and a SQL database have different use cases. Spark is a n analytics engine to do processing on data. SQL databases are for storage and retrieval of data(they just so happen to be able to do various analytical type stuff).
  - bsg75 10y ago
    Depends on where your bottleneck is. If its disk, maybe a one box approach is suitable. If CPU, where many current (F)OSS RDBMS are limited to one-core-per-query, a cluster can help. Additionally a cluster can manage a larger working data set in RAM than a single box can do.
    - vidarh 10y ago
      > If CPU, where many current (F)OSS RDBMS are limited to one-core-per-query, a cluster can help. If you're going to partition the data because of limitations like that, it may still be faster to run a "cluster" on a single server, depending on data volume. > Additionally a cluster can manage a larger working data set in RAM than a single box can do. That's true in theory. In practice I've never seen anyone max out the amount of RAM possible in off the shelf x86 servers. I'm sure it happens, but it's not common, and it's fairly unusual for people to even approach the point where adding ore RAM starts getting more expensive than adding more boxes. You can currently fit at least 8TB of RAM in the biggest x86 boxes; possibly more by now, and boxes that can fit 1TB-2TB are still relatively cheap. Unless you plan to exceed at least 1TB, it's likely going to be cheaper to stuff more memory in a box than add enough extra boxes to compensate for the communications overhead. Of course there are plenty of datasets where you would exceed 1TB for your dataset, but most people don't get anywhere close to that. I would say that before a hardware cluster (as opposed to dirty "hacks" like sharding on a single box) makes sense if you are buying new hardware (the maths of course looks different if one option is to make do with existing servers), you will need to have requirements that exceeds one or more of: - 48-64 x86 cores. - 1TB-2TB of RAM. - 2GB/sec aggregate disk bandwidth. For all of these, it's worth noting that you can't compare like for like, as the moment you go to a cluster you have the according overheads in both CPU, RAM and disk IO of having to spread the work out, write data to disk more places, duplicate data in memory etc.. Note also that you can do better than that on plain x86 hardware, but the above is the point where the incremental cost of increases starts to really hurt and eat into any savings vs. a cluster.
      - bsg75 10y ago
        > That's true in theory. In practice I've never seen anyone max out the amount of RAM possible in off the shelf x86 servers. I'm currently experimenting with a 4 node cluster running Apache Drill (on top of MapR's Hadoop distro), with 500GB RAM, 96 cores, and 28TB of SSD. For some analytic queries that span multiple date partitions, its actually easy to see 70% RAM capacity being utilized, and 80% of the CPU. The size of the data set is currently about 7TB, but that replicates for both data locality and redundancy. We are a Postgres shop, and have tested with some of the options that run a "single node cluster" but one of the limiting factors is that RAM is divided between those DB processes, and performance is impacted. Plus we then need to implement some form of replication to obtain redundancy, fault tolerance, or HA. Quite simply we are not using a cluster because our data won't fit, but because we can get better performance and reliability on 4 nodes. We likely won't have the need for 400. There are cases where for non-interactive data processing we definitely don't need a cluster. At its simplest, multiple copies of Awk scripts can be run using GNU Parallel, Python scripts can be parallelized with the Multiprocessing module, and Go has been useful for concurrency (especially to us Python programmers).
supergirl 10y ago
Yeah but can you store same amount of data on one ssd as on 10 disks?
- snovv_crash 10y ago
  This. If you can fit it on your soldered-in SSD, I'm not sure it really counts as "Big Data".
  - RBerenguel 10y ago
    Well, O(100GB) is where I start thinking of "big data". Processing 100 GB is slow in a single "normal" machine (depending on what, sure). Between 10 and 100 I usually do it in a cluster since it is less hassle (even if I could process it locally with some tweaks or patience). Less than 10 is usually locally run unless I'm already computing something else
    - matteuan 10y ago
      100GB can be handled without problems by mysql or postgre on a single node (yeah maybe not a laptop). You need to tune your database and take the right design decisions, but it's still less work than setting up a distributed system.
      - EdwardDiego 10y ago
        Depends on how you're setting it up. aws s3 cp --recursive bunch-o-data s3://some-bucket/ spark-ec2 --region eu-west-1 --identity-file s.pem --key-pair=spark --instance-type m3.2xlarge --slaves 40 -v 1.5.2 launch my-cluster Is significantly easier than making PG work easily at the 100GB scale in my experience. spark-ec2 is a script that ships with Spark to make it easy to set up a cluster in AWS.
        nchammas 10y ago
        As of Spark 2.0 spark-ec2 doesn't ship with Spark anymore [0]. It's been moved here [1]. [0] http://spark.apache.org/releases/spark-release-2-0-0.html#removals http://spark.apache.org/releases/spark-release-2-0-0.html#re... [1] https://github.com/amplab/spark-ec2 https://github.com/amplab/spark-ec2

techno_modus 10y ago

> The RDD programming model provides only distributed collections of object It is ok for a non-technical high level overview but RDD should probably be referred to as a data model (how data is represented and managed) while map-reduce is a programming model (how data is being processed).

kod 10y ago
No, the rdd interface has methods on it that directly relate to how data is processed.
- techno_modus 10y ago
  You are right, but what I meant is that delta between Spark and Hadoop is RDD (RDD - Resilient Distributed Dataset) which is data structure first of all. Thus Spark and Hadoop have the same data processing model implemented over different data representation models (and hence the performance gain).
  - marmaduke 10y ago
    > delta why can't we use the word difference? delta, mathematically, suggest some space and measure.

mydpy 10y ago

If you follow the Spark community closely, you won't find any new information here. However, as a software consultant, I find this to be a useful nontechnical overview to send to partners and clients.

sandGorgon 10y ago

The new programming model of spark is around Graphframes...which are a Pandas/Dataframes-esque mental model. Much nicer IMHO https://databricks.com/blog/2016/03/03/introducing-graphframes.html https://databricks.com/blog/2016/03/03/introducing-graphfram...

mziel 10y ago
GraphFrames are a graph-based model. Spark DataFrames is the model directly inspired by R/Pandas.
taeric 10y ago
I'm curious just how beneficial a giant all inclusive framework is for this. My gut is it will be a huge boon for most of us, but you will still find very valid use cases for rolling your own. That is, I see this as akin to linked lists. Something that people typically do not do on their own anymore, but those that do typically do so for a very good reason.
ap22213 10y ago
I was really looking forward to using Datasets. unfortunately, I've had much better results sticking with RDDs. I migrated to 2.0 a few months ago and have been banging my head ever since. RDDs are very straight forward. You get a distributed collection, and you apply functions to them. It's (almost) very explicit about what's going on. I've been trying to do similar things with Datasets that are super simple with RDDs, but the Dataset / SQL interface hides too many details. I really tried to give them a fair shot. But, I ended up falling back to the RDD interface. However, in my case, I'm dealing with 50+ TiB of data, so understanding how memory and processing are being used is very important. It's probably less important for casual users.
- mziel 10y ago
  Do you have a write-up of your experience by any chance? We're primarily using DataFrames (since we're doing lots of ML processing) and Pipelines API, but I was looking forward to switching to Dataset API.

lobster_johnson 10y ago

Question for those of you who use Spark: Is it possible to use with Go, Rust, Nim or C++? I'm not a fan of those other languages, and I'm also trying to reduce the amount of context switching these days.

perturbation 10y ago
It would be possible in principle (especially if you can write a Python module bridge that can wrap your C++ / Rust / etc. code), or something that you can load in Java. But, all of the RDD and SparkContext APIs are going to be available only in Java / Scala / Python without a significant amount of work. It's probably not worth it to use another language and wrap it in that way. I would recommend using Scala (for the static typing) or Python (numpy and other libraries are very useful with Spark). They're not hard to pick up, especially when using pyspark + ipython for prototyping locally.