3 ms·

Some additional background: * AWS has a managed Spark offering called EMR * EMR pricing (https://aws.amazon.com/emr/pricing/ https://aws.amazon.com/emr/pricin

Some additional background: * AWS has a managed Spark offering called EMR * EMR pricing (https://aws.amazon.com/emr/pricing/ https://aws.amazon.com/emr/pricing/) is lower than Databricks pricing (https://databricks.com/product/aws-pricing https://databricks.com/product/aws-pricing) * Databricks notebook development experience is better than EMR (but still really basic compared to IntelliJ / PyCharm text editing) * Both Databricks & EMR have proprietary Spark runtimes * Databricks is building a Spark runtime in C++ that might be faster (Delta Engine) * Spark lets you process massive datasets easily, with small teams. 2-3 person teams can build data ingestion pipelines to clean & process terabytes of data a day. It's an incredible technology. * The difference between the PySpark & Scala APIs confuse the hell out of people * Whether or not ppl can run Python machine learning models on Spark clusters confuses people * Overreliance on notebooks causes big issues (no version control, tests, deployment process, dependency management) The big data ecosystem is constantly evolving and you need to study constantly to keep up.

dragonwriter 6y ago
> Overreliance on notebooks causes big issues (no version control, tests, deployment process, dependency management) The last three of those things might be valid issues, but since when are notebooks not just as subject as any other source code format to version control (I get that the difference between the UI and the on-site structure may make typical diff tools less-than-ideal, but VC itself is unaffected.)
- HuwFulcher 6y ago
  In my experience the use of notebooks (exclusively) goes hand in hand with not knowing things such as version control, tests, deployment process or dependency management exist. I don't mean to sound harsh about other Data Scientists from a non software engineering background but the standard workflow is to fiddle around with a notebook until you can get a result. That's as far as it goes, no real robustness to it. That's a pretty big generalisation but in organisations where they "home grow" their Data Science capability many of the online courses don't cover production level Data Science.
  - MrPowers 6y ago
    Your experience aligns with what I've seen. All the notebooks are in one place. Some are for important production jobs, other are for data exploration. It's easy to make a little edit in a notebook and accidentally break production jobs. Comparatively harder to make an edit in a git repo and do a deploy that'll break production jobs (e.g. if the JAR doesn't compile or the CI errors out cause the tests don't pass). Notebook based production jobs get even more dangerous when NotebookA depends on NotebookB and so on.
    - alexott 6y ago
      Treat notebooks like other code - separate onto staging and production, with defined promotions between them - it’s possible. You can run tests in CI/CD pipelines, etc. You can set permissions so nobody can update production notebooks manually, ...
alexott 6y ago
The last point isn’t so dramatic. There is version control for notebooks, and better version is coming (right now it’s in preview - code name - Projects, it’s in official docs). Tests are possible - either Nutter from Microsoft, or home grown (about 20-30 lines on top of built in unittest). Deployment is also not so complicated - there are tools (provider for terraform, cicd-templates project, Databricks-cli, etc), you can refer MS Learn for short course about CI/CD for Databricks and Azure DevOps, etc...
- MrPowers 6y ago
  The last point was for teams that only rely on notebooks, sorry if I didn't make that clear. You're right that all those issues can be sidestepped if you build projects in version controlled Git repos, test the code, and deploy JAR / Wheel files. Speaking of testing, can you let me know if this PySpark testing fix worked for you ;) https://github.com/MrPowers/chispa/issues/6 https://github.com/MrPowers/chispa/issues/6
  - alexott 6y ago
    I’m sorry for delay, will fix ASAP... My point is that you can do that even without jars/wheels - you can do VC and tests of notebooks. For example, https://github.com/alexott/databricks-nutter-projects-demo https://github.com/alexott/databricks-nutter-projects-demo
  - Someone 6y ago
    As alexott says, you can link notebooks to a git repository and commit and rollback them from the Databricks UI (https://docs.databricks.com/notebooks/github-version-control.html https://docs.databricks.com/notebooks/github-version-control...) Problem is that it’s limited. You can’t, for example, commit multiple files in one go (that improves a little bit in https://docs.databricks.com/projects.html https://docs.databricks.com/projects.html), or merge changes a colleague made with your changes. You also have to use the UI Databricks provides. You can’t use a git CLI or whatever GUI you prefer. (all AFAIK, but I’m fairly certain about it)
nchammas 6y ago
> * AWS has a managed Spark offering called EMR There is also my rinky-dink open source project, Flintrock [0], that will launch open source Spark clusters on AWS for you. It's probably not the right tool for production use (and you would be right to wonder why Flintrock exists when we have EMR [1]), but I know of several companies that have used Flintrock at one point or other in production at large scale (like, 400+ node clusters). [0]: https://github.com/nchammas/flintrock https://github.com/nchammas/flintrock [1]: https://github.com/nchammas/flintrock#why-build-flintrock-when-we-have-emr https://github.com/nchammas/flintrock#why-build-flintrock-wh...
throwaway556179 6y ago
We recently ran a large clustering job over billions of records (and a few TBs of data on spinning disks) on a single machine with minimal command line tooling in a few hours. Not really optimized yet. People forget how fast modern hardware is and overestimate how much useful data they have (or need). I think I should start a company around minimalistic data tooling or the like - the amount of waste seems large across the industry. I saw the de-skilling a few year ago, where a guy stiched together a compete application from a couple of SaaS APIs. Cool, but it somehow does not impress me.
- paulryanrogers 6y ago
  Bare metal can be incredibly fast, if you can get access for a reasonable price. Virtualization is becoming a continuum but the overhead is always there.
  - Proven 6y ago
    what?? virtualization has negligible overhead.
- MrPowers 6y ago
  Great point, r5.metal instances have 96 CPUs and 768 GB of RAM. Lots of "big data problems" can actually be solved with a single big EC2 instance. Cluster computing should always be avoided when a single node will do.
  - throwaway556179 6y ago
    We had something like 24G of RAM, but we have 500GB RAM machines as well (we own the hardware) and the job would have been even more of a breeze there.
- bastawhiz 6y ago
  > People forget how fast modern hardware is and overestimate how much useful data they have (or need). Where your model breaks down is when you have folks throughout your engineering org who have data needs but don't have folks to spin up bespoke pipelines like this and spend time optimizing them. You have data, you want to write something SQL-ish or have some nice APIs, and you want to get results dumped into a predictable place. When you have mixed workloads, mixed data sources, and those jobs are being tweaked and changed frequently, the actual underlying compute cost is hardly the issue. Getting the data, chewing on it [fast enough] without having to spend much time optimizing, getting it to its destination, and making that happen regularly and reliably is where the value is.
nknealk 6y ago
> The difference between the PySpark & Scala APIs confuse the hell out of people I personally had this experience when first wanting to learn spark, and it really turned me off to the whole spark ecosystem. Curious if you have any suggested resources that do a good job on this?
- MrPowers 6y ago
  Spark offers Scala, Python, Java, and R APIs. Scala & Python are the most viable options (R lacks a lot of features and Java is only good for ppl that love Java). Scala & PySpark are both great options. Lots of devs are terrified of Scala, so PySpark is more popular now. I'd say Scala has a slight technical advantage, see here for more details: https://mungingdata.com/apache-spark/python-pyspark-scala-which-better/ https://mungingdata.com/apache-spark/python-pyspark-scala-wh.... Both are great overall. I wrote a book that's a practical introduction to Spark: https://leanpub.com/beautiful-spark/ https://leanpub.com/beautiful-spark/ Most of the training materials are theoretical, which makes Spark seem really intimidating. You can learn some basic Spark principles and get up-and-running with production workflows quickly.

dragonwriter 6y ago

> Overreliance on notebooks causes big issues (no version control, tests, deployment process, dependency management) The last three of those things might be valid issues, but since when are notebooks not just as subject as any other source code format to version control (I get that the difference between the UI and the on-site structure may make typical diff tools less-than-ideal, but VC itself is unaffected.)

HuwFulcher 6y ago
In my experience the use of notebooks (exclusively) goes hand in hand with not knowing things such as version control, tests, deployment process or dependency management exist. I don't mean to sound harsh about other Data Scientists from a non software engineering background but the standard workflow is to fiddle around with a notebook until you can get a result. That's as far as it goes, no real robustness to it. That's a pretty big generalisation but in organisations where they "home grow" their Data Science capability many of the online courses don't cover production level Data Science.
- MrPowers 6y ago
  Your experience aligns with what I've seen. All the notebooks are in one place. Some are for important production jobs, other are for data exploration. It's easy to make a little edit in a notebook and accidentally break production jobs. Comparatively harder to make an edit in a git repo and do a deploy that'll break production jobs (e.g. if the JAR doesn't compile or the CI errors out cause the tests don't pass). Notebook based production jobs get even more dangerous when NotebookA depends on NotebookB and so on.
  - alexott 6y ago
    Treat notebooks like other code - separate onto staging and production, with defined promotions between them - it’s possible. You can run tests in CI/CD pipelines, etc. You can set permissions so nobody can update production notebooks manually, ...

alexott 6y ago

The last point isn’t so dramatic. There is version control for notebooks, and better version is coming (right now it’s in preview - code name - Projects, it’s in official docs). Tests are possible - either Nutter from Microsoft, or home grown (about 20-30 lines on top of built in unittest). Deployment is also not so complicated - there are tools (provider for terraform, cicd-templates project, Databricks-cli, etc), you can refer MS Learn for short course about CI/CD for Databricks and Azure DevOps, etc...

MrPowers 6y ago
The last point was for teams that only rely on notebooks, sorry if I didn't make that clear. You're right that all those issues can be sidestepped if you build projects in version controlled Git repos, test the code, and deploy JAR / Wheel files. Speaking of testing, can you let me know if this PySpark testing fix worked for you ;) https://github.com/MrPowers/chispa/issues/6 https://github.com/MrPowers/chispa/issues/6
- alexott 6y ago
  I’m sorry for delay, will fix ASAP... My point is that you can do that even without jars/wheels - you can do VC and tests of notebooks. For example, https://github.com/alexott/databricks-nutter-projects-demo https://github.com/alexott/databricks-nutter-projects-demo
- Someone 6y ago
  As alexott says, you can link notebooks to a git repository and commit and rollback them from the Databricks UI (https://docs.databricks.com/notebooks/github-version-control.html https://docs.databricks.com/notebooks/github-version-control...) Problem is that it’s limited. You can’t, for example, commit multiple files in one go (that improves a little bit in https://docs.databricks.com/projects.html https://docs.databricks.com/projects.html), or merge changes a colleague made with your changes. You also have to use the UI Databricks provides. You can’t use a git CLI or whatever GUI you prefer. (all AFAIK, but I’m fairly certain about it)

nchammas 6y ago

> * AWS has a managed Spark offering called EMR There is also my rinky-dink open source project, Flintrock [0], that will launch open source Spark clusters on AWS for you. It's probably not the right tool for production use (and you would be right to wonder why Flintrock exists when we have EMR [1]), but I know of several companies that have used Flintrock at one point or other in production at large scale (like, 400+ node clusters). [0]: https://github.com/nchammas/flintrock https://github.com/nchammas/flintrock [1]: https://github.com/nchammas/flintrock#why-build-flintrock-when-we-have-emr https://github.com/nchammas/flintrock#why-build-flintrock-wh...

throwaway556179 6y ago

We recently ran a large clustering job over billions of records (and a few TBs of data on spinning disks) on a single machine with minimal command line tooling in a few hours. Not really optimized yet. People forget how fast modern hardware is and overestimate how much useful data they have (or need). I think I should start a company around minimalistic data tooling or the like - the amount of waste seems large across the industry. I saw the de-skilling a few year ago, where a guy stiched together a compete application from a couple of SaaS APIs. Cool, but it somehow does not impress me.

paulryanrogers 6y ago
Bare metal can be incredibly fast, if you can get access for a reasonable price. Virtualization is becoming a continuum but the overhead is always there.
- Proven 6y ago
  what?? virtualization has negligible overhead.
MrPowers 6y ago
Great point, r5.metal instances have 96 CPUs and 768 GB of RAM. Lots of "big data problems" can actually be solved with a single big EC2 instance. Cluster computing should always be avoided when a single node will do.
- throwaway556179 6y ago
  We had something like 24G of RAM, but we have 500GB RAM machines as well (we own the hardware) and the job would have been even more of a breeze there.
bastawhiz 6y ago
> People forget how fast modern hardware is and overestimate how much useful data they have (or need). Where your model breaks down is when you have folks throughout your engineering org who have data needs but don't have folks to spin up bespoke pipelines like this and spend time optimizing them. You have data, you want to write something SQL-ish or have some nice APIs, and you want to get results dumped into a predictable place. When you have mixed workloads, mixed data sources, and those jobs are being tweaked and changed frequently, the actual underlying compute cost is hardly the issue. Getting the data, chewing on it [fast enough] without having to spend much time optimizing, getting it to its destination, and making that happen regularly and reliably is where the value is.

nknealk 6y ago

> The difference between the PySpark & Scala APIs confuse the hell out of people I personally had this experience when first wanting to learn spark, and it really turned me off to the whole spark ecosystem. Curious if you have any suggested resources that do a good job on this?

MrPowers 6y ago
Spark offers Scala, Python, Java, and R APIs. Scala & Python are the most viable options (R lacks a lot of features and Java is only good for ppl that love Java). Scala & PySpark are both great options. Lots of devs are terrified of Scala, so PySpark is more popular now. I'd say Scala has a slight technical advantage, see here for more details: https://mungingdata.com/apache-spark/python-pyspark-scala-which-better/ https://mungingdata.com/apache-spark/python-pyspark-scala-wh.... Both are great overall. I wrote a book that's a practical introduction to Spark: https://leanpub.com/beautiful-spark/ https://leanpub.com/beautiful-spark/ Most of the training materials are theoretical, which makes Spark seem really intimidating. You can learn some basic Spark principles and get up-and-running with production workflows quickly.