9 ms·

Why Databricks Is Winning

dpq 6y ago
We've had mixed experience with Databricks, to be honest. The quality and responsiveness of support we were getting was not worth the amount of issues we were seeing + the considerable cost, so we decided to use "exit strategy to DIY Spark" (using the original post's terms) instead and we're pretty happy so far.
- benjaminwootton 6y ago
  Cost is one thing we didn't really get to grips with. They use a fairly abstract Databricks Processing Unit - https://databricks.com/product/aws-pricing https://databricks.com/product/aws-pricing - and the costs felt disjointed compared to the workloads we were running. This said, cost didn't really spiral or become an issue for us, especially when you take into account the cost avoidance of administering the Spark cluster and all of the tools. However, the pricing model did feel a little opaque.
- mathattack 6y ago
  Their sales team is somewhere between Oracle and a pro wrestling villain in terms of sleaziness.
mobileexpert 6y ago
Databricks seems on a convergent evolution towards Snowflake, between the two of them I’d rather be starting at the position Snowflake is versus Databricks.
- benjaminwootton 6y ago
  They are both going towards the "Data Lakehouse" end point which is driving some of the convergence. Silly term, but basically providing analytics and a database type experience over a data lake. That said, Databricks is a much broader platform, with all of the collaboration environments and is generally much more programmable than Snowflake.
  - nchammas 6y ago
    > providing analytics and a database type experience over a data lake. It's interesting how the modern data lake is developing in this way, recreating many patterns from the traditional database for distributed systems and massive scale: SQL and query optimization, transactions and time travel, schema evolution and data constraints... Having started out as a database developer / DBA many years ago, working with data lakes today reminds me in many ways of that early part of my career. I wrote a post tracing a common interface from the typical relational database to the modern data lake. https://nchammas.com/writing/modern-data-lake-database https://nchammas.com/writing/modern-data-lake-database
flowerlad 6y ago
"Fully managed Spark" sounded awesome a few years ago. But now, Spark can run on Kubernetes clusters [1]. If your infrastructure is already running on Kubernetes then you already have a cluster capable of running Spark. And because of the magic of Kubernetes you don't even have to dedicate nodes to Spark. [1] https://spark.apache.org/docs/latest/running-on-kubernetes.html https://spark.apache.org/docs/latest/running-on-kubernetes.h...
- bpodgursky 6y ago
  100%. The DataBricks notebooks have a lot of value (for now) but ever since running Spark on Kube... I have had literally 0 cluster issues. It's absolutely shocking, coming from a YARN-based environment, where I was constantly plagued with ResourceManager issues, autoscaling issues, preemption issues, network disconnect issues...
lykr0n 6y ago
The one thing I see in my current company, and a growing trend with SaaS apps is that companies are forgetting how to actually engineer. Like Boeing- the more you outsource the less able you're able to react to changing market forces and fix issues. We run Hadoop & Spark internally, but the team is underfunded and stuck in a constant cycle of fighting fires. And the result (and part of a larger push of the company due to the same cycle of under-funding and culture issues) is that we're moving our petabytes of data to cloud providers into their systems. Not only is the cost of doing this dwarfing that it would take to actually fix our issues, but we're going to lose the people who know how to design and manage petabyte scale hadoop clusters. We wind up in a situation where we locked up data fundamental to our company and our position in the market with a 3rd party, and losing the talent that would allow us to maintain full control over the data. If the service increases prices, changes it's offering, or we get to a point where the offering doesn't meet our needs- we're fucked. It's nice that Databricks has a nice "offramp" that you can take to go somewhere else, but the general idea is the same.
- random314 6y ago
  This happens with every technology stack from steam engines to software. There was a time when programmers could solder together an ALU using transistor gates.
  - doctor_eval 6y ago
    Well yeah I could probably have done that with some 7400 series TTL ... but the MIPS wouldn't nearly cut it in today's world :)
  - tluyben2 6y ago
    It might be the same with 'trains' (steam engines); I don't know much about that. But losing skills in 'the cloud' makes you prisoner to PaaS/SaaS providers; if these skills almost completely disappear (let's say, akin to your transistor gates, that almost no-one can install or maintain server setups anymore), the cloud providers really can do whatever they want price wise. For instance, if a cloud provider misconfigures something at scale in this great future, there is no way for anyone to verify that; it might be that you are paying 1000x what you should be paying for a particular database query that triggers a perfect storm in your set-up. As no-one in your company knows and no-one can check what or even if this is going on, it'll be considered 'normal' and you just pay. This is already happening, at scale, but not due to the cloud provider's fault, but because people are already not learning skills. If I would have $100 for every missing index I find per month in client's setups, I would be working a few hours/month for a very nice wage. Heck, most startup databases I see these days don't have any indexes at all; and then they wonder why they are paying $10k/month at aws while I can get them the same or better performance for a few bucks/month on a vps. Not because I'm so smart, but just doing everything 'low-level' wrong because no-one simply knows anymore how a database works (it's the cloud! we shouldn't have to!). It's not restricted to relational dbs either; sometimes I cry when I see dynamodb setups made by people who think this is just the cloud way of working; 'throw in whatever data, it'll always optimise itself'. So this is already happening at many companies and it's getting worse.
fmajid 6y ago
I'd say the difference in valuation between Snowflake and Databricks clearly shows the latter is not "winning".
- MrPowers 6y ago
  Snowflake and Databricks are different, sometimes complementary technologies. You can store data in Snowflake & query it with Databricks for example: https://github.com/snowflakedb/spark-snowflake https://github.com/snowflakedb/spark-snowflake Snowflake predicate pushdown filtering seems quite promising: https://www.snowflake.com/blog/snowflake-spark-part-2-pushing-query-processing/ https://www.snowflake.com/blog/snowflake-spark-part-2-pushin... Think both these companies can win.
  - jayparth 6y ago
    There's definitely place in the industry for a no 1 and a no 2, but no. I don't think they're complementary, I think they're competitive. Both are trying to own the whole storage + compute layer eventually. I guess you can see Snowflake as a storage layer but they have attached scalable compute to it. Databricks started out as scalable compute and is now becoming more focused on storage. The idea of storing your data in Snowflake and querying it in Databricks is pretty silly. Why would you want to do that? Why not just use Snowflake's compute? Sure you could argue Spark has some transformations that are hard to express in SQL, but that is why Snowflake introduced Snowpark.
- mr_toad 6y ago
  Snowflake spends an insane amount on marketing. And they’re Apples and Oranges. Snowflake is more comparable to Athena and BigQuery.
fs111 6y ago
DBFS is a complete joke though. A filesystem that has no timestamps? realy?
- cfeduke 6y ago
  It's an abstraction over other file systems, like S3 or Azure BLOB storage. More a convenience than anything else, and helps ease porting code between cloud providers. Not a traditional file system, or even a HDFS clone.
- stormwindy 6y ago
  A bit late but https://docs.databricks.com/release-notes/product/2021/february.html#added-modification_time-to-the-dbfs-rest-api-get-status-and-list-responses https://docs.databricks.com/release-notes/product/2021/febru...
MrPowers 6y ago
Some additional background: * AWS has a managed Spark offering called EMR * EMR pricing (https://aws.amazon.com/emr/pricing/ https://aws.amazon.com/emr/pricing/) is lower than Databricks pricing (https://databricks.com/product/aws-pricing https://databricks.com/product/aws-pricing) * Databricks notebook development experience is better than EMR (but still really basic compared to IntelliJ / PyCharm text editing) * Both Databricks & EMR have proprietary Spark runtimes * Databricks is building a Spark runtime in C++ that might be faster (Delta Engine) * Spark lets you process massive datasets easily, with small teams. 2-3 person teams can build data ingestion pipelines to clean & process terabytes of data a day. It's an incredible technology. * The difference between the PySpark & Scala APIs confuse the hell out of people * Whether or not ppl can run Python machine learning models on Spark clusters confuses people * Overreliance on notebooks causes big issues (no version control, tests, deployment process, dependency management) The big data ecosystem is constantly evolving and you need to study constantly to keep up.
- dragonwriter 6y ago
  > Overreliance on notebooks causes big issues (no version control, tests, deployment process, dependency management) The last three of those things might be valid issues, but since when are notebooks not just as subject as any other source code format to version control (I get that the difference between the UI and the on-site structure may make typical diff tools less-than-ideal, but VC itself is unaffected.)
  - HuwFulcher 6y ago
    In my experience the use of notebooks (exclusively) goes hand in hand with not knowing things such as version control, tests, deployment process or dependency management exist. I don't mean to sound harsh about other Data Scientists from a non software engineering background but the standard workflow is to fiddle around with a notebook until you can get a result. That's as far as it goes, no real robustness to it. That's a pretty big generalisation but in organisations where they "home grow" their Data Science capability many of the online courses don't cover production level Data Science.
simo7 6y ago
A (big?) part of Databricks' success is the complicated mess that Spark is to run.
- HuwFulcher 6y ago
  Considering the makers of Databricks are also the maintainers of Spark it makes you wonder whether this is deliberate
peterthehacker 6y ago
I’ve had a lot of success with Dask lately. It’s comparable to spark in some ways [0]. Being written in python and built on top of pandas/numpy it allows much more flexibility. It’s also easier to get adoption from data scientists who are usually more comfortable with python than scala. It also has great tools built on top of kubernetes making deployment quick and easy [1]. [0]https://docs.dask.org/en/latest/spark.html https://docs.dask.org/en/latest/spark.html [1]https://github.com/dask/dask-gateway https://github.com/dask/dask-gateway
- MrPowers 6y ago
  My feeling is that Spark is better for huge ETL jobs and Dask is better for certain types of model building (because of easy access to Python libraries), but don't have any benchmarks to back this up. Would love some benchmark results comparing processing tens of terabytes on equal sized i3.xlarge EC2 clusters to get a better idea of benchmarks. Have you seen any good Spark vs. Dask benchmarks?
  - peterthehacker 6y ago
    I would be interested in seeing some spark v dask benchmarks too. Haven’t seen any yet though. In my experience, Dask really shines when you implement custom numpy computations that could only be done in spark UDFs. We saw a decent performance difference there, but for common built-in computations I’d imagine that spark has better performance. Edit: after some googling I found this paper with benchmarks. https://arxiv.org/pdf/1907.13030.pdf https://arxiv.org/pdf/1907.13030.pdf > Results show that despite slight differences between Spark and Dask, both engines perform comparably. However, Dask pipelines risk being limited by Python’s GIL depending on task type and cluster configuration. In all cases, the major limiting factor was data transfer.
    - kornish 6y ago
      It's also worth noting that with Spark, you can perform arbitrary computation using the Dataset API and operating on case classes.
  - 6y ago
00117 6y ago
Why does nobody discuss Apache Pulsar as a viable alternative?
- peterthehacker 6y ago
  Apache Pulsar is a streaming platform more comparable to Kafka. It doesn’t have built in parallel computation APIs like spark. You can hook spark streaming up with pulsar as a data source though. https://pulsar.apache.org/docs/en/adaptors-spark/ https://pulsar.apache.org/docs/en/adaptors-spark/
spicyramen 6y ago
For reasons I don't understand our company ended up choosing Google Cloud and is a complete mess. You have BigQuery, Dataproc, AI Platform training, Colab, AI Platform notebooks, and many tools that do the same thing but not well integrated. My personal.favourite is AI Notebooks, but i need additional.plugins to interact with BigQuery, S3 and GCS. Requires a lot of customization, we used before Azure and Databricks where we had one stop shop. I heard Google is integrating some of their products but that will take a while. In the meantime we lose hours of productivity figuring out how to use their products (which stability is horrible. Example AI.platform Jobs and permission model)
- Jugurtha 6y ago
  Hey, I replied[0] to your submission[1] a couple of weeks ago and elaborated on our internal platform for collaborative notebooks, scheduled notebooks, and model management. How is your effort going so far? - [0]: https://news.ycombinator.com/item?id=25940923 https://news.ycombinator.com/item?id=25940923 - [1]: https://news.ycombinator.com/item?id=25938654 https://news.ycombinator.com/item?id=25938654
fractionalhare 6y ago
Interestingly, my team is actually moving off Databricks. I have anecdotally heard the same from other teams in the industry (buy side finance). We found that notebook-based development is actually an antipattern for software engineering. It was ostensibly helpful for the narrower "data science" use case, but we have a much more robust ETL and research platform we built on our own using Pandas, Dask, Prefect and AWS. And personally I hated writing code in notebooks. If you're attached to that, you can basically get the same thing by using PyCharm in scientific mode with cell execution.
ineedasername 6y ago
logging into the same system and interacting with the same datasets through the same Notebook based UI I don't think there's a good one size fits all UI that can be applied the different types of work that take place with data and consumption by users. This is evidenced in Databrick's own feature set which includes an integration with R Studio and Tableau to Databricks as a data source rather than a work environment.
mlthoughts2018 6y ago
I actually posted on a somewhat related topic here: - https://managingml.substack.com/p/ml-model-training-is-an-etl https://managingml.substack.com/p/ml-model-training-is-an-et... As an ML manager one of the things that I dislike about the Databricks model is the reliance on notebooks and bespoke interactive cluster workflows. ML workloads are stuck in the stone age because there’s no common pattern to plug them into existing ETL frameworks, but really that is what they are. Model training is an ETL that just happens to need unique domain tools to visualize how it’s succeeding / track observability metrics. But apart from that, you really need to automate model training and experimentation. It is a total antipattern to ever use notebooks, or even non-notebook code developed interactively. Rather you should be submitting tasks to a task queue that maps your ML exploratory workload or training workload to a scheduled compute resource, runs it with observability baked in, and treats outputs like schema’d outputs of more traditional ETLs. Instead of that, Databricks is much more of an early 2000s MATLAB model. Make it addictive and easy for unpragmatic researchers detached from real production use cases, then figure employers will have to adopt it since all the expensive-to-hire researchers can only be productive with it. Long term I think it’s a very bad gamble. Just consider how much open source Python tools have eaten MATLAB’s lunch in the past 10 years.
- Jugurtha 6y ago
  This is the direction we're going towards with our internal platform[0] that we built to make our life easier. We do have real-time collaborative notebooks. However, we also have long-running notebook scheduling where notebooks get picked by a runners and executed against certain environments. One cool feature is that you can watch the notebook's output as it runs, even if you had closed the browser or logged in from another computer. Anothr cool feature is that we do automatic experiment tracking, so we detect parameters, models, metrics, and the notebook automatically, removing the cognitive load from the use to "remember" doing that. People rarely remember clerical things, but even when they do, we didn't want to pollute the notebooks with tracking code of say MLflow. This happens automatically. These long-running notebooks are simply jobs. There are other types of jobs: building a Docker image for the model so that you can push it to a registry is also a job, so is deploying a model to get a "REST" endpoint and a live dashboard to monitor model performance. One of our guidelines is: "Anything that is possible with point and click must be possible with an API call", so we make sure to make everything programmable. For example, scheduling a long-running notebook can be done sending in a request. We also are working to add in webhooks and use cloudevents spec so that people can take information from our system, such as a job having finished, and trigger things in other systems.. Another is workload portability. Again, it's still early and we're prioritizing the issues that have frustrated us the most for the past years in our projects. There's a long way to go, especially that there are many such as yourself who may have worked on problems we've never seen, or a scale/data we haven't handled before. - [0]: https://iko.ai https://iko.ai
solresol 6y ago
Maybe I'm missing something, but revenue per employee doesn't seem to be rising, which is what I would expect to see out of a successful startup that has found product market fit and is beginning to gain economies of scale. All I see is a capital raise 2x the previous raise each year, with no sign of profit and only a proportional increase in revenue. I can build a business like that: sell Hadoop consulting services at a loss, below the market price. Win lots of business by being really cheap; grow as fast as your cashflow allows, which will require annual investment rounds of ever increasing size. When I look at Google Trends, Hadoop's long-term trajectory looks like "dead by 2024". Do we expect that Databricks will be able to justify a $8B capital raising if Hadoop doesn't really exist? If not, what's the path to profitability? (Not trying to say anything bad about Databricks or their management, I'm just genuinely wondering what's going on.)
- dataisamazing 6y ago
  For what it’s worth, we work with them and thus have been following closely. Databricks’ value add is that are end to end for interfaces to use your data. From ingest to exploration to DS/ML. 1. Your metrics are wrong about revenue/employee. We’ve heard they’re growing well by all standard SaaS company metrics 2. Hadoop consulting business is just a completely incorrect description of the platform. Hadoop is dying because of companies like Databricks, and all that needs to exist to justify their valuation is people bringing them large data workloads. Their biggest problem is release maturity. Between Azure being down and them being down, stuff is down too often

dpq 6y ago

We've had mixed experience with Databricks, to be honest. The quality and responsiveness of support we were getting was not worth the amount of issues we were seeing + the considerable cost, so we decided to use "exit strategy to DIY Spark" (using the original post's terms) instead and we're pretty happy so far.

benjaminwootton 6y ago
Cost is one thing we didn't really get to grips with. They use a fairly abstract Databricks Processing Unit - https://databricks.com/product/aws-pricing https://databricks.com/product/aws-pricing - and the costs felt disjointed compared to the workloads we were running. This said, cost didn't really spiral or become an issue for us, especially when you take into account the cost avoidance of administering the Spark cluster and all of the tools. However, the pricing model did feel a little opaque.
mathattack 6y ago
Their sales team is somewhere between Oracle and a pro wrestling villain in terms of sleaziness.

mobileexpert 6y ago

Databricks seems on a convergent evolution towards Snowflake, between the two of them I’d rather be starting at the position Snowflake is versus Databricks.

benjaminwootton 6y ago
They are both going towards the "Data Lakehouse" end point which is driving some of the convergence. Silly term, but basically providing analytics and a database type experience over a data lake. That said, Databricks is a much broader platform, with all of the collaboration environments and is generally much more programmable than Snowflake.
- nchammas 6y ago
  > providing analytics and a database type experience over a data lake. It's interesting how the modern data lake is developing in this way, recreating many patterns from the traditional database for distributed systems and massive scale: SQL and query optimization, transactions and time travel, schema evolution and data constraints... Having started out as a database developer / DBA many years ago, working with data lakes today reminds me in many ways of that early part of my career. I wrote a post tracing a common interface from the typical relational database to the modern data lake. https://nchammas.com/writing/modern-data-lake-database https://nchammas.com/writing/modern-data-lake-database

flowerlad 6y ago

"Fully managed Spark" sounded awesome a few years ago. But now, Spark can run on Kubernetes clusters [1]. If your infrastructure is already running on Kubernetes then you already have a cluster capable of running Spark. And because of the magic of Kubernetes you don't even have to dedicate nodes to Spark. [1] https://spark.apache.org/docs/latest/running-on-kubernetes.html https://spark.apache.org/docs/latest/running-on-kubernetes.h...

bpodgursky 6y ago
100%. The DataBricks notebooks have a lot of value (for now) but ever since running Spark on Kube... I have had literally 0 cluster issues. It's absolutely shocking, coming from a YARN-based environment, where I was constantly plagued with ResourceManager issues, autoscaling issues, preemption issues, network disconnect issues...

lykr0n 6y ago

The one thing I see in my current company, and a growing trend with SaaS apps is that companies are forgetting how to actually engineer. Like Boeing- the more you outsource the less able you're able to react to changing market forces and fix issues. We run Hadoop & Spark internally, but the team is underfunded and stuck in a constant cycle of fighting fires. And the result (and part of a larger push of the company due to the same cycle of under-funding and culture issues) is that we're moving our petabytes of data to cloud providers into their systems. Not only is the cost of doing this dwarfing that it would take to actually fix our issues, but we're going to lose the people who know how to design and manage petabyte scale hadoop clusters. We wind up in a situation where we locked up data fundamental to our company and our position in the market with a 3rd party, and losing the talent that would allow us to maintain full control over the data. If the service increases prices, changes it's offering, or we get to a point where the offering doesn't meet our needs- we're fucked. It's nice that Databricks has a nice "offramp" that you can take to go somewhere else, but the general idea is the same.

random314 6y ago
This happens with every technology stack from steam engines to software. There was a time when programmers could solder together an ALU using transistor gates.
- doctor_eval 6y ago
  Well yeah I could probably have done that with some 7400 series TTL ... but the MIPS wouldn't nearly cut it in today's world :)
- tluyben2 6y ago
  It might be the same with 'trains' (steam engines); I don't know much about that. But losing skills in 'the cloud' makes you prisoner to PaaS/SaaS providers; if these skills almost completely disappear (let's say, akin to your transistor gates, that almost no-one can install or maintain server setups anymore), the cloud providers really can do whatever they want price wise. For instance, if a cloud provider misconfigures something at scale in this great future, there is no way for anyone to verify that; it might be that you are paying 1000x what you should be paying for a particular database query that triggers a perfect storm in your set-up. As no-one in your company knows and no-one can check what or even if this is going on, it'll be considered 'normal' and you just pay. This is already happening, at scale, but not due to the cloud provider's fault, but because people are already not learning skills. If I would have $100 for every missing index I find per month in client's setups, I would be working a few hours/month for a very nice wage. Heck, most startup databases I see these days don't have any indexes at all; and then they wonder why they are paying $10k/month at aws while I can get them the same or better performance for a few bucks/month on a vps. Not because I'm so smart, but just doing everything 'low-level' wrong because no-one simply knows anymore how a database works (it's the cloud! we shouldn't have to!). It's not restricted to relational dbs either; sometimes I cry when I see dynamodb setups made by people who think this is just the cloud way of working; 'throw in whatever data, it'll always optimise itself'. So this is already happening at many companies and it's getting worse.

fmajid 6y ago

I'd say the difference in valuation between Snowflake and Databricks clearly shows the latter is not "winning".

MrPowers 6y ago
Snowflake and Databricks are different, sometimes complementary technologies. You can store data in Snowflake & query it with Databricks for example: https://github.com/snowflakedb/spark-snowflake https://github.com/snowflakedb/spark-snowflake Snowflake predicate pushdown filtering seems quite promising: https://www.snowflake.com/blog/snowflake-spark-part-2-pushing-query-processing/ https://www.snowflake.com/blog/snowflake-spark-part-2-pushin... Think both these companies can win.
- jayparth 6y ago
  There's definitely place in the industry for a no 1 and a no 2, but no. I don't think they're complementary, I think they're competitive. Both are trying to own the whole storage + compute layer eventually. I guess you can see Snowflake as a storage layer but they have attached scalable compute to it. Databricks started out as scalable compute and is now becoming more focused on storage. The idea of storing your data in Snowflake and querying it in Databricks is pretty silly. Why would you want to do that? Why not just use Snowflake's compute? Sure you could argue Spark has some transformations that are hard to express in SQL, but that is why Snowflake introduced Snowpark.
mr_toad 6y ago
Snowflake spends an insane amount on marketing. And they’re Apples and Oranges. Snowflake is more comparable to Athena and BigQuery.

fs111 6y ago

DBFS is a complete joke though. A filesystem that has no timestamps? realy?

cfeduke 6y ago
It's an abstraction over other file systems, like S3 or Azure BLOB storage. More a convenience than anything else, and helps ease porting code between cloud providers. Not a traditional file system, or even a HDFS clone.
stormwindy 6y ago
A bit late but https://docs.databricks.com/release-notes/product/2021/february.html#added-modification_time-to-the-dbfs-rest-api-get-status-and-list-responses https://docs.databricks.com/release-notes/product/2021/febru...

MrPowers 6y ago

Some additional background: * AWS has a managed Spark offering called EMR * EMR pricing (https://aws.amazon.com/emr/pricing/ https://aws.amazon.com/emr/pricing/) is lower than Databricks pricing (https://databricks.com/product/aws-pricing https://databricks.com/product/aws-pricing) * Databricks notebook development experience is better than EMR (but still really basic compared to IntelliJ / PyCharm text editing) * Both Databricks & EMR have proprietary Spark runtimes * Databricks is building a Spark runtime in C++ that might be faster (Delta Engine) * Spark lets you process massive datasets easily, with small teams. 2-3 person teams can build data ingestion pipelines to clean & process terabytes of data a day. It's an incredible technology. * The difference between the PySpark & Scala APIs confuse the hell out of people * Whether or not ppl can run Python machine learning models on Spark clusters confuses people * Overreliance on notebooks causes big issues (no version control, tests, deployment process, dependency management) The big data ecosystem is constantly evolving and you need to study constantly to keep up.

dragonwriter 6y ago
> Overreliance on notebooks causes big issues (no version control, tests, deployment process, dependency management) The last three of those things might be valid issues, but since when are notebooks not just as subject as any other source code format to version control (I get that the difference between the UI and the on-site structure may make typical diff tools less-than-ideal, but VC itself is unaffected.)
- HuwFulcher 6y ago
  In my experience the use of notebooks (exclusively) goes hand in hand with not knowing things such as version control, tests, deployment process or dependency management exist. I don't mean to sound harsh about other Data Scientists from a non software engineering background but the standard workflow is to fiddle around with a notebook until you can get a result. That's as far as it goes, no real robustness to it. That's a pretty big generalisation but in organisations where they "home grow" their Data Science capability many of the online courses don't cover production level Data Science.

simo7 6y ago

A (big?) part of Databricks' success is the complicated mess that Spark is to run.

HuwFulcher 6y ago
Considering the makers of Databricks are also the maintainers of Spark it makes you wonder whether this is deliberate

peterthehacker 6y ago

I’ve had a lot of success with Dask lately. It’s comparable to spark in some ways [0]. Being written in python and built on top of pandas/numpy it allows much more flexibility. It’s also easier to get adoption from data scientists who are usually more comfortable with python than scala. It also has great tools built on top of kubernetes making deployment quick and easy [1]. [0]https://docs.dask.org/en/latest/spark.html https://docs.dask.org/en/latest/spark.html [1]https://github.com/dask/dask-gateway https://github.com/dask/dask-gateway

MrPowers 6y ago
My feeling is that Spark is better for huge ETL jobs and Dask is better for certain types of model building (because of easy access to Python libraries), but don't have any benchmarks to back this up. Would love some benchmark results comparing processing tens of terabytes on equal sized i3.xlarge EC2 clusters to get a better idea of benchmarks. Have you seen any good Spark vs. Dask benchmarks?
- peterthehacker 6y ago
  I would be interested in seeing some spark v dask benchmarks too. Haven’t seen any yet though. In my experience, Dask really shines when you implement custom numpy computations that could only be done in spark UDFs. We saw a decent performance difference there, but for common built-in computations I’d imagine that spark has better performance. Edit: after some googling I found this paper with benchmarks. https://arxiv.org/pdf/1907.13030.pdf https://arxiv.org/pdf/1907.13030.pdf > Results show that despite slight differences between Spark and Dask, both engines perform comparably. However, Dask pipelines risk being limited by Python’s GIL depending on task type and cluster configuration. In all cases, the major limiting factor was data transfer.
  - kornish 6y ago
    It's also worth noting that with Spark, you can perform arbitrary computation using the Dataset API and operating on case classes.
- 6y ago

00117 6y ago

Why does nobody discuss Apache Pulsar as a viable alternative?

peterthehacker 6y ago
Apache Pulsar is a streaming platform more comparable to Kafka. It doesn’t have built in parallel computation APIs like spark. You can hook spark streaming up with pulsar as a data source though. https://pulsar.apache.org/docs/en/adaptors-spark/ https://pulsar.apache.org/docs/en/adaptors-spark/

spicyramen 6y ago

For reasons I don't understand our company ended up choosing Google Cloud and is a complete mess. You have BigQuery, Dataproc, AI Platform training, Colab, AI Platform notebooks, and many tools that do the same thing but not well integrated. My personal.favourite is AI Notebooks, but i need additional.plugins to interact with BigQuery, S3 and GCS. Requires a lot of customization, we used before Azure and Databricks where we had one stop shop. I heard Google is integrating some of their products but that will take a while. In the meantime we lose hours of productivity figuring out how to use their products (which stability is horrible. Example AI.platform Jobs and permission model)

Jugurtha 6y ago
Hey, I replied[0] to your submission[1] a couple of weeks ago and elaborated on our internal platform for collaborative notebooks, scheduled notebooks, and model management. How is your effort going so far? - [0]: https://news.ycombinator.com/item?id=25940923 https://news.ycombinator.com/item?id=25940923 - [1]: https://news.ycombinator.com/item?id=25938654 https://news.ycombinator.com/item?id=25938654

fractionalhare 6y ago

Interestingly, my team is actually moving off Databricks. I have anecdotally heard the same from other teams in the industry (buy side finance). We found that notebook-based development is actually an antipattern for software engineering. It was ostensibly helpful for the narrower "data science" use case, but we have a much more robust ETL and research platform we built on our own using Pandas, Dask, Prefect and AWS. And personally I hated writing code in notebooks. If you're attached to that, you can basically get the same thing by using PyCharm in scientific mode with cell execution.

ineedasername 6y ago

logging into the same system and interacting with the same datasets through the same Notebook based UI I don't think there's a good one size fits all UI that can be applied the different types of work that take place with data and consumption by users. This is evidenced in Databrick's own feature set which includes an integration with R Studio and Tableau to Databricks as a data source rather than a work environment.

mlthoughts2018 6y ago

I actually posted on a somewhat related topic here: - https://managingml.substack.com/p/ml-model-training-is-an-etl https://managingml.substack.com/p/ml-model-training-is-an-et... As an ML manager one of the things that I dislike about the Databricks model is the reliance on notebooks and bespoke interactive cluster workflows. ML workloads are stuck in the stone age because there’s no common pattern to plug them into existing ETL frameworks, but really that is what they are. Model training is an ETL that just happens to need unique domain tools to visualize how it’s succeeding / track observability metrics. But apart from that, you really need to automate model training and experimentation. It is a total antipattern to ever use notebooks, or even non-notebook code developed interactively. Rather you should be submitting tasks to a task queue that maps your ML exploratory workload or training workload to a scheduled compute resource, runs it with observability baked in, and treats outputs like schema’d outputs of more traditional ETLs. Instead of that, Databricks is much more of an early 2000s MATLAB model. Make it addictive and easy for unpragmatic researchers detached from real production use cases, then figure employers will have to adopt it since all the expensive-to-hire researchers can only be productive with it. Long term I think it’s a very bad gamble. Just consider how much open source Python tools have eaten MATLAB’s lunch in the past 10 years.

Jugurtha 6y ago
This is the direction we're going towards with our internal platform[0] that we built to make our life easier. We do have real-time collaborative notebooks. However, we also have long-running notebook scheduling where notebooks get picked by a runners and executed against certain environments. One cool feature is that you can watch the notebook's output as it runs, even if you had closed the browser or logged in from another computer. Anothr cool feature is that we do automatic experiment tracking, so we detect parameters, models, metrics, and the notebook automatically, removing the cognitive load from the use to "remember" doing that. People rarely remember clerical things, but even when they do, we didn't want to pollute the notebooks with tracking code of say MLflow. This happens automatically. These long-running notebooks are simply jobs. There are other types of jobs: building a Docker image for the model so that you can push it to a registry is also a job, so is deploying a model to get a "REST" endpoint and a live dashboard to monitor model performance. One of our guidelines is: "Anything that is possible with point and click must be possible with an API call", so we make sure to make everything programmable. For example, scheduling a long-running notebook can be done sending in a request. We also are working to add in webhooks and use cloudevents spec so that people can take information from our system, such as a job having finished, and trigger things in other systems.. Another is workload portability. Again, it's still early and we're prioritizing the issues that have frustrated us the most for the past years in our projects. There's a long way to go, especially that there are many such as yourself who may have worked on problems we've never seen, or a scale/data we haven't handled before. - [0]: https://iko.ai https://iko.ai

solresol 6y ago

Maybe I'm missing something, but revenue per employee doesn't seem to be rising, which is what I would expect to see out of a successful startup that has found product market fit and is beginning to gain economies of scale. All I see is a capital raise 2x the previous raise each year, with no sign of profit and only a proportional increase in revenue. I can build a business like that: sell Hadoop consulting services at a loss, below the market price. Win lots of business by being really cheap; grow as fast as your cashflow allows, which will require annual investment rounds of ever increasing size. When I look at Google Trends, Hadoop's long-term trajectory looks like "dead by 2024". Do we expect that Databricks will be able to justify a $8B capital raising if Hadoop doesn't really exist? If not, what's the path to profitability? (Not trying to say anything bad about Databricks or their management, I'm just genuinely wondering what's going on.)

dataisamazing 6y ago
For what it’s worth, we work with them and thus have been following closely. Databricks’ value add is that are end to end for interfaces to use your data. From ingest to exploration to DS/ML. 1. Your metrics are wrong about revenue/employee. We’ve heard they’re growing well by all standard SaaS company metrics 2. Hadoop consulting business is just a completely incorrect description of the platform. Hadoop is dying because of companies like Databricks, and all that needs to exist to justify their valuation is people bringing them large data workloads. Their biggest problem is release maturity. Between Azure being down and them being down, stuff is down too often