10 ms·

The end of big data

200 points by swyx 4y ago

nonrandomstring 4y ago
> the end of an over-hyped era Hype is part of what drives technology and I don't see it going away any time soon - because a lot of people make a buck purely by milking hype cycles. I've come to see there are two modes of intelligence. The ability to understand something and say 'yes' to it is complemented by bullshit detectors, a more mature ability to see through other people muddying the waters trying to look smart, and reject their nonsense. Every cycle has its real innovators and hype-pedlars. At the height of blockhain madness a decade ago they had worked themselves into a frenzied fetish of near-supernatural woowooism. Once all the smoke and hullabaloo has cleared, and all the grandiose promises have fallen to the wayside, what remains is a core of genuinely useful and relatively simple core concepts that enter the canon, along with standards and protocols. This line in TFA sums it up: > I needed a database. The same happened with "data". The hype problem there has always been a lack of telos reminiscent of the underpants gnome's flawed "profit?" scheme. Somewhere is a crucial missing step, where we ask "Why?". And the answer is "don't worry, the data plus magical "AI" fairy dust will reveal why". There is a quite religious (faith-like) flavour to this. I blame the NSA. The idea that "collect it all" is anything but a thugish brute force excuse to waste of billions dollars building data-centres, has led to commercial obsession with "data" as a panacea. It isn't "big data" per se that's a problem. Some applications, like in medicine or environmental science, absolutely thrive on large sets and sophisticated analytics. But the fact is that in most applications it's mostly useless, burdensome, energy-consuming, and space-wasting. But "data-hoarding" and over-analysis is pushed by those with sledgehammers to sell for cracking nuts.
- Swizec 4y ago
  > Once all the smoke and hullabaloo has cleared, and all the grandiose promises have fallen to the wayside, what remains is a core of genuinely useful and relatively simple core concepts that enter the canon, along with standards and protocols has this happened with blockchain yet? I'm not seeing it. Maybe I haven't been around the block enough because blockchain is the first hype tech I've seen that keeps not doing anything that existing tech can't do better. And yes I've looked, a lot. The underlying tech sounds super promising and I hope someone figures something out. Not much luck so far.
  - phil21 4y ago
    > has this happened with blockchain yet? I'm not seeing it. No one wants to admit it for some reason, but Bitcoin is the killer app for Blockchain. > The underlying tech sounds super promising and I hope someone figures something out. Not much luck so far. Plenty of luck for me at least. Bitcoin solved exactly the problem it told me it would when it was created - permissionless digital "cash" transactions that cannot be reversed. It's a very specific use case, but one I am quite happy exists when I need it. Those who have never had their money stolen by the banking system or denied the ability to use their money as they see fit likely won't understand my take. Those that do are using the currency today. All the other crud around blockchain is line noise hype. Cryptocurrency already solved exactly the problem it set out to, it's just not easy to get rich off it.
    - na85 4y ago
      >Bitcoin is the killer app for Blockchain If the killer app of the Blockchain is Bitcoin, what does it say about the underlaying technology that the "killer app" is useful only as an extremely volatile speculation vehicle, and an ecologically disastrous one at that, with a tertiary use case of paying for illegal goods and services?
      - correlator 4y ago
        I think OP was saying the ledger mechanics are the useful part here, not the volatile speculation.
        na85 4y ago
        The ledger mechanics have no practical use beyond Bitcoin, which itself useless except for the cases I outlined.
      - ByteJockey 4y ago
        > a tertiary use case of paying for illegal goods and services? This is the use case mentioned. The OP is saying that everything else about it is useless. Having the option to do something illegal should the need arise is a benefit to some people, and government control of the financial sector is not beloved by everyone.
bigcat12345678 4y ago
Can someone summarize what the article want to say? I am not happy about spending 10 minutes on it but realized I need another 10 minutes, and in the end would still be confused, and would need post this comment anyway. So I just stopped there and wrote this comment. Rip writing clarity, and the effective communication...
- rossdavidh 4y ago
  I think it's trying to say that the way to sell the new thing, is first to say that you can do exactly the old thing except better (limitless storage, faster, or whatever). Then, once the customers have the new thing, they may find that they like some of the new features. But don't try to sell them on the new features, because that's not what they're looking for.
- 0x008 4y ago
  Author is trying to say something like: If you want to sell people a tesla, first and foremost, you have to tell them it's a car and it behaves exactly like a car. If that is not a given, they won't buy it, no matter which bells and whistles it has additionally to getting you from A to B. So if we go back to the topic of the article, it means: If you want to sell a big data platform, first you must make people believe it can replace their old traditional database, but make it faster.
- jerglingu 4y ago
  I felt the same reading this. Seems like everything analytics/data is like this these days. You have to read things multiple times over to either translate the abstractions and ball-hiding into plain English, or until you recognize that there is no point and it's mostly stream-of-thought rambling. See especially: anything data mesh.
PLenz 4y ago
Big data didn't lose. It became so common that now what was once big data is just data.
- dehrmann 4y ago
  I think this is the point the article actually makes, but I went into it expecting to see how it's been hard to extract business value from all this data, and then expected it to explain how the demand these easy-to-scale DBs induced resulted in even less value, but it didn't really touch on that.
- zthrowaway 4y ago
  Agreed. The hype around Big Data is dead, but a decent portion of the tech that came into existence during this hype cycle is not dead. Things like Hadoop have their stay in Fortune 150 and 500 companies, there's plenty of Hadoop jobs still out there. It's certainly not sexy anymore though. Hadoop may be losing its foothold though from what I've seen, these days people are getting by just running beefy Linux machines and doing everything in Python using Dask. Most people are not doing petabyte scale data analysis.
mr_toad 4y ago
> But we never connected these dots during our evaluation because the story Databricks told was buzzwords. Snowflake’s was boring, and all we wanted—at least at first—was boring. How do you describe a chocolate gâteau to someone who has only ever eaten rice? > Databricks is a big, fast database that you can write SQL and Python against It really isn’t. Spark is not a database, certainly not an RDBMS. This reminds me of the people in the office who used to call the computer on their desk “the hard drive”. If you’ve never worked with tools like Pandas, or R, or SAS, you’re just not going to understand Spark and Databricks.
- garciasn 4y ago
  As someone who was a SAS developer first and later got into Pandas and R for a short time before going straight BigQuery SQL for the last 5+ years, please explain what you mean here because I don’t follow at all.
  - ACow_Adonis 4y ago
    For the record, he's literally listed off most of my languages: I have years of experience in SAS, R, Python, and SQL, and I also have no idea what he's saying. Also, for the record, that's largely how I think of databricks ideally. A big distributed database/storage layer into which I can write queries in my chosen language... So i...admit equal confusion...
    - adeelk93 4y ago
      As someone who does understand the distinction, for most end-users, I think the distinction doesn’t matter. It’s like the culinary vs botanical categorization of a tomato.
      - Eridrus 4y ago
        The performance of different queries is different across a typical RDBMS and something like Spark/BigQuery. A traditional database is efficient (i.e. cost effective) at doing a lot of the same query repeatedly, e.g. looking up a customer's account balance, whereas these query engines are good at doing infrequent queries with lots of complicated, expensive logic on very large datasets. You could run simple account lookup queries in a CRUD app with Spark, but you'd be setting a lot of compute/money on fire, and your latency would probably be terrible.
zjaffee 4y ago
I've worked in this space in the past, and I think the big weakness of databricks is that for them to be successful as an independent company, they need to charge a lot more than AWS/GCloud/ect, and while historically they had better performance for a lot of Spark jobs, this hasn't been the case for several years now. What Databricks offers that you can't find in essentially the exact same products from other cloud providers, is that they have a pretty UI and a much more integrated setup for more advanced operations like upserts on a data lake instead of having to manage your own Hudi or Iceberg setup. Their notebook is a better offering than others have, but the money for big data companies is in the long running daily or weekly jobs, not one off queries. Historically, databricks had been offering major performance improvements to their customers while not updating the Spark open source code with all of the same query optimizations, and other various major spark performance improvements like Dynamic Pruning and Adaptive Execution, they've since done an about face for the most part and have started pushing these improvements into the open source. But as I noted before, for these longer running daily jobs, you'd spend 10x less per instance hour in some cases running on a cloud providers generic offering rather than paying for databricks, as they've made performance improvements of their own such that the price performance almost universally will always beat databricks. The point is that databricks is absolutely a very innovative company, it's just that it's impossible to compete against the bigger players. The only real major lasting strength of databricks is that Azure fully integrated them into their service as opposed to trying to compete directly like AWS and Gcloud have.
- ramesh31 4y ago
  It all comes down to the price of pushing bits. The big cloud providers have brought that cost so low, that you have to provide an incredible amount of value with a SAAS offering now for it to make financial sense anymore.
  - anonymousDan 4y ago
    In retrospect the Spark authors should have had a licence that forbids cloud providers from running their own hosted services.
0xbadcafebee 4y ago
The author seems to be suggesting that Databricks/Snowflake are selling "storage", "language integration", and "a better Postgres", which is like saying that Toyota sells an air conditioner, entertainment unit, and leather chairs. Technically right... but missing the car for the parts.
nemo44x 4y ago
Databricks didn’t just do 800m revenue - they did it at over 100% growth. Shame they didn’t IPO last year though! But I’m sure when they do they’ll rake it in.
fdgsdfogijq 4y ago
I don't see how databricks doesn't end up like hortonworks, cloudera, or MapR.
- slt2021 4y ago
  fair point, I dont think that AWS/GCP will just let databricks eat their lunch. big cloud providers will try hard to steal market share from DB
  - RhodesianHunter 4y ago
    >I dont think that AWS/GCP will just let databricks eat their lunch. Why not? The data is still being stored in S3/Cloud storage and computed on EC2/GCE so why should Amazon and Google care that they've been relegated to selling the pick-axes into to the gold rush?
  - mr_toad 4y ago
    AWS already has EMR. If they added some of the delta lake features they could do that. Azure seems to be at the ‘embrace’ stage with Databricks. I’m waiting for the ‘extend’ phase, and we know what comes after that.
snidane 4y ago
> Take real-time products, for example. Most businesses have little use for true real-time experiences. But, all else being equal, real-time data is better than latent data. Only if you stare at a chart on a dashboard. If you do serious analytics and collaborate with other people you need the data to be stable and not changing under your hands as you work with it. Daily immutable snapshots produced by a daily batch are not only cheaper to produce but easier to work with and that's why the entire industry prefers batch. The only use case for streaming are 'standing queries' in which query result is continuously updated with new rows incoming. Eg. real time anomaly detection in access logs, ML inferrence, fraud detection, plotting numbers on a dashboard. All analysis is done offline in a batch setup. There is no 'stream analytics'. Stream processing only makes sense if the database is capable of hosting serving layer of the model inferrence application.
- gadflyinyoureye 4y ago
  Even then, there are specific forms of fraud for which streaming detection makes sense. If your upstream data is batch, there is little good reason to implement a streaming approach.
- slt2021 4y ago
  just because most business historically were content with batch, does not mean they will not benefit from streaming/realtime. batch is business as usual, while realtime is disruption. compare to banking: payments historically were batch (ACH), but realtime payments have disrupted it (zelle, venmo, fed wire, etc.)
- fnordpiglet 4y ago
  I think this is super narrow view of the world that applies to a lot of things but not everything. Any data use that involves reactionary decision making will benefit from real time streaming data. Financial services is replete with this, but so are monitoring applications, traffic analytics, advertising, etc. A lot of my career has been turning batch into real time and the commercials were there.
- KptMarchewa 4y ago
  >If you do serious analytics and collaborate with other people you need the data to be stable and not changing under your hands as you work with it. Apache Iceberg and Delta Lake take care of that. The fact that you're reading a particular snapshot of data does not prevent you from adding, updating or deleting data in the meantime. Also, your view is super narrow and focuses only on traditional analytics.
kietay 4y ago
So the article is basically criticising databricks for only being worth 39bn? And then it’s titled “The End of Big Data”… what?
ngcc_hk 4y ago
China
slt2021 4y ago
databricks is not only database. spark is not only for analytics. spark is distributed and fast runtime for applications. it can be a platform powering the entire company/startup and its backend data pipeline. rdbms like postgres cannot compete, because sql is about storage and querying (you need application code to do complex processing) and sql is hard to scale (unless you want to shard your db into million pieces and have ways to manipulate millions of db shards). also - how good luck shuffling data between thousands of rdbms shards, if you believe rdbms can replace spark. the problem with databricks is that their offering is not differentiated enough from opensource spark. So their only value add is "we will manage your managed spark cluster, pls pay us, instead of your Ops/IT team?"
MrPowers 4y ago
The author is suggesting that Databricks should sell a product with infinitely scalable storage and is queryable via SQL or Python... isn't that exactly what they're doing? Databricks has a product called Delta Lake that covers the infinitely scalable storage part. Here's a talk from a Delta Lake user that's writing hundreds of terabyes / up to a petabyte of data a day to Delta Lake: https://youtu.be/vqMuECdmXG0 https://youtu.be/vqMuECdmXG0 Databricks recently rewrote the Spark query engine in C++ (called Photon) and provides a new SQL query interface for data warehouse type developers. Databricks recently set the TPC-DS benchmark record, see this blog post: https://databricks.com/blog/2021/11/02/databricks-sets-official-data-warehousing-performance-record.html https://databricks.com/blog/2021/11/02/databricks-sets-offic... This article doesn't align with my view of reality.
- riku_iki 4y ago
  > Databricks has a product called Delta Lake that covers the infinitely scalable storage part I think it is just API layer on top of existing storage systems like s3 and hdfs: https://docs.delta.io/latest/delta-storage.html#amazon-s3 https://docs.delta.io/latest/delta-storage.html#amazon-s3
  - alexott 4y ago
    No, it’s not just API layer, but significant addon on top of existing tech… It brings transactions, faster queries, time travel, etc.
    - riku_iki 4y ago
      Maybe, but "infinitely scalable storage" part is because of lower lever layer.
      - Epa095 4y ago
        If course, but there are many ways one could make the storage format such that it would not scale nicely with the underlying layer.
chevman 4y ago
Living all day every day in enterprise land, Databricks current growth has been set on fire by their integration into Azure. All the bigCo's moving their legacy infra to Azure and finding out Databricks is available OOTB is fueling huge growth. I have multiple friends from Oracle, IBM, Salesforce, Adobe, etc that are now AEs at Databricks making 3 or 4x total comp they were making previously. Fun times!
baazaa 4y ago
The author is exactly right. Indeed I work for government and we're buying into databricks and the people responsible for that don't even know what Spark is. The pitch which sold them the product is the one the author basically identified, infinitely scalable storage that is queryable, even though there are plenty of other products that offer that. While government is an extreme example, I'm sure there's a lot of non-tech companies in a similar position. The people doing the procurement don't know anything about tech, but they do know that there's random databases all over the place that are a constant headache and they want to centralise everything in a single platform. Secondly they steadfastly refuse to hire competent tech people, if databricks ends up being a very expensive replacement for an OPs team then that's a big winner for any organisation that is unable to, or refuses to, hire an OPs team but has lots of money to burn.
dumbfounder 4y ago
We evaluated databricks a few years ago. Didn’t get how we start and do the simplest thing. For our company that didn’t know how to take the first step it was daunting. Then snowflake came along and it was so easy and it just worked and we understood it and could see how it could be extended. Maybe databricks is more powerful, I have no idea, it was going to be way too much effort to find out. It was a nonstarter for us.
- alexott 4y ago
  Maybe you can look again - things are changing very fast, few years there were mo DBSQL, Photon, Delta Sharing, etc.
  - dumbfounder 4y ago
    I am out of that game, but adding a bunch of shiny new stuff does not mean it is as simple as Snowflake, it usually means the opposite. With Snowflake the onboarding was so clean and then you land can expand. Databricks was a massive suite of complicated stuff and you don't know where to start. Have they reoriented to fix this?
    - alexott 4y ago
      It became better
non_sequitur 4y ago
Random observation as this quote stood out: "Snowflake is worth $70 billion, even after its stock’s brutal five-month slide" The article was written April 8, so ~6 weeks ago. As of May 26, Snowflake is worth $39B. Crazy how fast things can change.
- swyx 4y ago
  if you liked it at 70, you should LOVE it at 39 :) unfortunately their latest earnings showed a significant decline in revenue growth as well down from the often cited 100%yoy number down to 80%
antipaul 4y ago
It’s not really the end of “big data”. There is and will be tons of data to go around for the foreseeable future. Instead, it feels more like the end of “data science as the sexiest job of the 21st century”. DS is mostly running reports (so BI) or stats (perennially important niche) or manufacturing hype via “latest research”. Engineering has been, is, and will continue to be the foundation for success.
- ElemenoPicuares 4y ago
  Engineers usually think so. After spending some time with GitHub Copilot, as annoying as it is, I think we've got a decade, max, before the lower 3/4 of the software development trade is rendered unnecessary.
  - ElemenoPicuares 4y ago
    Downvote all you want but prove me wrong.
    - CRConrad 4y ago
      Hacker News doesn't work that way: We won't be able to reply to your comment in ten years when we know how it went.
      - ElemenoPicuares 4y ago
        I was using the word prove colloquially. That many professions are being automated is a given, and with NN people are trying extra hard to automate knowledge work that requires more trade-like knowledge than theoretical knowledge, like paraprofessionals and functionaries with domain knowledge. Recent developments in software lead me and many others to believe the automation of utilitarian development jobs is impending. Even Forbes thinks so and I guarantee you they hadn't seen GitHub Copilot when this article came out: https://www.forbes.com/sites/forbestechcouncil/2021/02/23/11-jobs-that-may-be-automated-in-the-next-decade/?sh=7d6869d8972c https://www.forbes.com/sites/forbestechcouncil/2021/02/23/11... So, given that enough industry analysts believe development will be heavily automated to make top-10 lists, and recent developments provide even stronger evidence (copilot writes better comments on existing code than my coworkers,) can someone providence a strong a priori argument or maybe even empirical evidence that utilitarian software development is somehow uniquely immune to this?
asasidh 4y ago
Death of the narrative that SiliconValley pushed to regular companies and made them believe that they had a big data problem. Mostly the companies that invented the solution had the problems. 99.99% of others didn’t.
datalopers 4y ago
Databricks/Spark is overpriced garbage for Python-toting data scientists/engineers/analysts who don't know how to model data or write SQL. Exactly like Hadoop was 15 years ago for the Java-toting data scientists/engineers.
- clpm4j 4y ago
  That might be hyperbolic, but directionally I agree. The interesting part is that there are enough of those people out there in enterprise who fall into that "don't really know what they're doing but want to try something cool" bucket to keep Databricks flush with cash for years on end.
  - datalopers 4y ago
    Yep. MongoDB earned $873M in revenue over the past 12 months servicing tech debt created between ‘10 - ‘16. Databricks will be around a long time for the same reason. It’s a viable business strategy for sure.
  - michael_j_ward 4y ago
    If one does not need petabytes of scale, but am otherwise interested in the data lineage / observability / workflow being sold by databricks, what would you suggest? Some evaluation Criteria: - Ease of maintenance and operation is almost paramount. - It's fine if the solution never lives anywhere but 1 single virtual server that scales vertically (data might grow to a couple TB, but not PETA BYTES) - Similarly, 20 9's is not a criteria. If the machine fails and it takes an hour till someone goes and re-deploys, that's fine. - Declarative, reproducible deployment with an easy upgrade story would be great - Ideal if the deployment can be run locally for quick developmnet
    - nooorofe 4y ago
      Talend Studio? https://www.talend.com/products/talend-open-studio/ https://www.talend.com/products/talend-open-studio/
- alexott 4y ago
spicyramen_ 4y ago
whoevercares 4y ago
Author has been using a great, modern data stack from the beginning of this article - this is so not realistic for most Enterprises which still live in caves
la6472 4y ago
What about unstructured data or is the realization dawning that there cannot be any data that is unstructured?
clpm4j 4y ago
Eventually the majority realizes that 'big data' doesn't apply to them. It just doesn't work for what they need to do. I think it follows that the same reasoning applies to ML - works great for a few, but not at all for most.
mborch 4y ago
The challenge for Databricks and Snowflake is that big cloud vendors are catching up with their own serverless products such as Redshift Serverless [0]. And they now provide Spark engines as well with associated notebooks. What's the unique offering in Databricks seen in this light? [0] https://aws.amazon.com/redshift/redshift-serverless/ https://aws.amazon.com/redshift/redshift-serverless/
unixhero 4y ago
Good writeup, but with a weak conclusion.
unixhero 4y ago
So Databricks is a hosted Apache spark? What is their data layer then, is it postures?
- hansonw 4y ago
  a bunch of Parquet files in S3 ;)
  - KptMarchewa 4y ago
    That's very reductionist, but yes. That, and metadata json files are delta lake.
itechhere 4y ago
itechhere 4y ago
[flagged]
Lucasoato 4y ago
I love Databricks, we can consider it as the Data Engineer's heaven. Everything makes sense, works smoothly and delta tables are truly revolutionary. Now that they are open sourcing even Delta table optimizations they will really become the n1 solution as a Data Engineering platform. The only downside is their marketing team. Sadly it's full of passive/aggressive people that won't let your company's engineers work because they'll bury you with hundreds of business questions just to resell a cool story internally. I've seen projects and deals going down for this. I really can't understand why they're trying so desperately to waste their customer's time.
- benjaminwootton 4y ago
  Yep, they absolutely nail the product. One platform for ETL, analytics, data scrience etc in your preferred language. Nice Notebook interface etc. I’m not sure why anyone would try to roll their own analytics stack.
- symfrog 4y ago
  > Now that they are open sourcing even Delta table optimizations Has Databricks recently open sourced additional Delta table features that were previously only available with a paid license? I can't seem to find a relevant announcement.
  - alexott 4y ago
    https://delta.io/blog/2022-05-05-delta-lake-1-2-released/ https://delta.io/blog/2022-05-05-delta-lake-1-2-released/ - big part is open sourced and more is coming (see roadmap on GitHub: https://github.com/delta-io/delta/issues/920 https://github.com/delta-io/delta/issues/920)
tomkwong 4y ago
100% click bait title and opening statement. Is journalism that bad these days? Ironically I quite like some of the points in the main content.
- MafellUser 4y ago
  substack is basically a blogging platform. Are bloggers journalists? Are cats actually dogs?
mywaifuismeta 4y ago
No idea what the author is trying to say. It's just a clickbait title with a rant and no content. "The end of big data" but then goes on to rant about how they bought "big data" products and ends with "Big Data is finally starting to live up to its potential" - this is below Medium quality.
pea 4y ago
I wonder if the quality of other data processing frameworks is also getting 'good enough' for many use-cases, and they often have less complexity than Spark. With Hadoop, many companies bought clusters because they thought they needed them, but they rotted because people didn't actually have suitable workloads. Does Databricks have a similar risk? Just based on users of our Python reporting framework[0] we are starting to see Dask/Ray displacing some workloads that would have been in Spark a few years ago. How quickly things progress in that space (e.g. Arrow/DuckDB[1]) is pretty incredible to watch. [0] https://github.com/datapane/datapane https://github.com/datapane/datapane [1] https://duckdb.org/ https://duckdb.org/

200 points by swyx 4y ago

> the end of an over-hyped era Hype is part of what drives technology and I don't see it going away any time soon - because a lot of people make a buck purely by milking hype cycles. I've come to see there are two modes of intelligence. The ability to understand something and say 'yes' to it is complemented by bullshit detectors, a more mature ability to see through other people muddying the waters trying to look smart, and reject their nonsense. Every cycle has its real innovators and hype-pedlars. At the height of blockhain madness a decade ago they had worked themselves into a frenzied fetish of near-supernatural woowooism. Once all the smoke and hullabaloo has cleared, and all the grandiose promises have fallen to the wayside, what remains is a core of genuinely useful and relatively simple core concepts that enter the canon, along with standards and protocols. This line in TFA sums it up: > I needed a database. The same happened with "data". The hype problem there has always been a lack of telos reminiscent of the underpants gnome's flawed "profit?" scheme. Somewhere is a crucial missing step, where we ask "Why?". And the answer is "don't worry, the data plus magical "AI" fairy dust will reveal why". There is a quite religious (faith-like) flavour to this. I blame the NSA. The idea that "collect it all" is anything but a thugish brute force excuse to waste of billions dollars building data-centres, has led to commercial obsession with "data" as a panacea. It isn't "big data" per se that's a problem. Some applications, like in medicine or environmental science, absolutely thrive on large sets and sophisticated analytics. But the fact is that in most applications it's mostly useless, burdensome, energy-consuming, and space-wasting. But "data-hoarding" and over-analysis is pushed by those with sledgehammers to sell for cracking nuts.

Swizec 4y ago
> Once all the smoke and hullabaloo has cleared, and all the grandiose promises have fallen to the wayside, what remains is a core of genuinely useful and relatively simple core concepts that enter the canon, along with standards and protocols has this happened with blockchain yet? I'm not seeing it. Maybe I haven't been around the block enough because blockchain is the first hype tech I've seen that keeps not doing anything that existing tech can't do better. And yes I've looked, a lot. The underlying tech sounds super promising and I hope someone figures something out. Not much luck so far.
- phil21 4y ago
  > has this happened with blockchain yet? I'm not seeing it. No one wants to admit it for some reason, but Bitcoin is the killer app for Blockchain. > The underlying tech sounds super promising and I hope someone figures something out. Not much luck so far. Plenty of luck for me at least. Bitcoin solved exactly the problem it told me it would when it was created - permissionless digital "cash" transactions that cannot be reversed. It's a very specific use case, but one I am quite happy exists when I need it. Those who have never had their money stolen by the banking system or denied the ability to use their money as they see fit likely won't understand my take. Those that do are using the currency today. All the other crud around blockchain is line noise hype. Cryptocurrency already solved exactly the problem it set out to, it's just not easy to get rich off it.
  - na85 4y ago
    >Bitcoin is the killer app for Blockchain If the killer app of the Blockchain is Bitcoin, what does it say about the underlaying technology that the "killer app" is useful only as an extremely volatile speculation vehicle, and an ecologically disastrous one at that, with a tertiary use case of paying for illegal goods and services?
    - correlator 4y ago
      I think OP was saying the ledger mechanics are the useful part here, not the volatile speculation.
      - na85 4y ago
        The ledger mechanics have no practical use beyond Bitcoin, which itself useless except for the cases I outlined.
    - ByteJockey 4y ago
      > a tertiary use case of paying for illegal goods and services? This is the use case mentioned. The OP is saying that everything else about it is useless. Having the option to do something illegal should the need arise is a benefit to some people, and government control of the financial sector is not beloved by everyone.

bigcat12345678 4y ago

Can someone summarize what the article want to say? I am not happy about spending 10 minutes on it but realized I need another 10 minutes, and in the end would still be confused, and would need post this comment anyway. So I just stopped there and wrote this comment. Rip writing clarity, and the effective communication...

rossdavidh 4y ago
I think it's trying to say that the way to sell the new thing, is first to say that you can do exactly the old thing except better (limitless storage, faster, or whatever). Then, once the customers have the new thing, they may find that they like some of the new features. But don't try to sell them on the new features, because that's not what they're looking for.
0x008 4y ago
Author is trying to say something like: If you want to sell people a tesla, first and foremost, you have to tell them it's a car and it behaves exactly like a car. If that is not a given, they won't buy it, no matter which bells and whistles it has additionally to getting you from A to B. So if we go back to the topic of the article, it means: If you want to sell a big data platform, first you must make people believe it can replace their old traditional database, but make it faster.
jerglingu 4y ago
I felt the same reading this. Seems like everything analytics/data is like this these days. You have to read things multiple times over to either translate the abstractions and ball-hiding into plain English, or until you recognize that there is no point and it's mostly stream-of-thought rambling. See especially: anything data mesh.

PLenz 4y ago

Big data didn't lose. It became so common that now what was once big data is just data.

dehrmann 4y ago
I think this is the point the article actually makes, but I went into it expecting to see how it's been hard to extract business value from all this data, and then expected it to explain how the demand these easy-to-scale DBs induced resulted in even less value, but it didn't really touch on that.
zthrowaway 4y ago
Agreed. The hype around Big Data is dead, but a decent portion of the tech that came into existence during this hype cycle is not dead. Things like Hadoop have their stay in Fortune 150 and 500 companies, there's plenty of Hadoop jobs still out there. It's certainly not sexy anymore though. Hadoop may be losing its foothold though from what I've seen, these days people are getting by just running beefy Linux machines and doing everything in Python using Dask. Most people are not doing petabyte scale data analysis.

mr_toad 4y ago

> But we never connected these dots during our evaluation because the story Databricks told was buzzwords. Snowflake’s was boring, and all we wanted—at least at first—was boring. How do you describe a chocolate gâteau to someone who has only ever eaten rice? > Databricks is a big, fast database that you can write SQL and Python against It really isn’t. Spark is not a database, certainly not an RDBMS. This reminds me of the people in the office who used to call the computer on their desk “the hard drive”. If you’ve never worked with tools like Pandas, or R, or SAS, you’re just not going to understand Spark and Databricks.

garciasn 4y ago
As someone who was a SAS developer first and later got into Pandas and R for a short time before going straight BigQuery SQL for the last 5+ years, please explain what you mean here because I don’t follow at all.
- ACow_Adonis 4y ago
  For the record, he's literally listed off most of my languages: I have years of experience in SAS, R, Python, and SQL, and I also have no idea what he's saying. Also, for the record, that's largely how I think of databricks ideally. A big distributed database/storage layer into which I can write queries in my chosen language... So i...admit equal confusion...
  - adeelk93 4y ago
    As someone who does understand the distinction, for most end-users, I think the distinction doesn’t matter. It’s like the culinary vs botanical categorization of a tomato.
    - Eridrus 4y ago
      The performance of different queries is different across a typical RDBMS and something like Spark/BigQuery. A traditional database is efficient (i.e. cost effective) at doing a lot of the same query repeatedly, e.g. looking up a customer's account balance, whereas these query engines are good at doing infrequent queries with lots of complicated, expensive logic on very large datasets. You could run simple account lookup queries in a CRUD app with Spark, but you'd be setting a lot of compute/money on fire, and your latency would probably be terrible.

zjaffee 4y ago

I've worked in this space in the past, and I think the big weakness of databricks is that for them to be successful as an independent company, they need to charge a lot more than AWS/GCloud/ect, and while historically they had better performance for a lot of Spark jobs, this hasn't been the case for several years now. What Databricks offers that you can't find in essentially the exact same products from other cloud providers, is that they have a pretty UI and a much more integrated setup for more advanced operations like upserts on a data lake instead of having to manage your own Hudi or Iceberg setup. Their notebook is a better offering than others have, but the money for big data companies is in the long running daily or weekly jobs, not one off queries. Historically, databricks had been offering major performance improvements to their customers while not updating the Spark open source code with all of the same query optimizations, and other various major spark performance improvements like Dynamic Pruning and Adaptive Execution, they've since done an about face for the most part and have started pushing these improvements into the open source. But as I noted before, for these longer running daily jobs, you'd spend 10x less per instance hour in some cases running on a cloud providers generic offering rather than paying for databricks, as they've made performance improvements of their own such that the price performance almost universally will always beat databricks. The point is that databricks is absolutely a very innovative company, it's just that it's impossible to compete against the bigger players. The only real major lasting strength of databricks is that Azure fully integrated them into their service as opposed to trying to compete directly like AWS and Gcloud have.

ramesh31 4y ago
It all comes down to the price of pushing bits. The big cloud providers have brought that cost so low, that you have to provide an incredible amount of value with a SAAS offering now for it to make financial sense anymore.
- anonymousDan 4y ago
  In retrospect the Spark authors should have had a licence that forbids cloud providers from running their own hosted services.

0xbadcafebee 4y ago

The author seems to be suggesting that Databricks/Snowflake are selling "storage", "language integration", and "a better Postgres", which is like saying that Toyota sells an air conditioner, entertainment unit, and leather chairs. Technically right... but missing the car for the parts.

nemo44x 4y ago

Databricks didn’t just do 800m revenue - they did it at over 100% growth. Shame they didn’t IPO last year though! But I’m sure when they do they’ll rake it in.

fdgsdfogijq 4y ago

I don't see how databricks doesn't end up like hortonworks, cloudera, or MapR.

slt2021 4y ago
fair point, I dont think that AWS/GCP will just let databricks eat their lunch. big cloud providers will try hard to steal market share from DB
- RhodesianHunter 4y ago
  >I dont think that AWS/GCP will just let databricks eat their lunch. Why not? The data is still being stored in S3/Cloud storage and computed on EC2/GCE so why should Amazon and Google care that they've been relegated to selling the pick-axes into to the gold rush?
- mr_toad 4y ago
  AWS already has EMR. If they added some of the delta lake features they could do that. Azure seems to be at the ‘embrace’ stage with Databricks. I’m waiting for the ‘extend’ phase, and we know what comes after that.

snidane 4y ago

> Take real-time products, for example. Most businesses have little use for true real-time experiences. But, all else being equal, real-time data is better than latent data. Only if you stare at a chart on a dashboard. If you do serious analytics and collaborate with other people you need the data to be stable and not changing under your hands as you work with it. Daily immutable snapshots produced by a daily batch are not only cheaper to produce but easier to work with and that's why the entire industry prefers batch. The only use case for streaming are 'standing queries' in which query result is continuously updated with new rows incoming. Eg. real time anomaly detection in access logs, ML inferrence, fraud detection, plotting numbers on a dashboard. All analysis is done offline in a batch setup. There is no 'stream analytics'. Stream processing only makes sense if the database is capable of hosting serving layer of the model inferrence application.

gadflyinyoureye 4y ago
Even then, there are specific forms of fraud for which streaming detection makes sense. If your upstream data is batch, there is little good reason to implement a streaming approach.
slt2021 4y ago
just because most business historically were content with batch, does not mean they will not benefit from streaming/realtime. batch is business as usual, while realtime is disruption. compare to banking: payments historically were batch (ACH), but realtime payments have disrupted it (zelle, venmo, fed wire, etc.)
fnordpiglet 4y ago
I think this is super narrow view of the world that applies to a lot of things but not everything. Any data use that involves reactionary decision making will benefit from real time streaming data. Financial services is replete with this, but so are monitoring applications, traffic analytics, advertising, etc. A lot of my career has been turning batch into real time and the commercials were there.
KptMarchewa 4y ago
>If you do serious analytics and collaborate with other people you need the data to be stable and not changing under your hands as you work with it. Apache Iceberg and Delta Lake take care of that. The fact that you're reading a particular snapshot of data does not prevent you from adding, updating or deleting data in the meantime. Also, your view is super narrow and focuses only on traditional analytics.

kietay 4y ago

So the article is basically criticising databricks for only being worth 39bn? And then it’s titled “The End of Big Data”… what?

ngcc_hk 4y ago

China

slt2021 4y ago

databricks is not only database. spark is not only for analytics. spark is distributed and fast runtime for applications. it can be a platform powering the entire company/startup and its backend data pipeline. rdbms like postgres cannot compete, because sql is about storage and querying (you need application code to do complex processing) and sql is hard to scale (unless you want to shard your db into million pieces and have ways to manipulate millions of db shards). also - how good luck shuffling data between thousands of rdbms shards, if you believe rdbms can replace spark. the problem with databricks is that their offering is not differentiated enough from opensource spark. So their only value add is "we will manage your managed spark cluster, pls pay us, instead of your Ops/IT team?"

MrPowers 4y ago

The author is suggesting that Databricks should sell a product with infinitely scalable storage and is queryable via SQL or Python... isn't that exactly what they're doing? Databricks has a product called Delta Lake that covers the infinitely scalable storage part. Here's a talk from a Delta Lake user that's writing hundreds of terabyes / up to a petabyte of data a day to Delta Lake: https://youtu.be/vqMuECdmXG0 https://youtu.be/vqMuECdmXG0 Databricks recently rewrote the Spark query engine in C++ (called Photon) and provides a new SQL query interface for data warehouse type developers. Databricks recently set the TPC-DS benchmark record, see this blog post: https://databricks.com/blog/2021/11/02/databricks-sets-official-data-warehousing-performance-record.html https://databricks.com/blog/2021/11/02/databricks-sets-offic... This article doesn't align with my view of reality.

riku_iki 4y ago
> Databricks has a product called Delta Lake that covers the infinitely scalable storage part I think it is just API layer on top of existing storage systems like s3 and hdfs: https://docs.delta.io/latest/delta-storage.html#amazon-s3 https://docs.delta.io/latest/delta-storage.html#amazon-s3
- alexott 4y ago
  No, it’s not just API layer, but significant addon on top of existing tech… It brings transactions, faster queries, time travel, etc.
  - riku_iki 4y ago
    Maybe, but "infinitely scalable storage" part is because of lower lever layer.
    - Epa095 4y ago
      If course, but there are many ways one could make the storage format such that it would not scale nicely with the underlying layer.

chevman 4y ago

Living all day every day in enterprise land, Databricks current growth has been set on fire by their integration into Azure. All the bigCo's moving their legacy infra to Azure and finding out Databricks is available OOTB is fueling huge growth. I have multiple friends from Oracle, IBM, Salesforce, Adobe, etc that are now AEs at Databricks making 3 or 4x total comp they were making previously. Fun times!

baazaa 4y ago

The author is exactly right. Indeed I work for government and we're buying into databricks and the people responsible for that don't even know what Spark is. The pitch which sold them the product is the one the author basically identified, infinitely scalable storage that is queryable, even though there are plenty of other products that offer that. While government is an extreme example, I'm sure there's a lot of non-tech companies in a similar position. The people doing the procurement don't know anything about tech, but they do know that there's random databases all over the place that are a constant headache and they want to centralise everything in a single platform. Secondly they steadfastly refuse to hire competent tech people, if databricks ends up being a very expensive replacement for an OPs team then that's a big winner for any organisation that is unable to, or refuses to, hire an OPs team but has lots of money to burn.

dumbfounder 4y ago

We evaluated databricks a few years ago. Didn’t get how we start and do the simplest thing. For our company that didn’t know how to take the first step it was daunting. Then snowflake came along and it was so easy and it just worked and we understood it and could see how it could be extended. Maybe databricks is more powerful, I have no idea, it was going to be way too much effort to find out. It was a nonstarter for us.

alexott 4y ago
Maybe you can look again - things are changing very fast, few years there were mo DBSQL, Photon, Delta Sharing, etc.
- dumbfounder 4y ago
  I am out of that game, but adding a bunch of shiny new stuff does not mean it is as simple as Snowflake, it usually means the opposite. With Snowflake the onboarding was so clean and then you land can expand. Databricks was a massive suite of complicated stuff and you don't know where to start. Have they reoriented to fix this?
  - alexott 4y ago
    It became better

non_sequitur 4y ago

Random observation as this quote stood out: "Snowflake is worth $70 billion, even after its stock’s brutal five-month slide" The article was written April 8, so ~6 weeks ago. As of May 26, Snowflake is worth $39B. Crazy how fast things can change.

swyx 4y ago
if you liked it at 70, you should LOVE it at 39 :) unfortunately their latest earnings showed a significant decline in revenue growth as well down from the often cited 100%yoy number down to 80%

antipaul 4y ago

It’s not really the end of “big data”. There is and will be tons of data to go around for the foreseeable future. Instead, it feels more like the end of “data science as the sexiest job of the 21st century”. DS is mostly running reports (so BI) or stats (perennially important niche) or manufacturing hype via “latest research”. Engineering has been, is, and will continue to be the foundation for success.

ElemenoPicuares 4y ago
Engineers usually think so. After spending some time with GitHub Copilot, as annoying as it is, I think we've got a decade, max, before the lower 3/4 of the software development trade is rendered unnecessary.
- ElemenoPicuares 4y ago
  Downvote all you want but prove me wrong.
  - CRConrad 4y ago
    Hacker News doesn't work that way: We won't be able to reply to your comment in ten years when we know how it went.
    - ElemenoPicuares 4y ago
      I was using the word prove colloquially. That many professions are being automated is a given, and with NN people are trying extra hard to automate knowledge work that requires more trade-like knowledge than theoretical knowledge, like paraprofessionals and functionaries with domain knowledge. Recent developments in software lead me and many others to believe the automation of utilitarian development jobs is impending. Even Forbes thinks so and I guarantee you they hadn't seen GitHub Copilot when this article came out: https://www.forbes.com/sites/forbestechcouncil/2021/02/23/11-jobs-that-may-be-automated-in-the-next-decade/?sh=7d6869d8972c https://www.forbes.com/sites/forbestechcouncil/2021/02/23/11... So, given that enough industry analysts believe development will be heavily automated to make top-10 lists, and recent developments provide even stronger evidence (copilot writes better comments on existing code than my coworkers,) can someone providence a strong a priori argument or maybe even empirical evidence that utilitarian software development is somehow uniquely immune to this?

asasidh 4y ago

Death of the narrative that SiliconValley pushed to regular companies and made them believe that they had a big data problem. Mostly the companies that invented the solution had the problems. 99.99% of others didn’t.

datalopers 4y ago

Databricks/Spark is overpriced garbage for Python-toting data scientists/engineers/analysts who don't know how to model data or write SQL. Exactly like Hadoop was 15 years ago for the Java-toting data scientists/engineers.

clpm4j 4y ago
That might be hyperbolic, but directionally I agree. The interesting part is that there are enough of those people out there in enterprise who fall into that "don't really know what they're doing but want to try something cool" bucket to keep Databricks flush with cash for years on end.
- datalopers 4y ago
  Yep. MongoDB earned $873M in revenue over the past 12 months servicing tech debt created between ‘10 - ‘16. Databricks will be around a long time for the same reason. It’s a viable business strategy for sure.
- michael_j_ward 4y ago
  If one does not need petabytes of scale, but am otherwise interested in the data lineage / observability / workflow being sold by databricks, what would you suggest? Some evaluation Criteria: - Ease of maintenance and operation is almost paramount. - It's fine if the solution never lives anywhere but 1 single virtual server that scales vertically (data might grow to a couple TB, but not PETA BYTES) - Similarly, 20 9's is not a criteria. If the machine fails and it takes an hour till someone goes and re-deploys, that's fine. - Declarative, reproducible deployment with an easy upgrade story would be great - Ideal if the deployment can be run locally for quick developmnet
  - nooorofe 4y ago
    Talend Studio? https://www.talend.com/products/talend-open-studio/ https://www.talend.com/products/talend-open-studio/
alexott 4y ago

spicyramen_ 4y ago

whoevercares 4y ago

Author has been using a great, modern data stack from the beginning of this article - this is so not realistic for most Enterprises which still live in caves

la6472 4y ago

What about unstructured data or is the realization dawning that there cannot be any data that is unstructured?

clpm4j 4y ago

Eventually the majority realizes that 'big data' doesn't apply to them. It just doesn't work for what they need to do. I think it follows that the same reasoning applies to ML - works great for a few, but not at all for most.

mborch 4y ago

The challenge for Databricks and Snowflake is that big cloud vendors are catching up with their own serverless products such as Redshift Serverless [0]. And they now provide Spark engines as well with associated notebooks. What's the unique offering in Databricks seen in this light? [0] https://aws.amazon.com/redshift/redshift-serverless/ https://aws.amazon.com/redshift/redshift-serverless/

unixhero 4y ago

Good writeup, but with a weak conclusion.

unixhero 4y ago

So Databricks is a hosted Apache spark? What is their data layer then, is it postures?

hansonw 4y ago
a bunch of Parquet files in S3 ;)
- KptMarchewa 4y ago
  That's very reductionist, but yes. That, and metadata json files are delta lake.

itechhere 4y ago

[flagged]

Lucasoato 4y ago

I love Databricks, we can consider it as the Data Engineer's heaven. Everything makes sense, works smoothly and delta tables are truly revolutionary. Now that they are open sourcing even Delta table optimizations they will really become the n1 solution as a Data Engineering platform. The only downside is their marketing team. Sadly it's full of passive/aggressive people that won't let your company's engineers work because they'll bury you with hundreds of business questions just to resell a cool story internally. I've seen projects and deals going down for this. I really can't understand why they're trying so desperately to waste their customer's time.

benjaminwootton 4y ago
Yep, they absolutely nail the product. One platform for ETL, analytics, data scrience etc in your preferred language. Nice Notebook interface etc. I’m not sure why anyone would try to roll their own analytics stack.
symfrog 4y ago
> Now that they are open sourcing even Delta table optimizations Has Databricks recently open sourced additional Delta table features that were previously only available with a paid license? I can't seem to find a relevant announcement.
- alexott 4y ago
  https://delta.io/blog/2022-05-05-delta-lake-1-2-released/ https://delta.io/blog/2022-05-05-delta-lake-1-2-released/ - big part is open sourced and more is coming (see roadmap on GitHub: https://github.com/delta-io/delta/issues/920 https://github.com/delta-io/delta/issues/920)

tomkwong 4y ago

100% click bait title and opening statement. Is journalism that bad these days? Ironically I quite like some of the points in the main content.

MafellUser 4y ago
substack is basically a blogging platform. Are bloggers journalists? Are cats actually dogs?

mywaifuismeta 4y ago

No idea what the author is trying to say. It's just a clickbait title with a rant and no content. "The end of big data" but then goes on to rant about how they bought "big data" products and ends with "Big Data is finally starting to live up to its potential" - this is below Medium quality.

pea 4y ago

I wonder if the quality of other data processing frameworks is also getting 'good enough' for many use-cases, and they often have less complexity than Spark. With Hadoop, many companies bought clusters because they thought they needed them, but they rotted because people didn't actually have suitable workloads. Does Databricks have a similar risk? Just based on users of our Python reporting framework[0] we are starting to see Dask/Ray displacing some workloads that would have been in Spark a few years ago. How quickly things progress in that space (e.g. Arrow/DuckDB[1]) is pretty incredible to watch. [0] https://github.com/datapane/datapane https://github.com/datapane/datapane [1] https://duckdb.org/ https://duckdb.org/