23 ms·

Big data is dead (2023)

585 points by armanke13 2y ago

clkao 2y ago
previous discussion: https://news.ycombinator.com/item?id=34694926 https://news.ycombinator.com/item?id=34694926
- spicyusername 2y ago
  I find it interesting that this comment section and that comment section seem to focus on different things, despite being triggered by the same input.
ahartmetz 2y ago
I guess that hype cycle ended at the plateau of being dead. A not uncommon outcome in this incredibly fashion-driven industry.
- silvestrov 2y ago
  It has just been rebranded as AI. AI also use all the data, just with a magick neural network to figure out what it all means.
  - quonn 2y ago
    The overlap in terms of the used technologies, the required skills, the actual products and the target market is minimal. AI is not mostly Hadoop, it's not MapReduce, the hardware is different, the software is different, the skillset is very different and a chatbot or image generator is very different from a batch job producing an answer to a query.
    - renegade-otter 2y ago
      But the underlying problem is the same - companies that use Big Data tech are clueless about data management. You can use unicorns - it's not going to do anything. "Garbage in, garbage out" is a timeless principle.
  - WesolyKubeczek 2y ago
    Given how often it hallucinates, it should be rebranded to "high data".
  - Cyberdog 2y ago
    Assuming you're serious for a moment, I don't think AI is really a practical tool for working with big data. - The "hallucination" factor means every result an AI tells you about big data is suspect. I'm sure some of you who really understand AI more than the average person can "um akshually" me on this and tell me how it's possible to configure ChatGPT to absolutely be honest 100% of the time but given the current state of what I've seen from general-purpose AI tools, I just can't trust it. In many ways this is worse than MongoDB just dropping data since at least Mongo won't make up conclusions about data that's not there. - At the end of the day - and I think we're going to be seeing this happen a lot in the future with other workflows as well - you're using this heavy, low-performance general-purpose tool to solve a problem which can be solved much more performatively by using tools which have been designed from the beginning to handle data management and analysis. The reason traditional SQL RDBMSes have endured and aren't going anywhere soon is partially because they've proven to be a very good compromise between general functionality and performance for the task of managing various types of data. AI is nowhere near as good of a balance for this task in almost all cases. All that being said, the same way Electron has proven to be a popular tool for writing widely-used desktop and mobile applications, performance and UI concerns be damned all the way to hell, I'm sure we'll be seeing AI-powered "big data" analysis tools very soon if they're not out there already, and they will suck but people will use them anyway to everyone's detriment.
    - silvestrov 2y ago
      A comment from the old post: https://news.ycombinator.com/item?id=34696065 https://news.ycombinator.com/item?id=34696065 > I used to joke that Data Scientists exist not to uncover insights or provide analysis, but merely to provide factoids that confirm senior management's prior beliefs. I think AI is used for the same purpose in companies: signal to the world that the company is using the latest tech and internally for supporting existing political beliefs. So same job. Hallucination is not a problem here as the AI conclusions are not used when they don't align to existing political beliefs.
    - vitus 2y ago
      > The "hallucination" factor means every result an AI tells you about big data is suspect. AI / ML means more than just LLM chat output, even if that's the current hype cycle of the last couple of years. ML can be used to build a perfectly serviceable classifier, or predictor, or outlier detector. It suffers from the lack of explainability that's always plagued AI / ML, especially as you start looking at deeper neural networks where you're more and more heavily reliant on their ability to approximate arbitrary functions as you add more layers. > you're using this heavy, low-performance general-purpose tool to solve a problem which can be solved much more performatively by using tools which have been designed from the beginning to handle data management and analysis You are not wrong here, but one challenge is that sometimes even your domain experts do not know how to solve the problem, and applying traditional statistical methods without understanding the space is a great way of identifying spurious correlations. (To be fair, this applies in equal measure to ML methods.)
blagie 2y ago
Overall, I agree with much of this post, but there are several caveats: 1) Mongo is a bad point of reference. The one lesson I've learned is that there is nothing Mongo does which postgresql doesn't do better. Big data solutions aren't nosql / mongo, but usually things like columnar databases, map/reduce, Cassandra, etc. 2) Plan for success 95% of businesses never become unicorns, but that's the goal for most (for the 5% which do). If you don't plan for it, you won't make it. The reason to architect for scalability when you have 5 customers is so if that exponential growth cycle hits, you can capitalize on it. That's not just architecture. To have any chance of becoming a unicorn, every part of the business needs to be planned for now and for later: How do we make this practical / sustainable today? How do we make sure it can grow later when we have millions of customers? A lot of this can be left as scaffolding (we'll swap in [X], but for now, we'll do [Y]). But the key lessons are correct: - Most data isn't big. I can fit data about every person in the world on a $100 Chromebook. (8 billion people * 8 bits of data = 8GB) - Most data is rarely queried, and most queries are tiny. The first step in most big data jobs I've done is taking terabytes of data and shrinking it down to the GB, MB, or oven KB-scale data I need. Caveat: I have no algorithm for predicting what I'll need in the future. - Cost of data is increasing with regulatory.
- deleted 2y ago
  [deleted]
- brtkdotse 2y ago
  > 95% of businesses never become unicorns, but that's the goal for most Is it really the general case or is it just a HN echo chamber meme? My pet peeve is that patterns used by companies that in theory could become global unicorns are mimicked by companies where 5000 paying customers would mean an immense success
  - blagie 2y ago
    It's neither. Lifestyle companies are fine, if that's what you're aiming for. I know plenty of people who run or work at ≈1-30 person companies with no intention to grow. However, if you're going for high-growth, you need to plan for success. I've seen many potential unicorns stopped by simple lack of planning early on. Despite all the pivots which happen, if you haven't outlined a clear path from 1-3 people in a metaphorical garage to reaching $1B, it almost never happens, and sometimes for stupid reasons. If your goal is 5000 paying customers at $100 per year and $500k in annual revenues, that can lead to a very decent life. However, it's an entire different ballgame: (1) Don't take in investment (2) You probably can't hire more than one person (3) You need a plan for break-even revenue before you need to quit your job / run out of savings. (4) You need much greater than the 1-in-10 odds of success. And it's very possible (and probably not even hard) to start a sustainable 1-5 person business with >>50% odds of success, especially late career: - Find a niche you're aware of from your job - Do ballpark numbers on revenues. These should land in the $500k-$10M range. Less, and you won't sustain. More, and there will be too much competition. - Do it better than the (likely incompetent or non-existent) people doing it now - Use your network of industry contacts to sell it That's not a big enough market you need to worry about a lot of competition, competitors with VC funding, etc. Especially ones with tall moats do well -- pick some unique skillset, technology, or market access, for example. However, IF you've e.g. taken in VC funding, then you do need to plan for growth, and part of that is planning for the small odds your customer base (and ergo, your data) does grow.
estheryo 2y ago
“MOST PEOPLE DON’T HAVE THAT MUCH DATA” That's really true
vegabook 2y ago
my experience is that while data keeps growing at an exponential rate, its information content does not. In finance at least, you can easily get 100 million data points per series per day if you want everything, and you might be dealing with thousands of series. That sample rate, and the number of series, is usually 99.99% redundant, because the eigenvalues drop off almost to zero very quickly after about 10 dimensions, and often far fewer. There's very little reason to store petabytes of ticks that you will never query. It's much more reasonable in many cases to do brutal (and yes, lossy) dimensionality reduction _at ingests time_, store the first few principal components + outliers, and monitor eigenvalue stability (in case some new, previously negligable, factor, starts increasing in importance). It results in a much smaller dataset that is tractable and in many cases revelatory, because it's actually usable.
- CoastalCoder 2y ago
  Could you point to something explaining that eigenvalue / dimensions topic? It sounds interesting, but it's totally new to me.
  - mk67 2y ago
    https://en.wikipedia.org/wiki/Principal_component_analysis https://en.wikipedia.org/wiki/Principal_component_analysis
  - beng-nl 2y ago
    Nog OP, but i think they are referring to the fact that you can use PCA (principal component analysis) on a matrix of datapoints to approximate it. Works out of the box in scikit-learn. You can do (lossy) compression on rows of vectors (treated like a matrix) by taking the top N eigenvectors (largest N eigenvalues) and using them to approximate the original matrix with increasing accuracy (as N grows) by some simple linear operations. If the numbers are highly correlated, you can get a huge amount of compression with minor loss this way. Personally I like to use it to visualize linear separability of a high dimensioned set of vectors by taking a 2-component PCA and plotting them as x/y values.
- bartart 2y ago
zurfer 2y ago
This is not fully correct. Originally big data was defined by 3 dimensions: - Volume (mostly what the author talks about) [solved] - Velocity, how fast data is processed etc [solved, but expensive] - Variety [not solved] Big Data today is not: I don't have enough storage or compute. It is: I don't have enough cognitive capacity to integrate and make sense of it.
- maayank 2y ago
  What do you mean by 'variety'?
  - boesboes 2y ago
    not op, but I think they mean the data is complex, heterogeneous and noisy. You won't be able to extract meaning trivially from it, you need something to find the (hidden) meaning in the data. So AI currently, probably ;)
  - threeseed 2y ago
    Data that isn't internally managed database exports. One of the reasons big data systems took off was because enterprises had exports out of third party systems that they didn't want to model since they didn't own it. As well as a bunch of unstructured data e.g. floor plans, images, logs, telemetry etc.
  - zurfer 2y ago
    the other comments get it. It means that data comes in a ton of different shapes with poorly described schemas (technically and semantically). From the typical CSV export out of an ERP system to a proprietary message format from your own custom embedded device software.
  - nairboon 2y ago
    it doesn't fit the relational model, e.g. you have some tables, but also tons of different types of images, video, sounds, raw text, etc.
- vishnugupta 2y ago
gbin 2y ago
IMHO the main driver for big data was company founders egos. Of course your company will explode and will be a planet scale success!! We need to design for scale! This is really a tragic mistake while your product only needs one SQLite DB until you reach series C.... All the energy should be focused on the product, not its scale yet.
- antupis 2y ago
  Well generally yes although there are a couple of exceptions like IoT and GIS stuff where is very common to see 10TB+ datasets.
- threeseed 2y ago
  No. Big data was driven by people who had big data problems. It started with Hadoop which was inspired by what existed at Google and became popular in enterprises all around the world who wanted a cheaper/better way to deal with their data than Oracle. Spark came about as a solution to the complexity of Hive/Pig etc. And then once companies were able to build reliable data pipelines we started to see AI being able to be layered on top.
- jandrewrogers 2y ago
  It depends on the kind of data you work with. Many kinds of important data models -- geospatial, sensing, telemetry, et al -- can hit petabyte volumes at "hello world". Data models generated by intentional human action e.g. clicking a link, sending a message, buying something, etc are universally small. There is a limit on the number of humans and the number of intentional events they can generate per second regardless of data model. Data models generated by machines, on the other hand, can be several orders of magnitude higher velocity and higher volume, and the data model size is unbounded. These are often some of the most interesting and under-utilized data models that exist because they can get at many facts about the world that are not obtainable from the intentional human data models.
tobilg 2y ago
I witness the overengineering regarding "big" data tools and pipelines since many years... For a lot of use cases, data warehouses and data lakes are only in the gigabytes or single-digit terabytes range, thus their architecture could be much more simplified, e.g. running DuckDB on a decent EC2 instance. In my experience, doing this will yield the query results faster than some other systems even starting the query execution (yes, I'm looking at you Athena)... I even think that a lot of queries can be run from a browser nowadays, that's why I created https://sql-workbench.com/ https://sql-workbench.com/ with the help of DuckDB WASM (https://github.com/duckdb/duckdb-wasm https://github.com/duckdb/duckdb-wasm) and perspective.js (https://github.com/finos/perspective https://github.com/finos/perspective).
geertj 2y ago
I agree with the article that most data sets comfortably fit into a single traditional DB system. But I don’t think that implies that big data is dead. To me big data is about storing data in a columnar storage format with a weak schema, and using a query system based on partitioning and predicate push down instead of indexes. This allows the data to be used in an ad-hoc way by data science or other engineers to answer questions you did not have when you designed the system. Most setups would be relatively small, but could be made to scale relatively well using this architecture.
Shrezzing 2y ago
This is a quite good allegory for the way AI is currently discussed (perhaps the outcome will be different this time round). Particularly the scary slide[1] with the up-and-to-the-right graph, which is used in a near identical fashion today to show an apparently inevitable march of progress in the AI space due to scaling laws. [2]https://motherduck.com/_next/image/?url=https%3A%2F%2Fweb-assets-prod.motherduck.com%2Fassets%2Fimg%2Fimage_2_0f68796072.jpg&w=1920&q=75 https://motherduck.com/_next/image/?url=https%3A%2F%2Fweb-as...
teleforce 2y ago
Not dead, it's just having it's winter time not unlike AI winter and once it has its similar "chatbot" moment, all will be well. My take on the killer application is the climate change for example earthquakes monitoring. For a case study China has just finished building world's largest earthquake monitoring system with the cost of around USD1 Billion across the country with 15K stations [1]. Somehow at the moment is just monitoring existing earthquakes. But let's say there is a big data analytics technique can reliably predicts impending earthquake within a few days, that can probably safe many people and China now still hold the records of the largest mortality and casualty numbers due to earthquakes. Is it probable, the answer is a positive yes based on our work and initial results it's already practical but in order to do that we need integration with comprehensive in-situ IoT networks with regular and frequent data sampling similar to that of China. Secondly, China also has the largest radio astronomy telescopes and these telescopes together with other radio telescopes collaborate in real-time through e-VLBI to form a virtual giant radio telescopes as big as the earth to monitor distance stars and galaxy. This is how the black hole got its first image but at the time due to logistics one of the telescope remote disks cannot be shipped to the main processing centers in US [2]. At that moment they are not using real-time e-VLBI onky VLBI, and it tooks them several months just to get the complete sets of the black holes observation data. With e-VLBI everything is real-time and with automatic processing it will be hours instead of month. These radio telescopes can also be used for other purposes like monitoring climate change in addition to imaging black holes, their data is astronomical pardon the pun [3]. [1] Chinese Nationwide Earthquake Early Warning System and Its Performance in the 2022 Lushan M6.1 Earthquake: https://www.mdpi.com/2072-4292/14/17/4269 https://www.mdpi.com/2072-4292/14/17/4269 [2] How Scientists Captured the First Image of a Black Hole: https://www.jpl.nasa.gov/edu/news/2019/4/19/how-scientists-captured-the-first-image-of-a-black-hole/ https://www.jpl.nasa.gov/edu/news/2019/4/19/how-scientists-c... [3] Alarmed by Climate Change, Astronomers Train Their Sights on Earth: https://www.nytimes.com/2024/05/14/science/astronomy-climate-change.html https://www.nytimes.com/2024/05/14/science/astronomy-climate...
- Shrezzing 2y ago
  I think these examples still loosely fits the author's argument: > There are some cases where big data is very useful. The number of situations where it is useful is limited Even though there are some great use-cases, the overwhelming majority organisations, institutions, and projects will never have a "let's query ten petabytes" scenario that forces them away from platforms like Postgres. Most datasets, even at very large companies, fit comfortably into RAM on a server - which is now cost-effective, even in the dozens of terabytes.
kmarc 2y ago
When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and many of them came up with astonishing tens of thousands of dollars yearly. The winner of course was the guy who understood that 6TiB is what 6 of us in the room could store on our smart phones, or a $199 enterprise HDD (or three of them for redundancy), and it could be loaded (multiple times) to memory as CSV and simply run awk scripts on it. I am prone to the same fallacy: when I learn how to use a hammer, everything looks like a nail. Yet, not understanding the scale of "real" big data was a no-go in my eyes when hiring.
- wslh 2y ago
  In my context 99% of the problem is the ETL, nothing to do with complex technology. I see people stuck when they need to get this from different sources in different technologies and/or APIs.
- geraldwhen 2y ago
  I ask a similar question on screens. Almost no one gives a good answer. They describe elaborate architectures for data that fits in memory, handily.
  - mcny 2y ago
    I think that’s the way we were taught in college / grad school. If the premise of the class is relational databases, the professor says, for the purpose of this course, assume the data does not fit in memory. Additionally, assume that some normalization is necessary and a hard requirement. Problem is most students don’t listen to the first part “for the purpose of this course”. The professor does not elaborate because that is beyond the scope of the course.
    - acomjean 2y ago
dventimi 2y ago
Question for the Big Data folks: where do sampling and statistics fit into this, if at all? Unless you're summing to the penny, why would you ever need to aggregate a large volume of data (the population) rather than a small volume of data (a sample)? I'm not saying there isn't a reason. I just don't know what it is. Any thoughts from people who have genuine experience in this realm?
- disgruntledphd2 2y ago
  Sampling is almost always heavily used here, because it's Ace. However, if you need to produce row level predictions then you can't sample as you by definition need the role level data. However you can aggregate user level info into just the features you need which will get you a looooonnnnggggg way.
- gregw2 2y ago
  Good question. I am not an expert but here’s my take from my time in this space. Big data folks typically do sampling and such, but that doesn’t eliminate the need for a big data environment where such sampling can occur. Just as a compiler can’t predict every branch that could happen at compile time (sorry VLIW!) and thus CPUs need dynamic branch predictors, so too a sampling function can’t be predicted in advance of an actual dataset. In a large dataset there are many ways the sample may not represent the whole. The real world is complex. You sample away that complexity at your peril. You will often find you want to go back to the original raw dataset. Second, in a large organization, sampling alone presumes you are only focused on org-level outcomes. But in a large org there may be individuals who care about the non-aggregated data relevant to their small domain. There can be thousands of such individuals. You do sample the whole but you also have to equip people at each level of abstraction to do the same. The cardinality of your data will in some way reflect the cardinality of your organization and you can’t just sample that away.
- banku_brougham 2y ago
  There is a problem on website data, where new features are only touching a subset of customers and you need results for every single one. You wont be partitioned for this case, but the compute you need is just filtering out this set. But sampling wont get what you want especially if you are doing QC at the business team level about whether the CX is behaving as expected.
oli5679 2y ago
BigQuery has a generous 1TB/month free tier, and $6/tb afterwards. If you have small data, it's a pragmatic option, just make sure to use partitioning and sensible query patterns to limit the number of full-data scans, as you approach 'medium data' region. There are some larger data-sizes, and query patterns, where either BigQuery Capacity compute pricing, or another vendor like Snowflake, becomes more economical. https://cloud.google.com/bigquery/pricing https://cloud.google.com/bigquery/pricing BigQuery offers a choice of two compute pricing models for running queries: On-demand pricing (per TiB). With this pricing model, you are charged for the number of bytes processed by each query. The first 1 TiB of query data processed per month is free. Queries (on-demand) - $6.25 per TiB - The first 1 TiB per month is free.
ml-anon 2y ago
Big data was always a buzzword that was weirdly coopted by database people in a way which makes 0 sense. Of course there are vanishingly small number of use cases where we need fast random access to any possible record to look up. However what ML systems and in particular LLMs rely on having access to millions (if not billions or trillions) of examples. The underlying infra of which is based on some of these tools. Big Data isn't dead, just this weird idea that the tools and usecases around querying databases has been finally recognised as being mostly useless to most people. It is and always has been about training ML models.
stakhanov 2y ago
The funny thing about "big data" was that it came with a perverse incentive to avoid even the most basic and obvious optimizations on the software level, because the hardware requirement was how you proved how badass you were. Like: "Look, boss, I can compute all those averages for that report on just my laptop, by ingesting a SAMPLE of the data, rather than making those computations across the WHOLE dataset". Boss: "What do you mean 'sample'? I just don't know what you're trying to imply with your mathmo engineeringy gobbledigook! Me having spent those millions on nothing can clearly not be it, right?"
- Spooky23 2y ago
  It came with a few cohorts of Xooglers cashing their options out. The amount of salesman hype and chatter about big data, followed by the dick measuring contests about whose data was big enough to be worthy was intense for awhile.
- bpodgursky 2y ago
  This is a pretty snarky outside view and just not actually true (I spent the first part of my career trying to reduce compute spend as a data engineer). It was extremely difficult to get > 64gb on a machine for a very long time, and implementation complexity gets hard FAST when you have a hard cap. And it's EXTREMELY disruptive to have a process that fails every 1/50 times, when data is slightly too large, because your team will be juggling dozens of these routine crons, and if each of them breaks regularly, you do nothing but dumb oncall trying to trim bits off of each strong. No, Hadoop and MapReduce were not hyperefficient, but it was OK if you write it correctly, and having something that ran reliably is WAY more valuable than boutique bit-optimized C++ crap that nobody trusts or can maintain and fails every thursday with insane segfaults. (nowdays, just use Snowflake. but it was a reasonable tool for the time).
  - stakhanov 2y ago
    Why do you assume it's an "outside" view? I was freelancing for many years on projects with "Data Scientist" / "Data Engineer" in my job title, having been hired by managers who heard about "Big Data" at an event by KPMG or Accenture or whatever. And I don't understand why you're reading "boutique bit-optimized C++ crap" into "most basic and obvious optimizations". One of those most basic and obvious optimizations is to avoid reading a dataframe into memory in its entirety, when the math that you want to do on top of it can actually be done as a running accumulator while reading the data from a stream. This is possible in 90% of realistic use cases, but the fraction of software written back then that took advantage of this was shockingly small. Solving the problem by buying more machines, chopping up the dataframe into smaller pieces, and farming out the payload through Hadoop had management buy-in. Yet, for some reason, doing the sane thing, namely rewriting poorly-written software, didn't.
lokimedes 2y ago
I was a researcher at the Large Hadron Collider around the time “Big Data” became a thing. We had one of the use cases where analyzing all the data made sense, since it boiled down to frequentist statistics, the more data, the better. Yet even with a global network of supercomputers at our disposal, we funnily figured out that fast local storage was better than waiting for huge jobs to finish. So, surprise, surprise, every single grad student managed somehow to boil the relevant data for her analysis down to exactly 1-5 TB, without much loss in analysis flexibility. There must be like a law of convenience here, that rivals Amdahl’s scaling law.
- msl09 2y ago
  I think that your law of convenience is spot on. One thing that got by talking with commercial systems devs is that they are always under pressure by their clients to make their systems as cheap as possible, reducing the database stored and the size of the computations is one great way to minimize the client's monthly bill.
- civilized 2y ago
  I think there is a law of convenience, and it also explains why many technologies improve at a consistent exponential rate. People are very good at finding convenient ways to make something a little better each year, but every idea takes some minimal time to execute.
- marcosdumay 2y ago
  Let me try one: "If you can't do your statistical analysis in 1 to 5 TB of data, your methodology is flawed" This is probably more about human limitations than math. There's a clear ceiling in how much flexibility we can use. That will also change with easier ways to run new kinds of analysis, but it increases with the logarithm of the amount of things we want to do.
- kwillets 2y ago
  Back in the 80's and 90's NASA built a National Aerodynamic Simulator, which was a big Cray or similar that could crunch FEA simulations (probably a low-range graphics card nowadays). IIRC they found that the queue for that was as long or longer than it took to run jobs on cheaper hardware; MPP systems such as Beowulf grew out of those efforts.
debarshri 2y ago
My first job was doing hadoop stuff around ~2011. I think one of the biggest motivators for big data or rather hadoop adoption was that it was open source. Back then most of the data warehousing platforms dominated by oracle, netezza, EMC, teradata etc. were super expensive on per GB basis. It was followed by lot of success stories about how facebook save $$$ or you could use google's mapreduce for free etc. Everyone was talking about data being the new "oil". Enterprise could basically deploy petabyte scale warehouse run HBase or Hive on top of it and build makeshift data-warehouses. It was also when the cloud was emerging, people started creating EMR clusters and deploy workloads there. I think it was solution looking for problem. And the problem existed only for a handful of companies. I think somehow, how cloud providers abstracted lot of these tools and databases, gave a better service and we kind of forgot about hadoop et al.
LightFog 2y ago
When working in a research lab we used to have people boast that their analysis was so big it ‘brought down the cluster’ - which outed them pretty quickly to the people who knew what they were doing.
- kjkjadksj 2y ago
  Must have been abusing the head node if they did that
corentin88 2y ago
No mention of Firebase? That might explain the slow decrease of MongoDB.
WesolyKubeczek 2y ago
I remember going on one of "big data" conferences back in 2015, when it was the buzzword of the day. The talks were all concentrated around topics like: ingesting and writing the data as quickly as possible, sharding data for the benefit of ingesting data, and centralizing IoT data from around the whole world. Back then I had questions which were shrugged off — back in the day it seemed to me — as extremely naïve, as if they signified that I was not the "in" crowd somehow for asking them. The questions were: 1) Doesn't optimizing highly for key-value access mean you need that you need to anticipate, predict, and implement ALL of the future access patterns? What if you need to change your queries a year in? The most concrete answer I got was that of course a good architect needs to know and design for all possible ways the data will be queried! I was amazed at either the level of prowess of the said architects — such predictive powers that I couldn't ever dream of attaining! — or the level of self-delusion, as the cynic in me put it. 2) How can it be faster if you keep shoving intermediate processing elements into your pipeline? It's not like you just mindlessly keep adding queues upon queues. That had never been answered. The processing speeds of high-speed pipelines may be impressive, but if some stupid awk over CSV can do it just as quickly on commodity hardware, something _must_ be wrong.
nottorp 2y ago
> How many workloads need more than 24TB of RAM or 445 CPU cores? <cough> Electron?
GuB-42 2y ago
AI is the new "big data". In fact, AI as it is done today is nothing without at least terabytes of data. What the article talks about is more like a particular type of database architecture (often collectively called "NoSQL") that was a fad a few years ago, and as all fads, it went down. It doesn't mean having lots of data is useless, or that NoSQL is useless, just that it is not the solution to every problem. And also that there is a reason why regular SQL databases have been in use since the 70s: except in specific situation most people don't encounter, they just work.
coldtea 2y ago
Selling "Big data" tooling and consulting was a nice money making scheme for while it lasted.
deadbabe 2y ago
Something similar will happen with generative AI someday. AI scientists will propose all sorts of elaborate complex solutions to problems using LLMs, and the dismissive responsive will be “Your problem is solvable with a couple if statements.” Most people just don’t have problems that require AI.
- doubloon 2y ago
  Thats one of the first things Andrew Ng said in his old ML course
  - deadbabe 2y ago
    This is why I personally just can’t find motivation to even pay attention to most AI developments. It’s a toy, it does some neat things, but there’s no problem I’ve heard of or encountered where LLM style AI was the only tool for the job, or even the best tool. The main use seems to be content creation and manipulation at scale, which the vast majority of companies simply don’t have to deal with. Similarly, a lot of companies talk about how they have tons of data, but there’s never any real application or game changing insight from it. Just a couple neat tricks and product managers patting themselves on the back. Setting up a good database is probably the peak of a typical company’s tech journey.
    - int_19h 2y ago
      Natural language processing is in obvious area in which LMs are consistently outperforming the best bunches of if-statements by a very large margin, and it has very broad applicability. E.g. I would argue that its translation capabilities alone make GPT-4 worthwhile, even if it literally couldn't do anything else.
fijiaarone 2y ago
The problem with big data is that people don’t have data, they have useless noise. For lack of data, they generate random bytes collected on every mouse movement on every page, and every packet that moves through their network. It doesn’t tell them anything because the only information that means anything is who clicks that one button on their checkout page after filling out the form with their information or that one request that breaches their system. That’s why big data is synonymous with meaningless charts on pointless dashboards sold to marketing and security managers who never look at them anyway It’s like tracking the wind and humidity and temperature and barometer data every tenth of a second every square meter. It won’t help you predict the weather any better than stepping outside and looking at the sky a couple times a day.
- thfuran 2y ago
  It absolutely would. You can't build a useful model off of occasionally looking outside.
liprais 2y ago
[dead]
iamleppert 2y ago
What he means to say, is the grift is dead. All the best fish have been fished in that data lake (pun intended), leaving most waiting on the line to truly catch an appealing mackerel. Most of the big data people I know have moved on to more lucrative grifts like crypto and (more recently) AI. There are still bags to be made if you can scare up a CTO or one of his lieutenants working for a small to medium size Luddite company. Add in storage on the blockchain and a talking AI parrot if you want some extra gristle in your grift. Long live the tech grifters!
RyanHamilton 2y ago
For 10 years he sold companies on Big Data they didn't need and he only just realised most people don't have big data. Now he's switched to small data tools we should use/buy that. Is it harsh to say either a) 10 Years = He isn't good at his job b) Jordan would sell whatever he get's paid to.
renegade-otter 2y ago
Big Data is not dead - it has been reborn as AI, which is essentially Big Data 2.0. And just in the same fashion, there was massive hype around Big Data 1.0. From 2013: https://hbr.org/2013/12/you-may-not-need-big-data-after-all https://hbr.org/2013/12/you-may-not-need-big-data-after-all Everyone has so much data that they must use AI in order to tame it. The reality is, however, is that most of their data is crap and all over the place, and no amount of Big Data 1.0 or 2.0 is ever going to fix it.
donatj 2y ago
The article only touches on it for a moment but GDPR killed big data. The vast majority of the data that any regular business would have and could be considered big almost certainly contained PII in one form or another. It became too much of a liability to keep that around. With GDPR, we went from keeping everything by default unless a customer explicitly requested it gone to deleting it all automatically after a certain number of days after their license expires. This makes opaque data lakes completely untenable. Don't get me wrong, this is all a net positive. The customers data is physically removed and they don't have to worry about future leaks or malicious uses, and we get a more efficient database. The only people really fussed were the sales team trying to lure people back with promises that they could pick right back up where they left off.
gigatexal 2y ago
DuckDb is nothing short of amazing. The only thing is when the dataset is bigger than system ram it falls apart. Spilling to disk is still broken.
surfingdino 2y ago
In my experience the only time I worked on a big data project was the public Twitter firehose. The team built an amazing pipeline and it did actually deal with masses of data. Any other team I've been on were delusional and kept building expensive and overcomplicated solutions that could be replaced with a single Postgres instance. The most overcomplicated system I've seen could not process 24hrs-worth of data in under 24 hours... I was happy to move on when an opportunity presented itself.
rr808 2y ago
You're never going to get a fang job with an approach like that. And basically most developers are working towards that goal.
ricardo81 2y ago
Is there a solid definition of big data nowadays? It seems to conflate somewhat with SV companies completely dismantling privacy concerns and hoovering up as much data as possible. Lots of scenarios I'm sure. I'm just thinking of FAANG in the general case.
nextworddev 2y ago
Big data is there so that it can justify Databricks and Snowflake valuations /s
cheptsov 2y ago
A good clickbait title. One should credit the author for that. As to the topic, IMO, there is a contradiction. The only way to handle big data is to divide it into chunks that aren’t expensive to query. In that sense, no data is "big" as long as it’s handled properly. Also, about big data being only a problem for 1 percent of companies: it's a ridiculous argument implying that big data was supposed to be a problem for everyone. I personally don’t see the point behind the article, with all due respect to the author. I also see many awk experts here who have never been in charge of building enterprise data pipelines.
markus_zhang 2y ago
I think one problem that arises from practical work is: Databases seem to be biased towards either transactional (including fetching single records) or aggregational workload, but in reality both are used extensively. This also brings difficulty in data modelling, when we DEs are mostly thinking about aggregating data while our users also want to investigate single records. Actually, now that I think about it, we should have two products for the users, one let them to query single records as fast as possible without hitting the production OLTP transactional database, even from really big data (find one record from PB level data in seconds), one to power the dashboards that ONLY show aggregation. Is Lakehouse a solution? I have never used it.
lukev 2y ago
This is a good post, but it's somewhat myopically focused on typical "business" data. The most interesting applications for "big data" are all (IMO) in the scientific computing space. Yeah, your e-commerce business probably won't ever need "big data" but load up a couple genomics research sets and you sure will.
maartet 2y ago
Reminds me of this gem from the previous decade: https://www.chrisstucchio.com/blog/2013/hadoop_hatred.html https://www.chrisstucchio.com/blog/2013/hadoop_hatred.html (Don't use Hadoop - your data isn't that big) Which, of course, was discussed on HN back then: https://news.ycombinator.com/item?id=6398650 https://news.ycombinator.com/item?id=6398650
unholyguy001 2y ago
One of the problems with the article is BigQuery becomes astronomically expensive at mid to high volumes. So there is a strong incentive to keep the data in BigQuery manageable or to even move off it as data volumes get higher. Also larger enterprises don’t even use gcp all that much to begin with
tored 2y ago
Former prime minister of Sweden Stefan Löfven got wind about Big data buzzword back in 2015 and used it in one of his speeches, that this is the future, however he used the Swedish translation of Big data, stordata. That generated some funny quips about where is lilldata. https://www.aftonbladet.se/senastenytt/ttnyheter/inrikes/a/8wXyzr/lofvens-industrikansler-pa-vag https://www.aftonbladet.se/senastenytt/ttnyheter/inrikes/a/8...
xiaodai 2y ago
Big memory is eating big data.
angarg12 2y ago
Previous years I would have completely agreed with this post. Nowadays with the AI and ML craze I'm not so sure. I've seen plenty of companies using using vast amounts of data to train ML models to incorporate to their products. Definitely more data that can be handle by a traditional DB, and well into Big Data territory. This isn't a value judgement about whether that's a good idea, just an observation from talking with many tech companies doing ML. This definitely feels like a bubble that will burst in due time, but for now ML is turbocharging Big Data.
- zxxh 2y ago
  Can you explain? It seems like you’re just using certain words without context.
- int_19h 2y ago
  But do they need to query that data?
deleted 2y ago
[deleted]
breckognize 2y ago
99.9%+ of data sets fit on an SSD, and 99%+ fit in memory. [1] This is the thesis for https://rowzero.io https://rowzero.io. We provide a real spreadsheet interface on top of these data sets, which gives you all the richness and interactivity that affords. In the rare cases you need more than that, you can hire engineers. The rest of the time, a spreadsheet is all you need. [1] I made these up.
BiboAlpha 2y ago
One of the major challenges with Big Data is that it often fails to deliver real business value, instead offering only misleading promises.
mavili 2y ago
Big Data hasn't been about storage, I thought it was always about processing. Guy obviously knows his stuff but I got the impression he stressed more about storage and how that's cheap and easy these days. When he does mention processing/computing, he mentions that most of the time people end up only querying recent data (ie small chunk of actual data they hold) but that bears the question: is querying only small chunk of data what businesses need, or are they doing it because querying the whole dataset is just not manageable? In other words, if processing all data at once was as easy as querying the most recent X percent, would most businesses still choose to only query the small chunk? I think there lies the answer whether Big Data (processing) is needed or not.
siliconc0w 2y ago
Storing data in object and querying from compute caching what you can basically scales until your queries are too expensive for a single node.
idunnoman1222 2y ago
I love how the way he talks about this. His paycheque from big-data is what is dead. His service offering was always bullshit.
fredstar 2y ago
I am in a software services company for more than 15 years. And to be honest, a lot of these big topics have always been some kind of sales talk or door opener. You write a white paper, nominate an 'expert' in your team and use these things in conversations with clients. Sure some trends are way more real and useful then others. But for me the article hits the nail on its head.
deleted 2y ago
[deleted]
aorloff 2y ago
What is dead is the notion that some ever expanding data lake is a mine full of gems and not a cost center.
schindlabua 2y ago
Used to work at a company that produced 20 gigs of analytics every day which is probably the biggest data I'll ever work on. My junior project was writing some data crunching jobs that did aggregations batched and in real time, and store the result in parquet blobs in azure. My boss was smart enough to have stakeholder meetings where they regularly discussed what to keep and what to throw away, and with some smart algorithms we were able to compress all that data down into like 200MB per day. We loaded the last 2 months into an sql server and the last 2 years further aggregated into another, and the whole company used the data in excel to do queries on it in reasonable time. The big data is rotting away on tape storage in case they ever need it in the future. My boss got a lot of stuff right and I learned a lot, though I only realized that in hindsight. Dude was a bad manager but he knew his data.
dang 2y ago
Related: Big data is dead - https://news.ycombinator.com/item?id=34694926 https://news.ycombinator.com/item?id=34694926 - Feb 2023 (433 comments) Big Data Is Dead - https://news.ycombinator.com/item?id=33631561 https://news.ycombinator.com/item?id=33631561 - Nov 2022 (7 comments)
kurts_mustache 2y ago
What was the source data here? It seems like a lot of the graphs are just the author's intuitive feel rather than being backed by any sort of hard data. Did I miss it in there somewhere?
lkdfjlkdfjlg 2y ago
TLDR: he used to say that you needed the thing he's selling. He changed his mind now and you need the opposite, which he's selling.
hsuduebc2 2y ago
I'm always waiting for this phrase for every one of the actual buzzwords.
rldjbpin 2y ago
"_ is dead" titles rarely represent reality. same case here. more accurately, "big data" or its main paradigms are not for everyone. on the other hand, most teams are still unable to reasonably consolidate their data to leverage the benefits of these tools. "AI" will have a similar fate in a few years time for similar reasons.
taylodl 2y ago
To me, the best thing to come out of the Big Data era was Spark Streaming. That tool provides a nice architecture for processing a stream of data in near real-time, from multiple data sources no less, that we didn't have before. Spark in general was a great tool to get out of the Big Data era.
drumdance 2y ago
I was CTO of an analytics startup in the early days of MongoDB hype. We built on MySQL. One of our competitors was MixPanel. We were having scaling issues with one particular customer, so I looked into the various solutions. They all looked pretty cool, but I didn't like that the developer communities were small compared to MySQL and PostgreSQL. So I ended up sticking with MySQL and building a simple sharding mechanism that allowed us to move big customers to their own server if necessary. It worked great. While I was working on that, MixPanel published a long, highly detailed blog post about why they had moved to MongoDB. That led me to have second thoughts. "If these guys are doing it, maybe we should too?" But I was pretty far down the sharding development path and couldn't justify making such a wholesale change based on one blog post. Fast forward about nine months, and MixPanel published a new blog post about why they had moved off MongoDB. Vindication! Ever since then I've had conservative disposition when it comes to deploying new technologies.

585 points by armanke13 2y ago

clkao 2y ago

previous discussion: https://news.ycombinator.com/item?id=34694926 https://news.ycombinator.com/item?id=34694926

spicyusername 2y ago
I find it interesting that this comment section and that comment section seem to focus on different things, despite being triggered by the same input.

ahartmetz 2y ago

I guess that hype cycle ended at the plateau of being dead. A not uncommon outcome in this incredibly fashion-driven industry.

silvestrov 2y ago
It has just been rebranded as AI. AI also use all the data, just with a magick neural network to figure out what it all means.
- quonn 2y ago
  The overlap in terms of the used technologies, the required skills, the actual products and the target market is minimal. AI is not mostly Hadoop, it's not MapReduce, the hardware is different, the software is different, the skillset is very different and a chatbot or image generator is very different from a batch job producing an answer to a query.
  - renegade-otter 2y ago
    But the underlying problem is the same - companies that use Big Data tech are clueless about data management. You can use unicorns - it's not going to do anything. "Garbage in, garbage out" is a timeless principle.
- WesolyKubeczek 2y ago
  Given how often it hallucinates, it should be rebranded to "high data".
- Cyberdog 2y ago
  Assuming you're serious for a moment, I don't think AI is really a practical tool for working with big data. - The "hallucination" factor means every result an AI tells you about big data is suspect. I'm sure some of you who really understand AI more than the average person can "um akshually" me on this and tell me how it's possible to configure ChatGPT to absolutely be honest 100% of the time but given the current state of what I've seen from general-purpose AI tools, I just can't trust it. In many ways this is worse than MongoDB just dropping data since at least Mongo won't make up conclusions about data that's not there. - At the end of the day - and I think we're going to be seeing this happen a lot in the future with other workflows as well - you're using this heavy, low-performance general-purpose tool to solve a problem which can be solved much more performatively by using tools which have been designed from the beginning to handle data management and analysis. The reason traditional SQL RDBMSes have endured and aren't going anywhere soon is partially because they've proven to be a very good compromise between general functionality and performance for the task of managing various types of data. AI is nowhere near as good of a balance for this task in almost all cases. All that being said, the same way Electron has proven to be a popular tool for writing widely-used desktop and mobile applications, performance and UI concerns be damned all the way to hell, I'm sure we'll be seeing AI-powered "big data" analysis tools very soon if they're not out there already, and they will suck but people will use them anyway to everyone's detriment.
  - silvestrov 2y ago
    A comment from the old post: https://news.ycombinator.com/item?id=34696065 https://news.ycombinator.com/item?id=34696065 > I used to joke that Data Scientists exist not to uncover insights or provide analysis, but merely to provide factoids that confirm senior management's prior beliefs. I think AI is used for the same purpose in companies: signal to the world that the company is using the latest tech and internally for supporting existing political beliefs. So same job. Hallucination is not a problem here as the AI conclusions are not used when they don't align to existing political beliefs.
  - vitus 2y ago
    > The "hallucination" factor means every result an AI tells you about big data is suspect. AI / ML means more than just LLM chat output, even if that's the current hype cycle of the last couple of years. ML can be used to build a perfectly serviceable classifier, or predictor, or outlier detector. It suffers from the lack of explainability that's always plagued AI / ML, especially as you start looking at deeper neural networks where you're more and more heavily reliant on their ability to approximate arbitrary functions as you add more layers. > you're using this heavy, low-performance general-purpose tool to solve a problem which can be solved much more performatively by using tools which have been designed from the beginning to handle data management and analysis You are not wrong here, but one challenge is that sometimes even your domain experts do not know how to solve the problem, and applying traditional statistical methods without understanding the space is a great way of identifying spurious correlations. (To be fair, this applies in equal measure to ML methods.)

blagie 2y ago

Overall, I agree with much of this post, but there are several caveats: 1) Mongo is a bad point of reference. The one lesson I've learned is that there is nothing Mongo does which postgresql doesn't do better. Big data solutions aren't nosql / mongo, but usually things like columnar databases, map/reduce, Cassandra, etc. 2) Plan for success 95% of businesses never become unicorns, but that's the goal for most (for the 5% which do). If you don't plan for it, you won't make it. The reason to architect for scalability when you have 5 customers is so if that exponential growth cycle hits, you can capitalize on it. That's not just architecture. To have any chance of becoming a unicorn, every part of the business needs to be planned for now and for later: How do we make this practical / sustainable today? How do we make sure it can grow later when we have millions of customers? A lot of this can be left as scaffolding (we'll swap in [X], but for now, we'll do [Y]). But the key lessons are correct: - Most data isn't big. I can fit data about every person in the world on a $100 Chromebook. (8 billion people * 8 bits of data = 8GB) - Most data is rarely queried, and most queries are tiny. The first step in most big data jobs I've done is taking terabytes of data and shrinking it down to the GB, MB, or oven KB-scale data I need. Caveat: I have no algorithm for predicting what I'll need in the future. - Cost of data is increasing with regulatory.

deleted 2y ago
[deleted]
brtkdotse 2y ago
> 95% of businesses never become unicorns, but that's the goal for most Is it really the general case or is it just a HN echo chamber meme? My pet peeve is that patterns used by companies that in theory could become global unicorns are mimicked by companies where 5000 paying customers would mean an immense success
- blagie 2y ago
  It's neither. Lifestyle companies are fine, if that's what you're aiming for. I know plenty of people who run or work at ≈1-30 person companies with no intention to grow. However, if you're going for high-growth, you need to plan for success. I've seen many potential unicorns stopped by simple lack of planning early on. Despite all the pivots which happen, if you haven't outlined a clear path from 1-3 people in a metaphorical garage to reaching $1B, it almost never happens, and sometimes for stupid reasons. If your goal is 5000 paying customers at $100 per year and $500k in annual revenues, that can lead to a very decent life. However, it's an entire different ballgame: (1) Don't take in investment (2) You probably can't hire more than one person (3) You need a plan for break-even revenue before you need to quit your job / run out of savings. (4) You need much greater than the 1-in-10 odds of success. And it's very possible (and probably not even hard) to start a sustainable 1-5 person business with >>50% odds of success, especially late career: - Find a niche you're aware of from your job - Do ballpark numbers on revenues. These should land in the $500k-$10M range. Less, and you won't sustain. More, and there will be too much competition. - Do it better than the (likely incompetent or non-existent) people doing it now - Use your network of industry contacts to sell it That's not a big enough market you need to worry about a lot of competition, competitors with VC funding, etc. Especially ones with tall moats do well -- pick some unique skillset, technology, or market access, for example. However, IF you've e.g. taken in VC funding, then you do need to plan for growth, and part of that is planning for the small odds your customer base (and ergo, your data) does grow.

estheryo 2y ago

“MOST PEOPLE DON’T HAVE THAT MUCH DATA” That's really true

vegabook 2y ago

my experience is that while data keeps growing at an exponential rate, its information content does not. In finance at least, you can easily get 100 million data points per series per day if you want everything, and you might be dealing with thousands of series. That sample rate, and the number of series, is usually 99.99% redundant, because the eigenvalues drop off almost to zero very quickly after about 10 dimensions, and often far fewer. There's very little reason to store petabytes of ticks that you will never query. It's much more reasonable in many cases to do brutal (and yes, lossy) dimensionality reduction _at ingests time_, store the first few principal components + outliers, and monitor eigenvalue stability (in case some new, previously negligable, factor, starts increasing in importance). It results in a much smaller dataset that is tractable and in many cases revelatory, because it's actually usable.

CoastalCoder 2y ago
Could you point to something explaining that eigenvalue / dimensions topic? It sounds interesting, but it's totally new to me.
- mk67 2y ago
  https://en.wikipedia.org/wiki/Principal_component_analysis https://en.wikipedia.org/wiki/Principal_component_analysis
- beng-nl 2y ago
  Nog OP, but i think they are referring to the fact that you can use PCA (principal component analysis) on a matrix of datapoints to approximate it. Works out of the box in scikit-learn. You can do (lossy) compression on rows of vectors (treated like a matrix) by taking the top N eigenvectors (largest N eigenvalues) and using them to approximate the original matrix with increasing accuracy (as N grows) by some simple linear operations. If the numbers are highly correlated, you can get a huge amount of compression with minor loss this way. Personally I like to use it to visualize linear separability of a high dimensioned set of vectors by taking a 2-component PCA and plotting them as x/y values.
bartart 2y ago

zurfer 2y ago

This is not fully correct. Originally big data was defined by 3 dimensions: - Volume (mostly what the author talks about) [solved] - Velocity, how fast data is processed etc [solved, but expensive] - Variety [not solved] Big Data today is not: I don't have enough storage or compute. It is: I don't have enough cognitive capacity to integrate and make sense of it.

maayank 2y ago
What do you mean by 'variety'?
- boesboes 2y ago
  not op, but I think they mean the data is complex, heterogeneous and noisy. You won't be able to extract meaning trivially from it, you need something to find the (hidden) meaning in the data. So AI currently, probably ;)
- threeseed 2y ago
  Data that isn't internally managed database exports. One of the reasons big data systems took off was because enterprises had exports out of third party systems that they didn't want to model since they didn't own it. As well as a bunch of unstructured data e.g. floor plans, images, logs, telemetry etc.
- zurfer 2y ago
  the other comments get it. It means that data comes in a ton of different shapes with poorly described schemas (technically and semantically). From the typical CSV export out of an ERP system to a proprietary message format from your own custom embedded device software.
- nairboon 2y ago
  it doesn't fit the relational model, e.g. you have some tables, but also tons of different types of images, video, sounds, raw text, etc.
vishnugupta 2y ago

gbin 2y ago

IMHO the main driver for big data was company founders egos. Of course your company will explode and will be a planet scale success!! We need to design for scale! This is really a tragic mistake while your product only needs one SQLite DB until you reach series C.... All the energy should be focused on the product, not its scale yet.

antupis 2y ago
Well generally yes although there are a couple of exceptions like IoT and GIS stuff where is very common to see 10TB+ datasets.
threeseed 2y ago
No. Big data was driven by people who had big data problems. It started with Hadoop which was inspired by what existed at Google and became popular in enterprises all around the world who wanted a cheaper/better way to deal with their data than Oracle. Spark came about as a solution to the complexity of Hive/Pig etc. And then once companies were able to build reliable data pipelines we started to see AI being able to be layered on top.
jandrewrogers 2y ago
It depends on the kind of data you work with. Many kinds of important data models -- geospatial, sensing, telemetry, et al -- can hit petabyte volumes at "hello world". Data models generated by intentional human action e.g. clicking a link, sending a message, buying something, etc are universally small. There is a limit on the number of humans and the number of intentional events they can generate per second regardless of data model. Data models generated by machines, on the other hand, can be several orders of magnitude higher velocity and higher volume, and the data model size is unbounded. These are often some of the most interesting and under-utilized data models that exist because they can get at many facts about the world that are not obtainable from the intentional human data models.

tobilg 2y ago

I witness the overengineering regarding "big" data tools and pipelines since many years... For a lot of use cases, data warehouses and data lakes are only in the gigabytes or single-digit terabytes range, thus their architecture could be much more simplified, e.g. running DuckDB on a decent EC2 instance. In my experience, doing this will yield the query results faster than some other systems even starting the query execution (yes, I'm looking at you Athena)... I even think that a lot of queries can be run from a browser nowadays, that's why I created https://sql-workbench.com/ https://sql-workbench.com/ with the help of DuckDB WASM (https://github.com/duckdb/duckdb-wasm https://github.com/duckdb/duckdb-wasm) and perspective.js (https://github.com/finos/perspective https://github.com/finos/perspective).

geertj 2y ago

I agree with the article that most data sets comfortably fit into a single traditional DB system. But I don’t think that implies that big data is dead. To me big data is about storing data in a columnar storage format with a weak schema, and using a query system based on partitioning and predicate push down instead of indexes. This allows the data to be used in an ad-hoc way by data science or other engineers to answer questions you did not have when you designed the system. Most setups would be relatively small, but could be made to scale relatively well using this architecture.

Shrezzing 2y ago

This is a quite good allegory for the way AI is currently discussed (perhaps the outcome will be different this time round). Particularly the scary slide[1] with the up-and-to-the-right graph, which is used in a near identical fashion today to show an apparently inevitable march of progress in the AI space due to scaling laws. [2]https://motherduck.com/_next/image/?url=https%3A%2F%2Fweb-assets-prod.motherduck.com%2Fassets%2Fimg%2Fimage_2_0f68796072.jpg&w=1920&q=75 https://motherduck.com/_next/image/?url=https%3A%2F%2Fweb-as...

teleforce 2y ago

Not dead, it's just having it's winter time not unlike AI winter and once it has its similar "chatbot" moment, all will be well. My take on the killer application is the climate change for example earthquakes monitoring. For a case study China has just finished building world's largest earthquake monitoring system with the cost of around USD1 Billion across the country with 15K stations [1]. Somehow at the moment is just monitoring existing earthquakes. But let's say there is a big data analytics technique can reliably predicts impending earthquake within a few days, that can probably safe many people and China now still hold the records of the largest mortality and casualty numbers due to earthquakes. Is it probable, the answer is a positive yes based on our work and initial results it's already practical but in order to do that we need integration with comprehensive in-situ IoT networks with regular and frequent data sampling similar to that of China. Secondly, China also has the largest radio astronomy telescopes and these telescopes together with other radio telescopes collaborate in real-time through e-VLBI to form a virtual giant radio telescopes as big as the earth to monitor distance stars and galaxy. This is how the black hole got its first image but at the time due to logistics one of the telescope remote disks cannot be shipped to the main processing centers in US [2]. At that moment they are not using real-time e-VLBI onky VLBI, and it tooks them several months just to get the complete sets of the black holes observation data. With e-VLBI everything is real-time and with automatic processing it will be hours instead of month. These radio telescopes can also be used for other purposes like monitoring climate change in addition to imaging black holes, their data is astronomical pardon the pun [3]. [1] Chinese Nationwide Earthquake Early Warning System and Its Performance in the 2022 Lushan M6.1 Earthquake: https://www.mdpi.com/2072-4292/14/17/4269 https://www.mdpi.com/2072-4292/14/17/4269 [2] How Scientists Captured the First Image of a Black Hole: https://www.jpl.nasa.gov/edu/news/2019/4/19/how-scientists-captured-the-first-image-of-a-black-hole/ https://www.jpl.nasa.gov/edu/news/2019/4/19/how-scientists-c... [3] Alarmed by Climate Change, Astronomers Train Their Sights on Earth: https://www.nytimes.com/2024/05/14/science/astronomy-climate-change.html https://www.nytimes.com/2024/05/14/science/astronomy-climate...

Shrezzing 2y ago
I think these examples still loosely fits the author's argument: > There are some cases where big data is very useful. The number of situations where it is useful is limited Even though there are some great use-cases, the overwhelming majority organisations, institutions, and projects will never have a "let's query ten petabytes" scenario that forces them away from platforms like Postgres. Most datasets, even at very large companies, fit comfortably into RAM on a server - which is now cost-effective, even in the dozens of terabytes.

kmarc 2y ago

When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and many of them came up with astonishing tens of thousands of dollars yearly. The winner of course was the guy who understood that 6TiB is what 6 of us in the room could store on our smart phones, or a $199 enterprise HDD (or three of them for redundancy), and it could be loaded (multiple times) to memory as CSV and simply run awk scripts on it. I am prone to the same fallacy: when I learn how to use a hammer, everything looks like a nail. Yet, not understanding the scale of "real" big data was a no-go in my eyes when hiring.

wslh 2y ago
In my context 99% of the problem is the ETL, nothing to do with complex technology. I see people stuck when they need to get this from different sources in different technologies and/or APIs.
geraldwhen 2y ago
I ask a similar question on screens. Almost no one gives a good answer. They describe elaborate architectures for data that fits in memory, handily.
- mcny 2y ago
  I think that’s the way we were taught in college / grad school. If the premise of the class is relational databases, the professor says, for the purpose of this course, assume the data does not fit in memory. Additionally, assume that some normalization is necessary and a hard requirement. Problem is most students don’t listen to the first part “for the purpose of this course”. The professor does not elaborate because that is beyond the scope of the course.
  - acomjean 2y ago

dventimi 2y ago

Question for the Big Data folks: where do sampling and statistics fit into this, if at all? Unless you're summing to the penny, why would you ever need to aggregate a large volume of data (the population) rather than a small volume of data (a sample)? I'm not saying there isn't a reason. I just don't know what it is. Any thoughts from people who have genuine experience in this realm?

disgruntledphd2 2y ago
Sampling is almost always heavily used here, because it's Ace. However, if you need to produce row level predictions then you can't sample as you by definition need the role level data. However you can aggregate user level info into just the features you need which will get you a looooonnnnggggg way.
gregw2 2y ago
Good question. I am not an expert but here’s my take from my time in this space. Big data folks typically do sampling and such, but that doesn’t eliminate the need for a big data environment where such sampling can occur. Just as a compiler can’t predict every branch that could happen at compile time (sorry VLIW!) and thus CPUs need dynamic branch predictors, so too a sampling function can’t be predicted in advance of an actual dataset. In a large dataset there are many ways the sample may not represent the whole. The real world is complex. You sample away that complexity at your peril. You will often find you want to go back to the original raw dataset. Second, in a large organization, sampling alone presumes you are only focused on org-level outcomes. But in a large org there may be individuals who care about the non-aggregated data relevant to their small domain. There can be thousands of such individuals. You do sample the whole but you also have to equip people at each level of abstraction to do the same. The cardinality of your data will in some way reflect the cardinality of your organization and you can’t just sample that away.
banku_brougham 2y ago
There is a problem on website data, where new features are only touching a subset of customers and you need results for every single one. You wont be partitioned for this case, but the compute you need is just filtering out this set. But sampling wont get what you want especially if you are doing QC at the business team level about whether the CX is behaving as expected.

oli5679 2y ago

BigQuery has a generous 1TB/month free tier, and $6/tb afterwards. If you have small data, it's a pragmatic option, just make sure to use partitioning and sensible query patterns to limit the number of full-data scans, as you approach 'medium data' region. There are some larger data-sizes, and query patterns, where either BigQuery Capacity compute pricing, or another vendor like Snowflake, becomes more economical. https://cloud.google.com/bigquery/pricing https://cloud.google.com/bigquery/pricing BigQuery offers a choice of two compute pricing models for running queries: On-demand pricing (per TiB). With this pricing model, you are charged for the number of bytes processed by each query. The first 1 TiB of query data processed per month is free. Queries (on-demand) - $6.25 per TiB - The first 1 TiB per month is free.

ml-anon 2y ago

Big data was always a buzzword that was weirdly coopted by database people in a way which makes 0 sense. Of course there are vanishingly small number of use cases where we need fast random access to any possible record to look up. However what ML systems and in particular LLMs rely on having access to millions (if not billions or trillions) of examples. The underlying infra of which is based on some of these tools. Big Data isn't dead, just this weird idea that the tools and usecases around querying databases has been finally recognised as being mostly useless to most people. It is and always has been about training ML models.

stakhanov 2y ago

The funny thing about "big data" was that it came with a perverse incentive to avoid even the most basic and obvious optimizations on the software level, because the hardware requirement was how you proved how badass you were. Like: "Look, boss, I can compute all those averages for that report on just my laptop, by ingesting a SAMPLE of the data, rather than making those computations across the WHOLE dataset". Boss: "What do you mean 'sample'? I just don't know what you're trying to imply with your mathmo engineeringy gobbledigook! Me having spent those millions on nothing can clearly not be it, right?"

Spooky23 2y ago
It came with a few cohorts of Xooglers cashing their options out. The amount of salesman hype and chatter about big data, followed by the dick measuring contests about whose data was big enough to be worthy was intense for awhile.
bpodgursky 2y ago
This is a pretty snarky outside view and just not actually true (I spent the first part of my career trying to reduce compute spend as a data engineer). It was extremely difficult to get > 64gb on a machine for a very long time, and implementation complexity gets hard FAST when you have a hard cap. And it's EXTREMELY disruptive to have a process that fails every 1/50 times, when data is slightly too large, because your team will be juggling dozens of these routine crons, and if each of them breaks regularly, you do nothing but dumb oncall trying to trim bits off of each strong. No, Hadoop and MapReduce were not hyperefficient, but it was OK if you write it correctly, and having something that ran reliably is WAY more valuable than boutique bit-optimized C++ crap that nobody trusts or can maintain and fails every thursday with insane segfaults. (nowdays, just use Snowflake. but it was a reasonable tool for the time).
- stakhanov 2y ago
  Why do you assume it's an "outside" view? I was freelancing for many years on projects with "Data Scientist" / "Data Engineer" in my job title, having been hired by managers who heard about "Big Data" at an event by KPMG or Accenture or whatever. And I don't understand why you're reading "boutique bit-optimized C++ crap" into "most basic and obvious optimizations". One of those most basic and obvious optimizations is to avoid reading a dataframe into memory in its entirety, when the math that you want to do on top of it can actually be done as a running accumulator while reading the data from a stream. This is possible in 90% of realistic use cases, but the fraction of software written back then that took advantage of this was shockingly small. Solving the problem by buying more machines, chopping up the dataframe into smaller pieces, and farming out the payload through Hadoop had management buy-in. Yet, for some reason, doing the sane thing, namely rewriting poorly-written software, didn't.

lokimedes 2y ago

I was a researcher at the Large Hadron Collider around the time “Big Data” became a thing. We had one of the use cases where analyzing all the data made sense, since it boiled down to frequentist statistics, the more data, the better. Yet even with a global network of supercomputers at our disposal, we funnily figured out that fast local storage was better than waiting for huge jobs to finish. So, surprise, surprise, every single grad student managed somehow to boil the relevant data for her analysis down to exactly 1-5 TB, without much loss in analysis flexibility. There must be like a law of convenience here, that rivals Amdahl’s scaling law.

msl09 2y ago
I think that your law of convenience is spot on. One thing that got by talking with commercial systems devs is that they are always under pressure by their clients to make their systems as cheap as possible, reducing the database stored and the size of the computations is one great way to minimize the client's monthly bill.
civilized 2y ago
I think there is a law of convenience, and it also explains why many technologies improve at a consistent exponential rate. People are very good at finding convenient ways to make something a little better each year, but every idea takes some minimal time to execute.
marcosdumay 2y ago
Let me try one: "If you can't do your statistical analysis in 1 to 5 TB of data, your methodology is flawed" This is probably more about human limitations than math. There's a clear ceiling in how much flexibility we can use. That will also change with easier ways to run new kinds of analysis, but it increases with the logarithm of the amount of things we want to do.
kwillets 2y ago
Back in the 80's and 90's NASA built a National Aerodynamic Simulator, which was a big Cray or similar that could crunch FEA simulations (probably a low-range graphics card nowadays). IIRC they found that the queue for that was as long or longer than it took to run jobs on cheaper hardware; MPP systems such as Beowulf grew out of those efforts.

debarshri 2y ago

My first job was doing hadoop stuff around ~2011. I think one of the biggest motivators for big data or rather hadoop adoption was that it was open source. Back then most of the data warehousing platforms dominated by oracle, netezza, EMC, teradata etc. were super expensive on per GB basis. It was followed by lot of success stories about how facebook save $$$ or you could use google's mapreduce for free etc. Everyone was talking about data being the new "oil". Enterprise could basically deploy petabyte scale warehouse run HBase or Hive on top of it and build makeshift data-warehouses. It was also when the cloud was emerging, people started creating EMR clusters and deploy workloads there. I think it was solution looking for problem. And the problem existed only for a handful of companies. I think somehow, how cloud providers abstracted lot of these tools and databases, gave a better service and we kind of forgot about hadoop et al.

LightFog 2y ago

When working in a research lab we used to have people boast that their analysis was so big it ‘brought down the cluster’ - which outed them pretty quickly to the people who knew what they were doing.

kjkjadksj 2y ago
Must have been abusing the head node if they did that

corentin88 2y ago

No mention of Firebase? That might explain the slow decrease of MongoDB.

WesolyKubeczek 2y ago

I remember going on one of "big data" conferences back in 2015, when it was the buzzword of the day. The talks were all concentrated around topics like: ingesting and writing the data as quickly as possible, sharding data for the benefit of ingesting data, and centralizing IoT data from around the whole world. Back then I had questions which were shrugged off — back in the day it seemed to me — as extremely naïve, as if they signified that I was not the "in" crowd somehow for asking them. The questions were: 1) Doesn't optimizing highly for key-value access mean you need that you need to anticipate, predict, and implement ALL of the future access patterns? What if you need to change your queries a year in? The most concrete answer I got was that of course a good architect needs to know and design for all possible ways the data will be queried! I was amazed at either the level of prowess of the said architects — such predictive powers that I couldn't ever dream of attaining! — or the level of self-delusion, as the cynic in me put it. 2) How can it be faster if you keep shoving intermediate processing elements into your pipeline? It's not like you just mindlessly keep adding queues upon queues. That had never been answered. The processing speeds of high-speed pipelines may be impressive, but if some stupid awk over CSV can do it just as quickly on commodity hardware, something _must_ be wrong.

nottorp 2y ago

> How many workloads need more than 24TB of RAM or 445 CPU cores? <cough> Electron?

GuB-42 2y ago

AI is the new "big data". In fact, AI as it is done today is nothing without at least terabytes of data. What the article talks about is more like a particular type of database architecture (often collectively called "NoSQL") that was a fad a few years ago, and as all fads, it went down. It doesn't mean having lots of data is useless, or that NoSQL is useless, just that it is not the solution to every problem. And also that there is a reason why regular SQL databases have been in use since the 70s: except in specific situation most people don't encounter, they just work.

coldtea 2y ago

Selling "Big data" tooling and consulting was a nice money making scheme for while it lasted.

deadbabe 2y ago

Something similar will happen with generative AI someday. AI scientists will propose all sorts of elaborate complex solutions to problems using LLMs, and the dismissive responsive will be “Your problem is solvable with a couple if statements.” Most people just don’t have problems that require AI.

doubloon 2y ago
Thats one of the first things Andrew Ng said in his old ML course
- deadbabe 2y ago
  This is why I personally just can’t find motivation to even pay attention to most AI developments. It’s a toy, it does some neat things, but there’s no problem I’ve heard of or encountered where LLM style AI was the only tool for the job, or even the best tool. The main use seems to be content creation and manipulation at scale, which the vast majority of companies simply don’t have to deal with. Similarly, a lot of companies talk about how they have tons of data, but there’s never any real application or game changing insight from it. Just a couple neat tricks and product managers patting themselves on the back. Setting up a good database is probably the peak of a typical company’s tech journey.
  - int_19h 2y ago
    Natural language processing is in obvious area in which LMs are consistently outperforming the best bunches of if-statements by a very large margin, and it has very broad applicability. E.g. I would argue that its translation capabilities alone make GPT-4 worthwhile, even if it literally couldn't do anything else.

fijiaarone 2y ago

The problem with big data is that people don’t have data, they have useless noise. For lack of data, they generate random bytes collected on every mouse movement on every page, and every packet that moves through their network. It doesn’t tell them anything because the only information that means anything is who clicks that one button on their checkout page after filling out the form with their information or that one request that breaches their system. That’s why big data is synonymous with meaningless charts on pointless dashboards sold to marketing and security managers who never look at them anyway It’s like tracking the wind and humidity and temperature and barometer data every tenth of a second every square meter. It won’t help you predict the weather any better than stepping outside and looking at the sky a couple times a day.

thfuran 2y ago
It absolutely would. You can't build a useful model off of occasionally looking outside.

liprais 2y ago

[dead]

iamleppert 2y ago

What he means to say, is the grift is dead. All the best fish have been fished in that data lake (pun intended), leaving most waiting on the line to truly catch an appealing mackerel. Most of the big data people I know have moved on to more lucrative grifts like crypto and (more recently) AI. There are still bags to be made if you can scare up a CTO or one of his lieutenants working for a small to medium size Luddite company. Add in storage on the blockchain and a talking AI parrot if you want some extra gristle in your grift. Long live the tech grifters!

RyanHamilton 2y ago

For 10 years he sold companies on Big Data they didn't need and he only just realised most people don't have big data. Now he's switched to small data tools we should use/buy that. Is it harsh to say either a) 10 Years = He isn't good at his job b) Jordan would sell whatever he get's paid to.

renegade-otter 2y ago

Big Data is not dead - it has been reborn as AI, which is essentially Big Data 2.0. And just in the same fashion, there was massive hype around Big Data 1.0. From 2013: https://hbr.org/2013/12/you-may-not-need-big-data-after-all https://hbr.org/2013/12/you-may-not-need-big-data-after-all Everyone has so much data that they must use AI in order to tame it. The reality is, however, is that most of their data is crap and all over the place, and no amount of Big Data 1.0 or 2.0 is ever going to fix it.

donatj 2y ago

The article only touches on it for a moment but GDPR killed big data. The vast majority of the data that any regular business would have and could be considered big almost certainly contained PII in one form or another. It became too much of a liability to keep that around. With GDPR, we went from keeping everything by default unless a customer explicitly requested it gone to deleting it all automatically after a certain number of days after their license expires. This makes opaque data lakes completely untenable. Don't get me wrong, this is all a net positive. The customers data is physically removed and they don't have to worry about future leaks or malicious uses, and we get a more efficient database. The only people really fussed were the sales team trying to lure people back with promises that they could pick right back up where they left off.

gigatexal 2y ago

DuckDb is nothing short of amazing. The only thing is when the dataset is bigger than system ram it falls apart. Spilling to disk is still broken.

surfingdino 2y ago

In my experience the only time I worked on a big data project was the public Twitter firehose. The team built an amazing pipeline and it did actually deal with masses of data. Any other team I've been on were delusional and kept building expensive and overcomplicated solutions that could be replaced with a single Postgres instance. The most overcomplicated system I've seen could not process 24hrs-worth of data in under 24 hours... I was happy to move on when an opportunity presented itself.

rr808 2y ago

You're never going to get a fang job with an approach like that. And basically most developers are working towards that goal.

ricardo81 2y ago

Is there a solid definition of big data nowadays? It seems to conflate somewhat with SV companies completely dismantling privacy concerns and hoovering up as much data as possible. Lots of scenarios I'm sure. I'm just thinking of FAANG in the general case.

nextworddev 2y ago

Big data is there so that it can justify Databricks and Snowflake valuations /s

cheptsov 2y ago

A good clickbait title. One should credit the author for that. As to the topic, IMO, there is a contradiction. The only way to handle big data is to divide it into chunks that aren’t expensive to query. In that sense, no data is "big" as long as it’s handled properly. Also, about big data being only a problem for 1 percent of companies: it's a ridiculous argument implying that big data was supposed to be a problem for everyone. I personally don’t see the point behind the article, with all due respect to the author. I also see many awk experts here who have never been in charge of building enterprise data pipelines.

markus_zhang 2y ago

I think one problem that arises from practical work is: Databases seem to be biased towards either transactional (including fetching single records) or aggregational workload, but in reality both are used extensively. This also brings difficulty in data modelling, when we DEs are mostly thinking about aggregating data while our users also want to investigate single records. Actually, now that I think about it, we should have two products for the users, one let them to query single records as fast as possible without hitting the production OLTP transactional database, even from really big data (find one record from PB level data in seconds), one to power the dashboards that ONLY show aggregation. Is Lakehouse a solution? I have never used it.

lukev 2y ago

This is a good post, but it's somewhat myopically focused on typical "business" data. The most interesting applications for "big data" are all (IMO) in the scientific computing space. Yeah, your e-commerce business probably won't ever need "big data" but load up a couple genomics research sets and you sure will.

maartet 2y ago

Reminds me of this gem from the previous decade: https://www.chrisstucchio.com/blog/2013/hadoop_hatred.html https://www.chrisstucchio.com/blog/2013/hadoop_hatred.html (Don't use Hadoop - your data isn't that big) Which, of course, was discussed on HN back then: https://news.ycombinator.com/item?id=6398650 https://news.ycombinator.com/item?id=6398650

unholyguy001 2y ago

One of the problems with the article is BigQuery becomes astronomically expensive at mid to high volumes. So there is a strong incentive to keep the data in BigQuery manageable or to even move off it as data volumes get higher. Also larger enterprises don’t even use gcp all that much to begin with

tored 2y ago

Former prime minister of Sweden Stefan Löfven got wind about Big data buzzword back in 2015 and used it in one of his speeches, that this is the future, however he used the Swedish translation of Big data, stordata. That generated some funny quips about where is lilldata. https://www.aftonbladet.se/senastenytt/ttnyheter/inrikes/a/8wXyzr/lofvens-industrikansler-pa-vag https://www.aftonbladet.se/senastenytt/ttnyheter/inrikes/a/8...

xiaodai 2y ago

Big memory is eating big data.

angarg12 2y ago

Previous years I would have completely agreed with this post. Nowadays with the AI and ML craze I'm not so sure. I've seen plenty of companies using using vast amounts of data to train ML models to incorporate to their products. Definitely more data that can be handle by a traditional DB, and well into Big Data territory. This isn't a value judgement about whether that's a good idea, just an observation from talking with many tech companies doing ML. This definitely feels like a bubble that will burst in due time, but for now ML is turbocharging Big Data.

zxxh 2y ago
Can you explain? It seems like you’re just using certain words without context.
int_19h 2y ago
But do they need to query that data?

deleted 2y ago

[deleted]

breckognize 2y ago

99.9%+ of data sets fit on an SSD, and 99%+ fit in memory. [1] This is the thesis for https://rowzero.io https://rowzero.io. We provide a real spreadsheet interface on top of these data sets, which gives you all the richness and interactivity that affords. In the rare cases you need more than that, you can hire engineers. The rest of the time, a spreadsheet is all you need. [1] I made these up.

BiboAlpha 2y ago

One of the major challenges with Big Data is that it often fails to deliver real business value, instead offering only misleading promises.

mavili 2y ago

Big Data hasn't been about storage, I thought it was always about processing. Guy obviously knows his stuff but I got the impression he stressed more about storage and how that's cheap and easy these days. When he does mention processing/computing, he mentions that most of the time people end up only querying recent data (ie small chunk of actual data they hold) but that bears the question: is querying only small chunk of data what businesses need, or are they doing it because querying the whole dataset is just not manageable? In other words, if processing all data at once was as easy as querying the most recent X percent, would most businesses still choose to only query the small chunk? I think there lies the answer whether Big Data (processing) is needed or not.

siliconc0w 2y ago

Storing data in object and querying from compute caching what you can basically scales until your queries are too expensive for a single node.

idunnoman1222 2y ago

I love how the way he talks about this. His paycheque from big-data is what is dead. His service offering was always bullshit.

fredstar 2y ago

I am in a software services company for more than 15 years. And to be honest, a lot of these big topics have always been some kind of sales talk or door opener. You write a white paper, nominate an 'expert' in your team and use these things in conversations with clients. Sure some trends are way more real and useful then others. But for me the article hits the nail on its head.

deleted 2y ago

[deleted]

aorloff 2y ago

What is dead is the notion that some ever expanding data lake is a mine full of gems and not a cost center.

schindlabua 2y ago

Used to work at a company that produced 20 gigs of analytics every day which is probably the biggest data I'll ever work on. My junior project was writing some data crunching jobs that did aggregations batched and in real time, and store the result in parquet blobs in azure. My boss was smart enough to have stakeholder meetings where they regularly discussed what to keep and what to throw away, and with some smart algorithms we were able to compress all that data down into like 200MB per day. We loaded the last 2 months into an sql server and the last 2 years further aggregated into another, and the whole company used the data in excel to do queries on it in reasonable time. The big data is rotting away on tape storage in case they ever need it in the future. My boss got a lot of stuff right and I learned a lot, though I only realized that in hindsight. Dude was a bad manager but he knew his data.

dang 2y ago

Related: Big data is dead - https://news.ycombinator.com/item?id=34694926 https://news.ycombinator.com/item?id=34694926 - Feb 2023 (433 comments) Big Data Is Dead - https://news.ycombinator.com/item?id=33631561 https://news.ycombinator.com/item?id=33631561 - Nov 2022 (7 comments)

kurts_mustache 2y ago

What was the source data here? It seems like a lot of the graphs are just the author's intuitive feel rather than being backed by any sort of hard data. Did I miss it in there somewhere?

lkdfjlkdfjlg 2y ago

TLDR: he used to say that you needed the thing he's selling. He changed his mind now and you need the opposite, which he's selling.

hsuduebc2 2y ago

I'm always waiting for this phrase for every one of the actual buzzwords.

rldjbpin 2y ago

"_ is dead" titles rarely represent reality. same case here. more accurately, "big data" or its main paradigms are not for everyone. on the other hand, most teams are still unable to reasonably consolidate their data to leverage the benefits of these tools. "AI" will have a similar fate in a few years time for similar reasons.

taylodl 2y ago

To me, the best thing to come out of the Big Data era was Spark Streaming. That tool provides a nice architecture for processing a stream of data in near real-time, from multiple data sources no less, that we didn't have before. Spark in general was a great tool to get out of the Big Data era.

drumdance 2y ago

I was CTO of an analytics startup in the early days of MongoDB hype. We built on MySQL. One of our competitors was MixPanel. We were having scaling issues with one particular customer, so I looked into the various solutions. They all looked pretty cool, but I didn't like that the developer communities were small compared to MySQL and PostgreSQL. So I ended up sticking with MySQL and building a simple sharding mechanism that allowed us to move big customers to their own server if necessary. It worked great. While I was working on that, MixPanel published a long, highly detailed blog post about why they had moved to MongoDB. That led me to have second thoughts. "If these guys are doing it, maybe we should too?" But I was pretty far down the sharding development path and couldn't justify making such a wholesale change based on one blog post. Fast forward about nine months, and MixPanel published a new blog post about why they had moved off MongoDB. Vindication! Ever since then I've had conservative disposition when it comes to deploying new technologies.