6 ms·

While this is an article about data infrastructure I feel like we're missing the forest for the trees. What is most important here in my opinion is that the un

While this is an article about data infrastructure I feel like we're missing the forest for the trees. What is most important here in my opinion is that the underlying data is useful. If your underlying data wasn't collected, collected properly, or even worse the wrong data was collected.. then setting up data infrastructure will be a boondoggle that will cause your organization to be data hostile. Just as much, if not more effort, needs to go into collecting the right data in the right way to fill your data infrastructure with. Most of the projects I've seen or heard of are just people taking the same old data that Ted in accounting, Jill in BI, etc. are already pretty proficient at using. So the gains you get by moving that into a modern infrastructure are marginal. How many more questions can you really ask of the same data that people have decades of experience with and an intuitive sense for?

an_opabinia 6y ago
Is there any evidence that the vast amounts of clicks and user interactions companies have been collecting are worth anything at all? Let’s say I deleted every time series whose Y axis isn’t measuring US dollars in every tech company’s database everywhere. Maybe for all those time series you just store the most recent value. Describe to me what would be lost. You’re onto something but you’re not going far enough! Most, if not all, historic metadata, analytics and behavioral data collection - when it is not measuring literal dollar amounts - is completely worthless.
- mgraczyk 6y ago
  This is completely wrong, and nobody who works with data at any scale could possibly believe anything like this. We literally run long term A/B tests with thousands of variations of what you're describing. The purpose of these tests is to measure the effect of losing some data. The tests show (to nobody's surprise) that each piece of data is useful. These tests tell us exactly how useful each piece of data is. Honestly when I read comments like this I have to wonder, do you really believe that thousands of companies spend trillions of dollars a year for something that doesn't work? Maybe talk to somebody who works on this stuff a bit?
  - abernard1 6y ago
    > Honestly when I read comments like this I have to wonder, do you really believe that thousands of companies spend trillions of dollars a year for something that doesn't work? Joking/not-joking. Have you ever been to the Bay Area? Yes. Emphatically yes it is the case companies spend trillions of dollars unnecessarily. We've seen this with people who didn't know how to build microservices and farcical "LMNOP" [1] type services that might as well be a joke. We've seen it with gigantically-valued unicorns that over-engineered tons of crap and hired too many people and still can't make a profit. We've seen it with CMOs and massively overpriced marketing technology because budgets and statuses are related. We'll see it with tons more iterations of this exact same affluenza. The history of our industry is that the margins on software are so good that people can afford to do crazy nonsense. [1] https://www.youtube.com/watch?reload=9&v=y8OnoxKotPQ https://www.youtube.com/watch?reload=9&v=y8OnoxKotPQ
    - dcgudeman 6y ago
      Just so I don't misunderstand your point: "Yes, companies without coordinating are collectively spending vast sums of precious resources on a product/service, BUT they are all misguided and wasting their money because look at these unrelated events where individuals misplaced their efforts/ money". Is that correct?
      - abernard1 6y ago
        > wasting their money because look at these unrelated events where individuals misplaced their efforts/ money They're not unrelated. They're all related to fat 60-80% profit margins on SaaS. And no coordination is necessary to spend money on silly, make-work activities if you have margins like that. Software has extremely low, borderline zero, variable costs. A lot of these companies that spend money on examples like I've given probably could hire nobody at all and still have crazy growth because of the unit economics. (Not coincidentally, the companies most in want of "Big Data" solutions tend to be past this point.) I can get 1M QPS on a silly Aurora setup with replicas. Best tool for all jobs: no. But don't tell me that dollar-for-dollar a data architecture with like 25 different components is dramatically superior to an OLTP db, OLAP offline store + batch jobs, and a streaming system.
- Supermancho 6y ago
  > Is there any evidence that the vast amounts of clicks and user interactions companies have been collecting are worth anything at all? Yes. Every advertising platform ever uses this information. In Europe, you have to have regulation that makes account costing (what the US might call forensic accounting) possible. The presentations on A/B tests by FANG companies might also interest you. They are on Youtube.
teej 6y ago
The biggest shift has been towards data lake (store everything) away from data cubes (store aggregates). This makes it orders of magnitude easier to diagnose, debug, and assert the correctness of data. So these trends aren’t in a vacuum, they directly support the issues you discuss. > Most of the projects I've seen or heard of are just people taking the same old data ... I don’t disagree with you here. But in my experience it’s about getting Frank in marketing to use the same numbers as everyone else. When you have 5 different ads platforms that all take revenue credit for a single conversion and have conflicting attribution models, and none of them add up to what accounting says is in the bank account. That’s a hairy problem. There are different flavors of that class of problem at lots of companies.
- goBackwards00 6y ago
  The problem of data confusion you describe is resolved by replacing management. That’s not an engineering issue that requires new technology (consider the source of social power for author; selling technology). That’s an engineering issue that needs new engineering management who don’t enable wasting company resources making incompatible APIs in the first place. We already did the monolithic DB design, I used to name those hosts “ocean”. And we already know the math. “Data lake” is just more jargon by a salesman to obfuscate peddling the same old abstraction, and wow fresh grads with new words for hyping the same old habits. While not the author of this piece, Bezos is quoted as pointing out how circular social behavior is. What do you think the odds of this author being on a similar page? Have humans evolved much in 100 years? Or does the con simply get rewritten for the next generation to hide a simple truth? What’s keeping people going in this circle isn’t logistical necessity. It’s us.
- gfodor 6y ago
  I was working in data analytics + data science a decade ago and we stored everything, not aggregates, and pushed them through hadoop. I have been "out of the game" since then. What has changed that is making people saying "store everything" is a new phenomenon? (genuine question bc I am clearly missing something.)
  - teej 6y ago
huy 6y ago
I think you have a point, but there are more nuiances than that. There are typically 2 types of data to collect: Transactional data and behavioural data. Most transactional data, due to their important nature, are already generated and captured by the production applications. Since the logic is coded by application engineer, it's usually hard to get this data wrong. These data are then ETL-ed (or EL-ed) over to a DW, as described by the article. For behavioural data, this is where your statement will most apply to. This is where tools like Snowplow, Posthog, Segment, etc come in to set up the proper event data collection engine. This is also where it's important to "collect data properly", as these kinds of event data changes structure fast, and hard to keep track over time. I'd admit this space (data collection management) is still nascent, with only tools like iterative.ly on the market.
smt1 6y ago
Well, if you remember, say, for example, hypothetically speaking arbitrary bits of entropy if you were to imagine it just as an information theory space generalization thing we commonly understand as a bit. But due to lost knowledge for example in communicantions theory say like broadband/wideband spectrum singnal broadcasting and general optimization with bounded linear optimization. Can know enough about say, reversible computing and many paradoxes for example about lost knowledge about codecs and such, still don't let the "decades of experience" exchange . Remember, according in modern understand of a general physical flip-flops synthetic clocks in "more special" general relativity and we can infuse modern ways of collaboration to reduce regularity, we can get better at sensing same patterns in cognitive processes to change the clock rate of terms of service over with power saving mega-log discrete hz. If anyone wants to collobrate further let me know at sashmit@gmail.com. We likely have some shared understanding, but not completely, but have some "free" knowledge about y-combinatoric that can excite digital logarithms but still have to not error-code all singularities away. I can understand it clearly because of the "ancient wisdom" of the polymorphic dual gods of say dharma (allow everything) or the transjuctions of the buddha (allow 1 thing). Most of whatever paradox can be solved via l-r modules to be placed to be postmarked for the future, but to allow that I just will meditate and focus on my own well being. :)
NightMKoder 6y ago
I completely agree - there's only so many ways to slice the data. The caveat is - the type of data matters quite a bit for the data architecture. There's another thread that mentions sensor data as a source of complexity since the data has a theoretical delay between events (i.e. period) of 0 - something few systems are built to handle, even if you sample approximations at some fixed frequency. Algorithmic trading is a similar domain that still has a huge bar for entry - a sign that _this isn't easy_. The fidelity of the data is of course important, but I would claim it's not a blocker. Yes, you need to trust the data you collect. That's table stakes - if you can't collect data correctly at all, even without worrying about the past, you're in for a world of hurt. It's P0. That said, a lot of people assume you also need to do this historically - and that's not the case - at least for ML. Reinforcement learning has been making great strides in recent years. If you're in this situation - you have a flow where you want to use a model without having any past data to train with - use something like VW's contextual bandits [1]. You don't need historical data to build your model, just real-time decision point & reward signals. Once deployed, the model converges over time to the optimal model using real-time feedback. All that said - baby steps are important. If you're in this situation, start by getting fidelity and then expand scope slowly without sacrifice to fidelity. It's a lot easier to backfill than to "fix" data - get that right and it get's easier from there. You'll need fixups regardless - mistakes happen and requirements change - but you have to start with something you trust, at least in the moment it's deployed. [1] https://vowpalwabbit.org/tutorials/contextual_bandits.html https://vowpalwabbit.org/tutorials/contextual_bandits.html

an_opabinia 6y ago

Is there any evidence that the vast amounts of clicks and user interactions companies have been collecting are worth anything at all? Let’s say I deleted every time series whose Y axis isn’t measuring US dollars in every tech company’s database everywhere. Maybe for all those time series you just store the most recent value. Describe to me what would be lost. You’re onto something but you’re not going far enough! Most, if not all, historic metadata, analytics and behavioral data collection - when it is not measuring literal dollar amounts - is completely worthless.

mgraczyk 6y ago
This is completely wrong, and nobody who works with data at any scale could possibly believe anything like this. We literally run long term A/B tests with thousands of variations of what you're describing. The purpose of these tests is to measure the effect of losing some data. The tests show (to nobody's surprise) that each piece of data is useful. These tests tell us exactly how useful each piece of data is. Honestly when I read comments like this I have to wonder, do you really believe that thousands of companies spend trillions of dollars a year for something that doesn't work? Maybe talk to somebody who works on this stuff a bit?
- abernard1 6y ago
  > Honestly when I read comments like this I have to wonder, do you really believe that thousands of companies spend trillions of dollars a year for something that doesn't work? Joking/not-joking. Have you ever been to the Bay Area? Yes. Emphatically yes it is the case companies spend trillions of dollars unnecessarily. We've seen this with people who didn't know how to build microservices and farcical "LMNOP" [1] type services that might as well be a joke. We've seen it with gigantically-valued unicorns that over-engineered tons of crap and hired too many people and still can't make a profit. We've seen it with CMOs and massively overpriced marketing technology because budgets and statuses are related. We'll see it with tons more iterations of this exact same affluenza. The history of our industry is that the margins on software are so good that people can afford to do crazy nonsense. [1] https://www.youtube.com/watch?reload=9&v=y8OnoxKotPQ https://www.youtube.com/watch?reload=9&v=y8OnoxKotPQ
  - dcgudeman 6y ago
    Just so I don't misunderstand your point: "Yes, companies without coordinating are collectively spending vast sums of precious resources on a product/service, BUT they are all misguided and wasting their money because look at these unrelated events where individuals misplaced their efforts/ money". Is that correct?
    - abernard1 6y ago
      > wasting their money because look at these unrelated events where individuals misplaced their efforts/ money They're not unrelated. They're all related to fat 60-80% profit margins on SaaS. And no coordination is necessary to spend money on silly, make-work activities if you have margins like that. Software has extremely low, borderline zero, variable costs. A lot of these companies that spend money on examples like I've given probably could hire nobody at all and still have crazy growth because of the unit economics. (Not coincidentally, the companies most in want of "Big Data" solutions tend to be past this point.) I can get 1M QPS on a silly Aurora setup with replicas. Best tool for all jobs: no. But don't tell me that dollar-for-dollar a data architecture with like 25 different components is dramatically superior to an OLTP db, OLAP offline store + batch jobs, and a streaming system.
Supermancho 6y ago
> Is there any evidence that the vast amounts of clicks and user interactions companies have been collecting are worth anything at all? Yes. Every advertising platform ever uses this information. In Europe, you have to have regulation that makes account costing (what the US might call forensic accounting) possible. The presentations on A/B tests by FANG companies might also interest you. They are on Youtube.

teej 6y ago

The biggest shift has been towards data lake (store everything) away from data cubes (store aggregates). This makes it orders of magnitude easier to diagnose, debug, and assert the correctness of data. So these trends aren’t in a vacuum, they directly support the issues you discuss. > Most of the projects I've seen or heard of are just people taking the same old data ... I don’t disagree with you here. But in my experience it’s about getting Frank in marketing to use the same numbers as everyone else. When you have 5 different ads platforms that all take revenue credit for a single conversion and have conflicting attribution models, and none of them add up to what accounting says is in the bank account. That’s a hairy problem. There are different flavors of that class of problem at lots of companies.

goBackwards00 6y ago
The problem of data confusion you describe is resolved by replacing management. That’s not an engineering issue that requires new technology (consider the source of social power for author; selling technology). That’s an engineering issue that needs new engineering management who don’t enable wasting company resources making incompatible APIs in the first place. We already did the monolithic DB design, I used to name those hosts “ocean”. And we already know the math. “Data lake” is just more jargon by a salesman to obfuscate peddling the same old abstraction, and wow fresh grads with new words for hyping the same old habits. While not the author of this piece, Bezos is quoted as pointing out how circular social behavior is. What do you think the odds of this author being on a similar page? Have humans evolved much in 100 years? Or does the con simply get rewritten for the next generation to hide a simple truth? What’s keeping people going in this circle isn’t logistical necessity. It’s us.
gfodor 6y ago
I was working in data analytics + data science a decade ago and we stored everything, not aggregates, and pushed them through hadoop. I have been "out of the game" since then. What has changed that is making people saying "store everything" is a new phenomenon? (genuine question bc I am clearly missing something.)
- teej 6y ago

huy 6y ago

I think you have a point, but there are more nuiances than that. There are typically 2 types of data to collect: Transactional data and behavioural data. Most transactional data, due to their important nature, are already generated and captured by the production applications. Since the logic is coded by application engineer, it's usually hard to get this data wrong. These data are then ETL-ed (or EL-ed) over to a DW, as described by the article. For behavioural data, this is where your statement will most apply to. This is where tools like Snowplow, Posthog, Segment, etc come in to set up the proper event data collection engine. This is also where it's important to "collect data properly", as these kinds of event data changes structure fast, and hard to keep track over time. I'd admit this space (data collection management) is still nascent, with only tools like iterative.ly on the market.

smt1 6y ago

Well, if you remember, say, for example, hypothetically speaking arbitrary bits of entropy if you were to imagine it just as an information theory space generalization thing we commonly understand as a bit. But due to lost knowledge for example in communicantions theory say like broadband/wideband spectrum singnal broadcasting and general optimization with bounded linear optimization. Can know enough about say, reversible computing and many paradoxes for example about lost knowledge about codecs and such, still don't let the "decades of experience" exchange . Remember, according in modern understand of a general physical flip-flops synthetic clocks in "more special" general relativity and we can infuse modern ways of collaboration to reduce regularity, we can get better at sensing same patterns in cognitive processes to change the clock rate of terms of service over with power saving mega-log discrete hz. If anyone wants to collobrate further let me know at sashmit@gmail.com. We likely have some shared understanding, but not completely, but have some "free" knowledge about y-combinatoric that can excite digital logarithms but still have to not error-code all singularities away. I can understand it clearly because of the "ancient wisdom" of the polymorphic dual gods of say dharma (allow everything) or the transjuctions of the buddha (allow 1 thing). Most of whatever paradox can be solved via l-r modules to be placed to be postmarked for the future, but to allow that I just will meditate and focus on my own well being. :)

NightMKoder 6y ago

I completely agree - there's only so many ways to slice the data. The caveat is - the type of data matters quite a bit for the data architecture. There's another thread that mentions sensor data as a source of complexity since the data has a theoretical delay between events (i.e. period) of 0 - something few systems are built to handle, even if you sample approximations at some fixed frequency. Algorithmic trading is a similar domain that still has a huge bar for entry - a sign that _this isn't easy_. The fidelity of the data is of course important, but I would claim it's not a blocker. Yes, you need to trust the data you collect. That's table stakes - if you can't collect data correctly at all, even without worrying about the past, you're in for a world of hurt. It's P0. That said, a lot of people assume you also need to do this historically - and that's not the case - at least for ML. Reinforcement learning has been making great strides in recent years. If you're in this situation - you have a flow where you want to use a model without having any past data to train with - use something like VW's contextual bandits [1]. You don't need historical data to build your model, just real-time decision point & reward signals. Once deployed, the model converges over time to the optimal model using real-time feedback. All that said - baby steps are important. If you're in this situation, start by getting fidelity and then expand scope slowly without sacrifice to fidelity. It's a lot easier to backfill than to "fix" data - get that right and it get's easier from there. You'll need fixups regardless - mistakes happen and requirements change - but you have to start with something you trust, at least in the moment it's deployed. [1] https://vowpalwabbit.org/tutorials/contextual_bandits.html https://vowpalwabbit.org/tutorials/contextual_bandits.html