17 ms·

Emerging Architectures for Modern Data Infrastructure

431 points by soumyadeb 6y ago

malisper 6y ago
For a post detailing the modern data infrastructure I'm surprised they intentionally leave out SaaS analytics tools. I find this especially surprising given a16z has invested >$65M into Mixpanel. Based on my experience working at an analytics company and running one myself, what this post misses out is that an increasing number of people working with data today are not engineers. These people can range from product managers who are trying to figure out what features the company should focus on building, marketers to figure out how to drive more traffic to their website, or even the CEO trying to understand how their business as a whole is doing. For that reason, you'll still see many companies pay for full stack analytics tools (Mixpanel, Amplitude, Heap) in addition to building out their own data stack internally. It's becoming more and more important that the data is accessible to everyone at your company including the non-technical users. If you try to get everyone to use your own in-house built system, that's not going to happen.
- soumyadeb 6y ago
  That's a great point. On similar vein, marketing teams too are increasingly data driven and would tools like Braze, CustomerIO etc to run personalized data driven campaigns. Support teams are using tools like GainSight All these tools need to be fed data about user behavior - from apps, server backends, other tools etc. It's a messy data connection problem, not just one way from SaaS to warehouse. Mobile App->SaaS; SaaS->SaaS; Warehouse->SaaS; SaaS->Warehouse and so on.
  - shostack 6y ago
    Don't forget the challenges around identity resolution and privacy compliance to properly join it all up effectively and accurately.
    - soumyadeb 6y ago
      Indeed. Even when you have the full identity graph in warehouse and just want to assign a cannonical-ID (by doing a transitive closure), it is not easy in SQL. We wrote a blog on it (sorry for the shameless plug) https://rudderstack.com/blog/identity-graph-and-identity-resolution-in-sql/ https://rudderstack.com/blog/identity-graph-and-identity-res... Creating the ID graph is a next level problem altogether!!. How do you know a record in Salesforce is the same as the anonymous visitor on your website. Requires joining across at-least 3 (possibly more IDs) - anonymousID, userID email (if the user signs up) and Salesforce record email. Should the data pipe do this automatically? If not, what API abstraction should be exposed to the user?
- whoisjuan 6y ago
  I don’t think Mixpanel fits here. Mixpanel it’s just one end-to-end suite, mostly behavioral data that is captured from user sessions or user derived events/sub-events. Basically web analytics. The whole point of data infrastructure is that sometimes you’re collecting data from the most random places. Many of that data is not necessarily user behavior. Sometimes it’s things like temperatures, latencies, CPU usage or instrument tallies. Sometimes it’s a stream of minute to minute weather data or timings or anything, really. Besides many companies have been collecting data for decades but it all live in silos where it can’t be used for anything. Mixpanel can’t capture all that data, or query it, or analyze it. Mixpanel is just capturing a super small subset of web event data and it happens to provide an analysis suite on top that data they collect. That’s why Segment shows up in this list instead. They help to move a lot of siloed data into a common systems. Mixpanel is just another source of data. You need something like Snowflake to put everything together and be able to do queries across multiple datasets.
dm03514 6y ago
I'm really excited about the state of data infrastructure and the emergence of the data lake. I feel like the technical aspects of data engineering is reduced to getting data into some cloud storage (s3) as parquet. Transforms are "solved" using ELT from the data lake, or streaming using kafka/spark. I think executing this in orgs with legacy data technologies is hard but it is much more a people problem than a tech problem. In orgs that have achieved this foundation it's really cool to see the business and analytic impact to the company.
- chrisweekly 6y ago
  "it is much more a people problem than a tech problem" ^ This holds true for nearly every aspect of nearly every company.
- spullara 6y ago
  Snowflake (and others) will let you either pull that in and query it or as an external query that queries it in place. You can, if it makes sense for your use case, now just T from the data lake.
ethanwillis 6y ago
While this is an article about data infrastructure I feel like we're missing the forest for the trees. What is most important here in my opinion is that the underlying data is useful. If your underlying data wasn't collected, collected properly, or even worse the wrong data was collected.. then setting up data infrastructure will be a boondoggle that will cause your organization to be data hostile. Just as much, if not more effort, needs to go into collecting the right data in the right way to fill your data infrastructure with. Most of the projects I've seen or heard of are just people taking the same old data that Ted in accounting, Jill in BI, etc. are already pretty proficient at using. So the gains you get by moving that into a modern infrastructure are marginal. How many more questions can you really ask of the same data that people have decades of experience with and an intuitive sense for?
- an_opabinia 6y ago
  Is there any evidence that the vast amounts of clicks and user interactions companies have been collecting are worth anything at all? Let’s say I deleted every time series whose Y axis isn’t measuring US dollars in every tech company’s database everywhere. Maybe for all those time series you just store the most recent value. Describe to me what would be lost. You’re onto something but you’re not going far enough! Most, if not all, historic metadata, analytics and behavioral data collection - when it is not measuring literal dollar amounts - is completely worthless.
  - mgraczyk 6y ago
    This is completely wrong, and nobody who works with data at any scale could possibly believe anything like this. We literally run long term A/B tests with thousands of variations of what you're describing. The purpose of these tests is to measure the effect of losing some data. The tests show (to nobody's surprise) that each piece of data is useful. These tests tell us exactly how useful each piece of data is. Honestly when I read comments like this I have to wonder, do you really believe that thousands of companies spend trillions of dollars a year for something that doesn't work? Maybe talk to somebody who works on this stuff a bit?
fouc 6y ago
The recent HN threads about excel made me think there's definitely room for a new kind of excel that works well for big data.
- webmaven 6y ago
  > The recent HN threads about excel made me think there's definitely room for a new kind of excel that works well for big data. Check out Google Connected Sheets: https://cloudblog.withgoogle.com/products/g-suite/connected-sheets-is-generally-available/ https://cloudblog.withgoogle.com/products/g-suite/connected-...
- pm 6y ago
  I remember reading about Looker, before Google bought them out. I never used it myself, but it may have fit the description.
- manigandham 6y ago
  That's just SQL. And there are dozens of charting/visualization/business-intelligence vendors to do whatever you want beyond or on top of that SQL structure.
- plaidfuji 6y ago
  This, to me, is now the rate limiting step in this architecture; there are probably 1000x as many people who can operate in Excel than people who can operate on a “data stack”. Yes, the fundamental goal of these data stacks is to enable insight and decisions “at scale”.. but beyond that you have probably hundreds or thousands of employees who just need to do quick analyses for one-off decisions that can be handled by Excel. But there’s usually a benefit to those analyses being “operationalized” and integrated into the broader architecture.. having a live connection to the central database, and having results piped back... so many Excel spreadsheets get emailed back and forth, completely out of the stack’s purview. Will MS modify Excel 365 fast enough to meet this need? Will another spreadsheet program disrupt Excel’s dominance? Will another player come in with the ability to “ingest” arbitrary Excel files? Another major issue is Excel’s massive failure when it comes to handling uncertainty in data. I’ll be curious to see how it all plays out.
huy 6y ago
For those who're interested in learning more about the history and evolution of data infrastructure/BI - basically why and how it has come to this stage - check out this short guidebook [1] that my colleagues and I put together a few months back. It goes into details how much relevance the practices of the past (OLAP, Kimball's modeling) has with the current changes in by the cloud era (MPP, cheap storage/compute, etc). Chapter 4 will be most interesting for HN audience: It walks through the different waves of data adoption ever since BI was invented in the 60-70s. https://holistics.io/books/setup-analytics/ https://holistics.io/books/setup-analytics/
- sradman 6y ago
  This sounds like an in-depth discussion of what the a16z document calls Blueprint 1: Modern Business Intelligence. I don’t know if the other two blueprints for Multimodal and AI are explored.
cageface 6y ago
While I think data science is a very interesting field with a lot of beneficial applications it also seems to be the one that's right at the heart of a lot of the negative impact some tech is having on society right now. I seriously considered specializing in it for a while but ultimately decided it was too likely I'd be asked to work on things that make me uncomfortable.
- malux85 6y ago
  Power(ful tools) can be wielded for good or evil, the courageous thing to do is to learn it AND act ethically, not shy away from it. Otherwise the spoils of war go to the unethical evil because they are now unchallenged.
  - cageface 6y ago
    Maybe so but I'm not really in a position to martyr myself professionally right now so I'm just avoiding it instead.
    - malux85 6y ago
      False dichotomy
      - cageface 6y ago
        How so? If I level up on data science and then go work somewhere as a data scientist and refuse to work on the tasks assigned to me I don't see that working out so well in my performance reviews. To the downvoters - grow up and learn to present an argument.
        coddle-hark 6y ago
        You could say the same thing about lots of knowledge though: AI, economics, behavioral psychology, crypto currencies etc. I think most people would agree that there’s lots of positive applications that you could use your data science skills for. But if you can’t then good for you for staying out of the field.
m3kw9 6y ago
I wonder how many of those companies in the proposed architecture have A16z as investors?
- huy 6y ago
  I counted 6. Fivetran, dbt, Preset (Superset/Airflow), Sisu, Imply and Databricks. Though, as someone who's in this space a while, I think they did a decently fair job at articulating the 'modern' data infrastructure landscape.
utasinger 6y ago
Are you looking to hire a hacker for any type of hacking services dm on t witter @ harnyhvck101 or email directly hackanonymously0617@gmail . com.. Help is here
jandrewrogers 6y ago
This article has a large gap in the story: it ignores sensor data sources, which are both the highest velocity and highest volume data models by multiple orders of magnitude. They have become ubiquitous in diverse, medium-sized industrial enterprises and it has turned them into some of the largest customers of cloud providers due to the data intensity. Organizations routinely spend $100M/year to deal with this data, and the workloads are literally growing exponentially. Almost no one provides tooling and platforms that address it. (This is not idle speculation, I’ve run just about every platform you can name through lab tests in anger. They are uniformly inadequate for these data models, everyone relies on bespoke platforms designed by specialists if they can afford the tariff.) If you add real-time sensor data sources to the mix, the rest of the architecture model kind of falls apart. Requirements upstream have cascading effects on architecture downstream. The deficiencies are both technical and economic. First, you need a single ordinary server (like EC2) to be able to ingest, transform, and store about 10M events per second continuously, while making that data fully online for basic queries. You can’t afford the latency overhead and systems cost of these being separate systems. You need this efficiency because the raw source may be 1B events per second; even at that rate, you’ll need a fantastic cluster architecture. Most of the open source platforms tap out at 100k events per second per server for these kinds of mixed workloads and no one can afford to run 20k+ servers because the software architecture is throughput limited (never mind the cluster management aspects at that scale). Second, storage cost and data motion are the primary culprits that make these data models uneconomical. Open source tends to be profligate in these dimensions, and when you routinely operate on endless petabytes of data, it makes the entire enterprise problematic. To be fair, this is not to blame open source platforms per se, they were never designed for workloads where storage and latency costs were critical for viability. It can be done, but it was never a priority and you would design the software very differently if it was. I will make a prediction. When software that can address sensor data models becomes a platform instead of bespoke, it will eat the lunch of a lot of adjacent data platforms that aren’t targeted at sensor data for a simple reason: the extreme operational efficiency of data infrastructure required to handle sensor data models applies just as much to any other data model, there simply hasn’t been an existential economic incentive to build it for those other data models. I've seen this happen several times; someone pays for bespoke sensor data infrastructure and realizes they can adapt it to run their large-scale web analytics (or whatever) many times faster and at a fraction of the infrastructure cost, even though it wasn't designed for it. And it works.
fluffy87 6y ago
Citation needed? We connect all our sensors to an edge AI Server that handles sensor data, and only uploads to the cloud what’s actually relevant. It works quite well, and there are many OEMs that offer such systems, with accelerators for inference, sensor data compression, 5G, etc.
msolujic 6y ago
Good start for this vast and complex topic. One thing that pops out here as missing is Data Mesh [1] It is emerging pattern for complex data management and data exchange between multiple products and product components/services. [1] https://martinfowler.com/articles/data-monolith-to-mesh.html https://martinfowler.com/articles/data-monolith-to-mesh.html
cblconfederate 6y ago
What's the point of data hoarding? Intelligent systems in nature ingest the data, learn, and discard them
tuckerconnelly 6y ago
The ELT (rather than ETL) insight was really cool, hadn't heard of that before. Unless though, you're on a massive, massive scale, Just Use Postgres, and write your ETL (ELT now?) queues normally. Keep It Simple Stupid.
nicholast 6y ago
I considered this piece as sort of a loose validation that the Automunge library is filling an unmet need for data scientists. Intended for tabular data preprocessing in the steps immediately preceding the application of machine learning.
ca123 6y ago
Great article, but surprising that it does not mention or use the concept of DataOps. Even Gartner has recently written at length about the role of DataOps [1], and of course, we at Composable [2] are biased as they just name us as a Cool Vendor in DataOps [3]. [1] https://www.gartner.com/en/documents/3970916/introducing-dataops-into-your-data-management-discipline https://www.gartner.com/en/documents/3970916/introducing-dat... [2] https://composable.ai https://composable.ai [3] https://www.gartner.com/en/documents/3991447/cool-vendors-in-dataops https://www.gartner.com/en/documents/3991447/cool-vendors-in...

431 points by soumyadeb 6y ago

For a post detailing the modern data infrastructure I'm surprised they intentionally leave out SaaS analytics tools. I find this especially surprising given a16z has invested >$65M into Mixpanel. Based on my experience working at an analytics company and running one myself, what this post misses out is that an increasing number of people working with data today are not engineers. These people can range from product managers who are trying to figure out what features the company should focus on building, marketers to figure out how to drive more traffic to their website, or even the CEO trying to understand how their business as a whole is doing. For that reason, you'll still see many companies pay for full stack analytics tools (Mixpanel, Amplitude, Heap) in addition to building out their own data stack internally. It's becoming more and more important that the data is accessible to everyone at your company including the non-technical users. If you try to get everyone to use your own in-house built system, that's not going to happen.

soumyadeb 6y ago
That's a great point. On similar vein, marketing teams too are increasingly data driven and would tools like Braze, CustomerIO etc to run personalized data driven campaigns. Support teams are using tools like GainSight All these tools need to be fed data about user behavior - from apps, server backends, other tools etc. It's a messy data connection problem, not just one way from SaaS to warehouse. Mobile App->SaaS; SaaS->SaaS; Warehouse->SaaS; SaaS->Warehouse and so on.
- shostack 6y ago
  Don't forget the challenges around identity resolution and privacy compliance to properly join it all up effectively and accurately.
  - soumyadeb 6y ago
    Indeed. Even when you have the full identity graph in warehouse and just want to assign a cannonical-ID (by doing a transitive closure), it is not easy in SQL. We wrote a blog on it (sorry for the shameless plug) https://rudderstack.com/blog/identity-graph-and-identity-resolution-in-sql/ https://rudderstack.com/blog/identity-graph-and-identity-res... Creating the ID graph is a next level problem altogether!!. How do you know a record in Salesforce is the same as the anonymous visitor on your website. Requires joining across at-least 3 (possibly more IDs) - anonymousID, userID email (if the user signs up) and Salesforce record email. Should the data pipe do this automatically? If not, what API abstraction should be exposed to the user?
whoisjuan 6y ago
I don’t think Mixpanel fits here. Mixpanel it’s just one end-to-end suite, mostly behavioral data that is captured from user sessions or user derived events/sub-events. Basically web analytics. The whole point of data infrastructure is that sometimes you’re collecting data from the most random places. Many of that data is not necessarily user behavior. Sometimes it’s things like temperatures, latencies, CPU usage or instrument tallies. Sometimes it’s a stream of minute to minute weather data or timings or anything, really. Besides many companies have been collecting data for decades but it all live in silos where it can’t be used for anything. Mixpanel can’t capture all that data, or query it, or analyze it. Mixpanel is just capturing a super small subset of web event data and it happens to provide an analysis suite on top that data they collect. That’s why Segment shows up in this list instead. They help to move a lot of siloed data into a common systems. Mixpanel is just another source of data. You need something like Snowflake to put everything together and be able to do queries across multiple datasets.

dm03514 6y ago

I'm really excited about the state of data infrastructure and the emergence of the data lake. I feel like the technical aspects of data engineering is reduced to getting data into some cloud storage (s3) as parquet. Transforms are "solved" using ELT from the data lake, or streaming using kafka/spark. I think executing this in orgs with legacy data technologies is hard but it is much more a people problem than a tech problem. In orgs that have achieved this foundation it's really cool to see the business and analytic impact to the company.

chrisweekly 6y ago
"it is much more a people problem than a tech problem" ^ This holds true for nearly every aspect of nearly every company.
spullara 6y ago
Snowflake (and others) will let you either pull that in and query it or as an external query that queries it in place. You can, if it makes sense for your use case, now just T from the data lake.

ethanwillis 6y ago

While this is an article about data infrastructure I feel like we're missing the forest for the trees. What is most important here in my opinion is that the underlying data is useful. If your underlying data wasn't collected, collected properly, or even worse the wrong data was collected.. then setting up data infrastructure will be a boondoggle that will cause your organization to be data hostile. Just as much, if not more effort, needs to go into collecting the right data in the right way to fill your data infrastructure with. Most of the projects I've seen or heard of are just people taking the same old data that Ted in accounting, Jill in BI, etc. are already pretty proficient at using. So the gains you get by moving that into a modern infrastructure are marginal. How many more questions can you really ask of the same data that people have decades of experience with and an intuitive sense for?

an_opabinia 6y ago
Is there any evidence that the vast amounts of clicks and user interactions companies have been collecting are worth anything at all? Let’s say I deleted every time series whose Y axis isn’t measuring US dollars in every tech company’s database everywhere. Maybe for all those time series you just store the most recent value. Describe to me what would be lost. You’re onto something but you’re not going far enough! Most, if not all, historic metadata, analytics and behavioral data collection - when it is not measuring literal dollar amounts - is completely worthless.
- mgraczyk 6y ago
  This is completely wrong, and nobody who works with data at any scale could possibly believe anything like this. We literally run long term A/B tests with thousands of variations of what you're describing. The purpose of these tests is to measure the effect of losing some data. The tests show (to nobody's surprise) that each piece of data is useful. These tests tell us exactly how useful each piece of data is. Honestly when I read comments like this I have to wonder, do you really believe that thousands of companies spend trillions of dollars a year for something that doesn't work? Maybe talk to somebody who works on this stuff a bit?

fouc 6y ago

The recent HN threads about excel made me think there's definitely room for a new kind of excel that works well for big data.

webmaven 6y ago
> The recent HN threads about excel made me think there's definitely room for a new kind of excel that works well for big data. Check out Google Connected Sheets: https://cloudblog.withgoogle.com/products/g-suite/connected-sheets-is-generally-available/ https://cloudblog.withgoogle.com/products/g-suite/connected-...
pm 6y ago
I remember reading about Looker, before Google bought them out. I never used it myself, but it may have fit the description.
manigandham 6y ago
That's just SQL. And there are dozens of charting/visualization/business-intelligence vendors to do whatever you want beyond or on top of that SQL structure.
plaidfuji 6y ago
This, to me, is now the rate limiting step in this architecture; there are probably 1000x as many people who can operate in Excel than people who can operate on a “data stack”. Yes, the fundamental goal of these data stacks is to enable insight and decisions “at scale”.. but beyond that you have probably hundreds or thousands of employees who just need to do quick analyses for one-off decisions that can be handled by Excel. But there’s usually a benefit to those analyses being “operationalized” and integrated into the broader architecture.. having a live connection to the central database, and having results piped back... so many Excel spreadsheets get emailed back and forth, completely out of the stack’s purview. Will MS modify Excel 365 fast enough to meet this need? Will another spreadsheet program disrupt Excel’s dominance? Will another player come in with the ability to “ingest” arbitrary Excel files? Another major issue is Excel’s massive failure when it comes to handling uncertainty in data. I’ll be curious to see how it all plays out.

huy 6y ago

For those who're interested in learning more about the history and evolution of data infrastructure/BI - basically why and how it has come to this stage - check out this short guidebook [1] that my colleagues and I put together a few months back. It goes into details how much relevance the practices of the past (OLAP, Kimball's modeling) has with the current changes in by the cloud era (MPP, cheap storage/compute, etc). Chapter 4 will be most interesting for HN audience: It walks through the different waves of data adoption ever since BI was invented in the 60-70s. https://holistics.io/books/setup-analytics/ https://holistics.io/books/setup-analytics/

sradman 6y ago
This sounds like an in-depth discussion of what the a16z document calls Blueprint 1: Modern Business Intelligence. I don’t know if the other two blueprints for Multimodal and AI are explored.

cageface 6y ago

While I think data science is a very interesting field with a lot of beneficial applications it also seems to be the one that's right at the heart of a lot of the negative impact some tech is having on society right now. I seriously considered specializing in it for a while but ultimately decided it was too likely I'd be asked to work on things that make me uncomfortable.

malux85 6y ago
Power(ful tools) can be wielded for good or evil, the courageous thing to do is to learn it AND act ethically, not shy away from it. Otherwise the spoils of war go to the unethical evil because they are now unchallenged.
- cageface 6y ago
  Maybe so but I'm not really in a position to martyr myself professionally right now so I'm just avoiding it instead.
  - malux85 6y ago
    False dichotomy
    - cageface 6y ago
      How so? If I level up on data science and then go work somewhere as a data scientist and refuse to work on the tasks assigned to me I don't see that working out so well in my performance reviews. To the downvoters - grow up and learn to present an argument.
      - coddle-hark 6y ago
        You could say the same thing about lots of knowledge though: AI, economics, behavioral psychology, crypto currencies etc. I think most people would agree that there’s lots of positive applications that you could use your data science skills for. But if you can’t then good for you for staying out of the field.

m3kw9 6y ago

I wonder how many of those companies in the proposed architecture have A16z as investors?

huy 6y ago
I counted 6. Fivetran, dbt, Preset (Superset/Airflow), Sisu, Imply and Databricks. Though, as someone who's in this space a while, I think they did a decently fair job at articulating the 'modern' data infrastructure landscape.

utasinger 6y ago

Are you looking to hire a hacker for any type of hacking services dm on t witter @ harnyhvck101 or email directly hackanonymously0617@gmail . com.. Help is here

jandrewrogers 6y ago

This article has a large gap in the story: it ignores sensor data sources, which are both the highest velocity and highest volume data models by multiple orders of magnitude. They have become ubiquitous in diverse, medium-sized industrial enterprises and it has turned them into some of the largest customers of cloud providers due to the data intensity. Organizations routinely spend $100M/year to deal with this data, and the workloads are literally growing exponentially. Almost no one provides tooling and platforms that address it. (This is not idle speculation, I’ve run just about every platform you can name through lab tests in anger. They are uniformly inadequate for these data models, everyone relies on bespoke platforms designed by specialists if they can afford the tariff.) If you add real-time sensor data sources to the mix, the rest of the architecture model kind of falls apart. Requirements upstream have cascading effects on architecture downstream. The deficiencies are both technical and economic. First, you need a single ordinary server (like EC2) to be able to ingest, transform, and store about 10M events per second continuously, while making that data fully online for basic queries. You can’t afford the latency overhead and systems cost of these being separate systems. You need this efficiency because the raw source may be 1B events per second; even at that rate, you’ll need a fantastic cluster architecture. Most of the open source platforms tap out at 100k events per second per server for these kinds of mixed workloads and no one can afford to run 20k+ servers because the software architecture is throughput limited (never mind the cluster management aspects at that scale). Second, storage cost and data motion are the primary culprits that make these data models uneconomical. Open source tends to be profligate in these dimensions, and when you routinely operate on endless petabytes of data, it makes the entire enterprise problematic. To be fair, this is not to blame open source platforms per se, they were never designed for workloads where storage and latency costs were critical for viability. It can be done, but it was never a priority and you would design the software very differently if it was. I will make a prediction. When software that can address sensor data models becomes a platform instead of bespoke, it will eat the lunch of a lot of adjacent data platforms that aren’t targeted at sensor data for a simple reason: the extreme operational efficiency of data infrastructure required to handle sensor data models applies just as much to any other data model, there simply hasn’t been an existential economic incentive to build it for those other data models. I've seen this happen several times; someone pays for bespoke sensor data infrastructure and realizes they can adapt it to run their large-scale web analytics (or whatever) many times faster and at a fraction of the infrastructure cost, even though it wasn't designed for it. And it works.

fluffy87 6y ago

Citation needed? We connect all our sensors to an edge AI Server that handles sensor data, and only uploads to the cloud what’s actually relevant. It works quite well, and there are many OEMs that offer such systems, with accelerators for inference, sensor data compression, 5G, etc.

msolujic 6y ago

Good start for this vast and complex topic. One thing that pops out here as missing is Data Mesh [1] It is emerging pattern for complex data management and data exchange between multiple products and product components/services. [1] https://martinfowler.com/articles/data-monolith-to-mesh.html https://martinfowler.com/articles/data-monolith-to-mesh.html

cblconfederate 6y ago

What's the point of data hoarding? Intelligent systems in nature ingest the data, learn, and discard them

tuckerconnelly 6y ago

The ELT (rather than ETL) insight was really cool, hadn't heard of that before. Unless though, you're on a massive, massive scale, Just Use Postgres, and write your ETL (ELT now?) queues normally. Keep It Simple Stupid.

nicholast 6y ago

I considered this piece as sort of a loose validation that the Automunge library is filling an unmet need for data scientists. Intended for tabular data preprocessing in the steps immediately preceding the application of machine learning.

ca123 6y ago

Great article, but surprising that it does not mention or use the concept of DataOps. Even Gartner has recently written at length about the role of DataOps [1], and of course, we at Composable [2] are biased as they just name us as a Cool Vendor in DataOps [3]. [1] https://www.gartner.com/en/documents/3970916/introducing-dataops-into-your-data-management-discipline https://www.gartner.com/en/documents/3970916/introducing-dat... [2] https://composable.ai https://composable.ai [3] https://www.gartner.com/en/documents/3991447/cool-vendors-in-dataops https://www.gartner.com/en/documents/3991447/cool-vendors-in...