5 ms·

Both indicators (along with python) that this is an article written by a yet another data scientist that thinks engineering is unecessary. Data science is a fi

Both indicators (along with python) that this is an article written by a yet another data scientist that thinks engineering is unecessary. Data science is a field that is plagued by individuals having very specific narrow experiences that are often easy to engineer making them think they are experts in data engineering. Vertical scalability has a hard bound, if your algorithm changes complexity it could become literally impossible to move forward. Horizontal scalability gives you options, this is why we like it, not just because we like to overengineer systems, but because we don't like embracing hard limitations on the systems we engineer, especially when we often don't know yet what problem we are solving. Engineers often don't get the liberty of solving extremely specific instances of problems like data scientists do.

realusername 7y ago
> Data science is a field that is plagued by individuals having very specific narrow experiences that are often easy to engineer making them think they are experts in data engineering. That's on point, some of the worst and most inefficient code I had to maintain was written by data scientists. They are for sure leagues above any of my data science knowledge in their respective field but the average engineering knowledge and best practices seems pretty low.
deleted 7y ago
[deleted]
kyllo 7y ago
Most of the time it's not that data scientists think engineering is "unnecessary," it's that they're expected to deliver analyses and models without the help of engineers, because the engineers are on another team, behind a massive backlog. So they make do with the tools they have.
- nautilus12 7y ago
  This is totally fair, why we need to bring the two disciplines closer together rather than have one try to obsolete the other.
Animats 7y ago
Um, yes. From the article: "The simplest and most common way to implement indexing is by naming files in a directory:" mydata/ 2019-Jan.csv 2019-Feb.csv 2019-Mar.csv 2019-Apr.csv Um.
- kyllo 7y ago
  This is time partitioning, not indexing, but it's commonly done in data lakes for raw telemetry/log data. mydata/ 2019/ 11/ 12/ 2019111200.csv 2019111201.csv Where each one of those CSV files could be 10-100 GB in size. Usually you want to process it into a columnar format like Parquet from there, though

realusername 7y ago

> Data science is a field that is plagued by individuals having very specific narrow experiences that are often easy to engineer making them think they are experts in data engineering. That's on point, some of the worst and most inefficient code I had to maintain was written by data scientists. They are for sure leagues above any of my data science knowledge in their respective field but the average engineering knowledge and best practices seems pretty low.

deleted 7y ago

[deleted]

kyllo 7y ago

Most of the time it's not that data scientists think engineering is "unnecessary," it's that they're expected to deliver analyses and models without the help of engineers, because the engineers are on another team, behind a massive backlog. So they make do with the tools they have.

nautilus12 7y ago
This is totally fair, why we need to bring the two disciplines closer together rather than have one try to obsolete the other.

Animats 7y ago

Um, yes. From the article: "The simplest and most common way to implement indexing is by naming files in a directory:" mydata/ 2019-Jan.csv 2019-Feb.csv 2019-Mar.csv 2019-Apr.csv Um.

kyllo 7y ago
This is time partitioning, not indexing, but it's commonly done in data lakes for raw telemetry/log data. mydata/ 2019/ 11/ 12/ 2019111200.csv 2019111201.csv Where each one of those CSV files could be 10-100 GB in size. Usually you want to process it into a columnar format like Parquet from there, though