3 ms·
I've joined projects for a few big data pipelines now and every single time people were using R (shudder) or Pandas for what ultimately could have been done wit
by CalRobert 4y ago
I've joined projects for a few big data pipelines now and every single time people were using R (shudder) or Pandas for what ultimately could have been done with SQL (ideally with DBT, which makes testing much easier). And every time it was because data scientists apparently thought every single problem needed R or Pandas and SQL was beneath them.
- benjaminwootton 4y agoFortunately the field is splitting into data engineers who build the pipelines and data scientists who analyse the results. I think that’s a good thing as the pipeline should look more like software engineering.
- CalRobert 4y agoUnfortunately I have found myself dealing with trying to "productionize" appallingly bad code from DS. https://news.ycombinator.com/item?id=33787270 https://news.ycombinator.com/item?id=33787270 has a good discussion on this.