4 ms·
I recently ran into this dilemma. I was tasked with writing a feature to allow importing a bunch of CSV into a system. Do I 1) stand up Azure Data Factory and
by shireboy 4y ago
I recently ran into this dilemma. I was tasked with writing a feature to allow importing a bunch of CSV into a system. Do I 1) stand up Azure Data Factory and build a pipeline to take the CSV from blob storage, massage it, and upsert the db or 2) write a few lines of C#+SQL to suck in the CSV and do the same? #2 is not quite as trivial as it may seem in real-world. Customer wants preview of data before update, undo capability, etc. But even so, I went with #2. I'm wary of doing things because they seem "easier" to me, but these huge data pipeline tools just seem like overkill at least for the use cases I'm tasked with.
- benjaminwootton 4y agoMost people don’t actually have a big data problem. If you can load and manipulate your CSV in memory within your app then it’s not worth all of that data infrastructure.
- RobinL 4y ago2) seems very reasonable to me. as a Python user, I'd have done something similar using the Python duckdb library (i.e. steering clear of any heavyweight tools)