2 ms·
Call me crazy, but for something as simple as taking a CSV file and shoving its data into a database, why wouldn't you just use a combination of shell tools? Do
by benmanbs 8y ago
Call me crazy, but for something as simple as taking a CSV file and shoving its data into a database, why wouldn't you just use a combination of shell tools? Don't get me wrong, I'm a Java programmer, but sometimes you don't need the overhead of the JVM, and something like awk would save you lots of time and energy (and memory).
- rhombocombus 8y agoThat was my first thought, I do ETL work at a large insurance company and we use shell tools almost exclusively unless we need to do more complex transformations. Unix tools are really low overhead and work extremely well for this task. Also, you can parallelize a lot of them with xargs: https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html https://adamdrake.com/command-line-tools-can-be-235x-faster-...
- btbuildem 8y agoAbsolutely - use the simplest available approach first.