Oh yeah I love simply avoiding memory allocation at all costs and keeping things to the processor cache and streaming APIs. awk/sed is fantastic for this if you

Oh yeah I love simply avoiding memory allocation at all costs and keeping things to the processor cache and streaming APIs. awk/sed is fantastic for this if you're just working with CSV data, but I've done it in my own custom code processing hundreds of gigabytes of JSON in seconds as well. I think data scientists just aren't really hugely concerned with programming optimizations or bottlnecks or whatever. Most of them are just intermediate-level python programmers, and that's completely fine until they think they need a hadoop cluster for whatever they're doing and the costs start piling up.