4 ms·
This is not true at all. Most libraries do not reimplement csv parsing - I’m saying only a select few libraries do this because they have very different design
by mlthoughts2018 6y ago
This is not true at all. Most libraries do not reimplement csv parsing - I’m saying only a select few libraries do this because they have very different design goals. For example optimizing for maximum rows per second parsed is a totally different design goal from optimizing to read into Spark DataFrame types to connect to Spark. Prioritizing one of these might come at the cost of the other, which is totally fine and having more than one library to choose from is a good thing.
> “ switching from C to python and then back to C would ruin performance.”
This is explicitly wrong in regards to Python. Optimizing Python often involves writing an implementation directly in C (or better, Cython) and autogenerating bindings that marshal between CPython native C types and the types of your extension module. This is extremely fast - as fast as calling native C code, there is no performance penalty for this.