4 ms·
Regarding 2), code like "for row in pandas.DataFrame(...).iterrows():" where the DataFrame was populated by some array-based reader is creating an indirection.
by dmw_ng 6y ago
Regarding 2), code like "for row in pandas.DataFrame(...).iterrows():" where the DataFrame was populated by some array-based reader is creating an indirection. Most good pandas code of course doesn't look like this, but it's regularly unavoidable, hence "There are use cases where either representation is preferred". Without specifying our use case, it is meaningless to talk about "faster" or "slower".
For 3) there inherently is no optimal solution to "a problem with large performance-correctness tradeoffs", additionally users do not care only about speed, for a huge number of applications, absolute reproduction of precision may be the difference between a good simulation and data science quackery. We saw elsewhere in the thread that Julia's float parsing code at least presently handles certain syntaxes incorrectly. Ignoring performance, this is criteria for deciding whether a solution is fit for purpose at all.
The overall point otherwise makes sense: most DS folk don't and maybe shouldn't care about this stuff, however this is HN, and elsewhere in the comments there are plenty of examples of people who do.
- StefanKarpinski 6y ago> there inherently is no optimal solution to "a problem with large performance-correctness tradeoffs" You seem to be under the misapprehension that mainstream CSV libraries parse float values with reduced precision in order to go faster. They do not. A decimal floating-point string represents a precise mathematical value and the only acceptable value for a CSV parser to produce is that value correctly rounded to a Float64 (or some other type if requested). Any CSV parser that does anything else is simply incorrect. > We saw elsewhere in the thread that Julia's float parsing code at least presently handles certain syntaxes incorrectly. You may be referring to a bug mentioned elsewhere that fields like `-123.` with a leading minus sign and a trailing decimal point are misinterpreted as strings. That is a bug, plain and simple. It has nothing to do with float parsing speed/accuracy tradeoffs.
- deleted 6y ago[deleted]