3 ms·
I was helping out a friend of mine in the risk department of a Big 4; he was parsing CSV data from a client's portfolio. Once he started parsing it, he was gett
by grenoire 6y ago
I was helping out a friend of mine in the risk department of a Big 4; he was parsing CSV data from a client's portfolio. Once he started parsing it, he was getting random NaNs (pandas' nan type, to be more accurate).
I couldn't get access to the original dataset but the column gave it away. Namibia's 2-letter ISO country code is NA—which happens to be in pandas' default list of NaN equivalent strings.
It was a headache and a half...
- grenoire 6y agoVerbatim from the docs, on read-csv: na_valuesscalar, str, list-like, or dict, default None Additional strings to recognize as NA/NaN. If dict passed, specific per-column NA values. By default the following values are interpreted as NaN: ‘’, ‘#N/A’, ‘#N/A N/A’, ‘#NA’, ‘-1.#IND’, ‘-1.#QNAN’, ‘-NaN’, ‘-nan’, ‘1.#IND’, ‘1.#QNAN’, ‘<NA>’, ‘N/A’, ‘NA’, ‘NULL’, ‘NaN’, ‘n/a’, ‘nan’, ‘null’. You fix it by using `keep_default_na=False`, by the way.
- mseepgood 6y agoA Ms True also broke Apple's iCloud: https://twitter.com/RachelTrue/status/1365461618977476610 https://twitter.com/RachelTrue/status/1365461618977476610
- grenoire 6y agoThat looks like an interesting hard-coded check, I wonder what it intended to fix.
- fanf2 6y agoThere’s some analysis in this twitter thread: https://twitter.com/badedgecases/status/1368362392573317120 https://twitter.com/badedgecases/status/1368362392573317120 tl;dr: there are a bunch of fields of various types that arrive as strings, and they get coerced but without paying attention to which field should have which type