4 ms·
I ran into an issue recently specific to Pandas and python3 with unicode. pd.read_excel(filepath) will read an entire dataset even if it contains unicode chara
by oarabbus_ 7y ago
I ran into an issue recently specific to Pandas and python3 with unicode.
pd.read_excel(filepath) will read an entire dataset even if it contains unicode characters.
pd.ExcelFile() silently drops(!!) unicode rows. The resulting object will simply skip unicode-containing rows (in ANY column) them without even a warning.
For example, if you had an excel file:
word
---
"hello"
"hello"
你早
你早
"hello"
then pd.read_excel() would give you a dataframe with 5 rows. ExcelFile() on the other hand would return (silently!) a dataframe with only the first two and the last row.
Maybe this is a pandas issue, not a python issue, but it was really horrendous to debug for such a long time only to realize this was the issue.
- j88439h84 7y agoIs there an issue for this in the bug tracker?
- oarabbus_ 7y agoI searched, and this is the closest thing (https://github.com/pandas-dev/pandas/issues/11503 https://github.com/pandas-dev/pandas/issues/11503) but it is not the issue I experienced. I'm not sure how to submit a bug report, to be honest.