4 ms·
This is great. I love using DuckDB to do initial analytics on new datasets we get at work. The issue I run into a lot is that our sources can give us data that
by hifikuno 3y ago
This is great. I love using DuckDB to do initial analytics on new datasets we get at work. The issue I run into a lot is that our sources can give us data that has non UTF-8 characters in it which DuckDB won't read so I spend a bit of time just trying to find the few non UTF-8 characters in a multi-GB file.
SQLite does not seem to care about the bogus characters so I would be really interested in using Stanchion and SQLite to see if it speeds up my exploration.
Also, I do not know how I have never known about SQLite extensions before now. I feel like I just found out that Narwhals are actually real.
- gregw2 3y agoRe non-UTF8 inputs, more could be said (resolve root encoding conversion issues with iconv), but see: https://unix.stackexchange.com/questions/201751/replace-non-printable-characters-in-perl-and-sed https://unix.stackexchange.com/questions/201751/replace-non-...