2 ms·
This is fantastic, thanks. My solution is a lot less smart - I loop through every record and keep track of which potential types I've seen for each column: htt
by simonw 4y ago
This is fantastic, thanks.
My solution is a lot less smart - I loop through every record and keep track of which potential types I've seen for each column: https://sqlite-utils.datasette.io/en/latest/python-api.html#detecting-column-types-using-typetracker https://sqlite-utils.datasette.io/en/latest/python-api.html#...
Implementation here: https://github.com/simonw/sqlite-utils/blob/3fbe8a784cc2f3fa0bfa8612fec9752ff9068a2b/sqlite_utils/utils.py#L318-L331 https://github.com/simonw/sqlite-utils/blob/3fbe8a784cc2f3fa...
- mytherin 4y agoThat works and is similar to what DuckDB does for small CSV files. We have the sampling step primarily for larger CSV files, when you might not want to do two passes over the file. This way we can keep the benefits of streamed file reading while offering type inference that "just works" most of the time without a major performance penalty.