3 ms·
Offtopic: Does anyone know a quick way (that does not require manual work) to convert a simple mysql db into sqlite?
by 101008 3y ago
Offtopic: Does anyone know a quick way (that does not require manual work) to convert a simple mysql db into sqlite?
- jitl 3y agoDump a CSV, import to SQLite. All your columns will be type TEXT and your nulls will be “” (empty string) but hey at least you didn’t do any manual work
- hiAndrewQuinn 3y agoSimone's csvs-to-sqlite would be the next level up, I think it does actual type conversions.
- jitl 3y agoNice, I needed to do this task recently for large CSVs (15gb) and didn’t come across this tool. Under the hood it manipulates the CSV using pandas data frames, then finally dumps the data frame into sqlite using `dataframe.to_sql(sqlite_conn)`. I wonder how fast this is, if it has to take each row from the data frame through Python it can’t be that quick but maybe there’s something smart happening under the hood that can punch a native-to-native pathway through the Python abstractions. I ended up writing my own in Golang since the ones I found in C and other languages didn’t use threads or did too much memory copying.
- hiAndrewQuinn 3y agoHa, yes, Go is probably going to be at least an order of magnitude speedup from Simon's Python based tool. I'd be curious to give your tool a test run if the code is up anywhere.