4 ms·

Could you possibly post the importer? I'm interested in replicating/learning from this. Thanks!

• vessenes 11y ago
  Sure. https://gist.github.com/vessenes/1485b5e88e00c898199e https://gist.github.com/vessenes/1485b5e88e00c898199e It's dead simple, though. I'm not sure you'll learn much. :) The three things I did (one for coding simplicity, one for speed and one for correctness): * Used shlex to parse out the quotes and commas in the csv file. shlex seems very slow, anecdotally, and I believe this is most of the time in the import. * I used executemany to do multiple inserts into sqlite. This saves some time because the sqlite module compiles the insert query each time. In my case, I made the simple trade off to do this once per customer file. There's probably some optimal number of lines to use depending on system memory and other configuration knobs, but not worth it in this case to benchmark. * I committed the inserts every hundred customers; in general you don't want a transaction to get too large with most databases, committing will clear that buffer out. I have no idea how performant sqlite is in this case, I just picked a random number for the commits.
  • Y0s 11y ago
    Thanks! How did you set up the schema in the sqlite db, as I get the following error on running import.py: sqlite3.OperationalError: no such table: txs