4 ms·

happily, i have multiple gigabytes of memory …

• philwelch 4y ago
  Sure, but a 40 GB SQLite database on a machine with 16 GB of RAM is not gonna be happy
  • qbasic_forever 4y ago
    You're not going to do better with pandas or similar tools. If it can't fit in memory, it's going to be painful. SQLite is the least painful in my experience, and it sets you up for working with the data in a proper DB like postgres or similar for when you get fed up with the memory constraints.
    • philwelch 4y ago
      I wouldn’t use pandas in that situation either.
  • kosherhurricane 4y ago
    In the original article, they were trying to read a 10GB file into 30GB RAM.