3 ms·
It would appear us SQLite zealots have encountered the final boss. Petabytes uncompressed would be tricky if you need to slice those columns. SQLite caps out a
by bob1029 4y ago
It would appear us SQLite zealots have encountered the final boss.
Petabytes uncompressed would be tricky if you need to slice those columns. SQLite caps out at ~281 terabytes of storage before it can't track any additional pages.
None of this is to say you couldn't partition the data across a lot of SQLite instances in varying ways. I will probably take a shot at it this weekend. Looking to see just how unlimited my AT&T fiber connection is anyways.
- topspin 4y ago> It would appear us SQLite zealots have encountered the final boss. That's cute. :) There isn't much value in feeding it all into a conventional RDBMS. OLAPs and columnar stores are what is needed here. But first it will need a great deal of grooming and ETL work.
- gizmodo59 4y agoYeah.. It would be much easier to copy the data to S3/any object storage (better to convert it into a columnar format like parquet) and query it directly using a SQL on lake engine like Dremio or Athena or S3Select would work too.
- salawat 4y ago>It would appear us SQLite zealots have encountered the final boss. Just wait. It's actually a multi-boss fight, since you have to wrangle the Pharmacy Benefits Management datasets, plus Medispan, plus Medicare, plus all the MedicAid datasets, plus VA. Are you and all your mightiest boxen bad enough dudes to make sense of the entire U.S. Healthcare industry? <Actuary Stormrage in the background> You are not prepared!