3 ms·
A 6T hard drive and Pandas will cost you a couple hundred bucks, one time purchase, and then last you for years (and several other data analysis jobs). It also
by nostrademons 2y ago
A 6T hard drive and Pandas will cost you a couple hundred bucks, one time purchase, and then last you for years (and several other data analysis jobs). It also doesn't require that you be connected to the Internet, doesn't require that you trust 3rd-party services, and is often faster (even in execution time) than spooling up BigQuery.
You can always save an intermediate data set partitioned and massaged into whatever format makes subsequent queries easy, but that's usually application-dependent, and so you want that control over how you actually store your intermediate results.
- data-ottawa 2y agoI wouldn't make a purchase of either without knowing a bit more about the lifecycle and requirements. If you only needed this once, the BQ approach requires very little setup and many places already have a billing account. If this is recurring then you need to figure out what the ownership plan of the hard drive is (what's it connected to, who updates this computer, what happens when it goes down, etc.).