3 ms·
>Would you say the big data threshold moves every year? It moves with Moore's law. Big data is anything that cannot reasonably fit into memory for a single ser
by ramesh31 4y ago
>Would you say the big data threshold moves every year?
It moves with Moore's law. Big data is anything that cannot reasonably fit into memory for a single server, so yes that number is well over 1TB now.
- orangepurple 4y agoI know this isn't the correct definition but I think of "big data" as the set of data which takes me more than 15 minutes to query on average with a moderately complex Postgres SQL join on well indexed information. I use JSONB in Postgres regularly and have indices on that too. So far I have gotten really far with increasing Postgres work_mem to a gig or more, a fast SSD, and strategically placed materialized views. These kinds of operations in Pandas make my computer billow smoke by comparison.