4 ms·
I like the idea of using AWK for this. But you can give kdb/q a try. 250M rows is nothing for kdb, and it seemed that you can afford the license.
by co_dh 6y ago
I like the idea of using AWK for this.
But you can give kdb/q a try. 250M rows is nothing for kdb,
and it seemed that you can afford the license.
- lenkite 6y agokdb/q is amazing. I have seen some experienced folks make jaw dropping data computations at the drop of a hat on gigabyte sized data. Makes everything else look laughably puny. I suspect if it was open source, it would probably be the most popular big-data storage and computing platform.
- labelbias 6y agoHmm... 256M records is still not that much for postgres or mysql so it shouldn't be too much for kdb/q. Several gb of data also shouldn't be too much.
- greggyb 6y agokdb/q, or many other mature data platforms. 250M records isn't big for a lot of data platforms. I've seen plenty of solutions in traditional RDBMSs that would easily scale an order of magnitude larger than this, if not several. Moving into anything analytically focused, like an OLAP engine, or just a columnstore in an RDBMS easily gives you another order of magnitude or few.