3 ms·
Not strictly today. But I discovered that there exists a special class of algorithms that are designed with the use case of streaming data to your program. I ju
by InfinityByTen 9mo ago
Not strictly today.
But I discovered that there exists a special class of algorithms that are designed with the use case of streaming data to your program. I just used one to get a uniformly distributed sample from a 10Gb log file.
I knew this was something coding interviews delved into: "if it doesn't fit in memory", but until like yesterday I never went down the rabbit hole. I have to say it was a nifty trick.
- dapperdrake 9mo agohttps://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html https://adamdrake.com/command-line-tools-can-be-235x-faster-...