7 ms·
What do you mean by "large amounts of data"? A friend of mine is a data analyst at a large market research company and analyzes more than 10 billion cookies a
by realtarget 10y ago
What do you mean by "large amounts of data"?
A friend of mine is a data analyst at a large market research company and analyzes more than 10 billion cookies a month using Hadoop from apache:
http://hadoop.apache.org/ http://hadoop.apache.org/
- nstart 10y agowell let's say you want to create a minute by minute report on URL's visited in an ecommerce site that has a large amount of usage. Every minute I want to process all those URLs and decide if I want to create a flash sale on the site. Since this was an example I just thought of, let's imagine that there are 400k URLs visited every minute. That would be a decent definition of what I meant by large amounts of data. Sorry for not making that clearer before. Your friend's example is pretty good. Curious how they work on the data in memory.
- ahoka 10y agoIt depends on how many unique URLs you have. If you have like a few hundreds or even thousand I would just use a simple hash table with counters to see what's a hot sale right now (I guess that's what you want in your example). If you have several millions than maybe I would put it into Redis or something similar where I can utilize ready-to-use HyperLogLog cardinality estimation.