3 ms·
That $12M/yr figure for indexing 1B documents seems way off to me.
by olalonde 10y ago
That $12M/yr figure for indexing 1B documents seems way off to me.
- svantana 10y agoYes and if you look at the footnote where the arithmetic is laid out, it seems quite confused. It conflates querying with indexing and also flatly assumes that each machine will index 5M pages, which seems low to me.
- astrieanna 10y agoToo high or too low?
- olalonde 10y agoToo high. See http://news.appbase.io/scaling-elasticsearch-writes/ http://news.appbase.io/scaling-elasticsearch-writes/ for example: > In doing these tests, we indexed over 1 billion documents under just two hours. The peak merge rate we saw was 129 MB/s, yes our ElasticSearch cluster on a 24 node setup is gobbling data at this speed. > Most mind boggling of all: At our peak sustained ingestion rate, we would have added over 13 billion documents in a single day and this would have cost us a mere USD 244. Their test documents are much smaller (100 bytes) than a typical web page but if you multiply that figure by 50 to get to 5kb documents (probably a realistic average web page size minus all markup and assets), you get 12200$ for 13 billion documents or 1220$ for 1.3 billion documents (assuming that scales linearly). I'm probably way off in my estimate myself but I'm convinced that a few thousand dollars is much closer than the proposed 12M$.