3 ms·

craziest thing to me about this whole article is this one company has 40 billion social media posts!

spullara 8y ago
I personally have 25B just for fun / side projects / analysis. The surprising thing for me is that for them it is petabyte scale (25k+ each!) and for me they take up 2TB (80 bytes each, compressed). Must have a tremendous amount of metadata or elastic search is super inefficient.
- karlney 8y ago
  The social data are fairly small documents, but we index lots of other things as well that is larger in size. Another factor is our large replica count, especially for the hot data. We also have quite a lot of metadata.
- mikljohansson 8y ago
  Our whole dataset exported as compressed JSON is likely not much more than 100TB. The petabytes come from all the index datastructures Elasticsearch/Lucene builds, as well as the high replication factor needed to keep up with the query throughput. We index a lot of NLP and other enrichments on our documents. This also adds a lot of storage on top of the base text. And like Karl mentioned, we have lots of small social media documents available for analytics (currently 34B, sometime next year upward of 230B). But also 7+ billion documents of news, blogs and other long-text articles indexed, basically 10 years of all news media from the entire world online and available for analytics. We're building the https://fairhair.ai/ https://fairhair.ai/ data science platform to allow other companies to access, and run online search and analytics on top of this massive dataset and compute clusters. For example to embed analytics over this dataset into their own SaaS products
  - user5994461 8y ago
    In my experience, Elasticsearch is triple the size of the data. First is the actual json data in quasi plain text. Second is the _source field that duplicates the original input object (necessary for reindexing/rebuilding) Third is the _all field that duplicates the json data as text (only used for some text search, better disable it). Finally, the index is duplicated to replicas, at least one if you want any redundancy. Index compression with lz4 takes 20 or 30% off, new feature of elasticsearch v5.0, on by default.

I personally have 25B just for fun / side projects / analysis. The surprising thing for me is that for them it is petabyte scale (25k+ each!) and for me they take up 2TB (80 bytes each, compressed). Must have a tremendous amount of metadata or elastic search is super inefficient.

karlney 8y ago
The social data are fairly small documents, but we index lots of other things as well that is larger in size. Another factor is our large replica count, especially for the hot data. We also have quite a lot of metadata.
mikljohansson 8y ago
Our whole dataset exported as compressed JSON is likely not much more than 100TB. The petabytes come from all the index datastructures Elasticsearch/Lucene builds, as well as the high replication factor needed to keep up with the query throughput. We index a lot of NLP and other enrichments on our documents. This also adds a lot of storage on top of the base text. And like Karl mentioned, we have lots of small social media documents available for analytics (currently 34B, sometime next year upward of 230B). But also 7+ billion documents of news, blogs and other long-text articles indexed, basically 10 years of all news media from the entire world online and available for analytics. We're building the https://fairhair.ai/ https://fairhair.ai/ data science platform to allow other companies to access, and run online search and analytics on top of this massive dataset and compute clusters. For example to embed analytics over this dataset into their own SaaS products
- user5994461 8y ago
  In my experience, Elasticsearch is triple the size of the data. First is the actual json data in quasi plain text. Second is the _source field that duplicates the original input object (necessary for reindexing/rebuilding) Third is the _all field that duplicates the json data as text (only used for some text search, better disable it). Finally, the index is duplicated to replicas, at least one if you want any redundancy. Index compression with lz4 takes 20 or 30% off, new feature of elasticsearch v5.0, on by default.