3 ms·
You indexed hundreds of millions of records in under an hour? Doubt.
by mdoms 5y ago
You indexed hundreds of millions of records in under an hour? Doubt.
- lmilcin 5y agoI have obviously not indexed the records. I just made an application that would index new and modified records and indexing existing ones would have to be performed separately. Also you need to realize it wouldn't be very wise to run your just written code directly on prod.
- orhmeh09 5y agoI could believe it — I’m pretty sure I’ve done it before using Postgres. Benchmarks appear to confirm this possibility —- 100 million towns indexed in about 35 mins using the standard b tree index, and 20 mins for BRIN on a laptop in 2019: https://blog.crunchydata.com/blog/postgresql-brin-indexes-big-data-performance-with-minimal-storage https://blog.crunchydata.com/blog/postgresql-brin-indexes-bi...
- lmilcin 5y agoElasticsearch is famous for slow indexing. This is basically because it is trading off slow indexing for fast search later. And data to be indexed was incredibly complex and rich. So every detail of every car you own, family member information, addresses, phone numbers, emails, identification documents, details of your contracts, and so on.
- orhmeh09 5y agoNeat! I’ve never used Elastic but might have a use for it un the near future. Do you recommend any resources?
- Andys 5y agoIf you insert in batches, elastic is quite fast. Also it shards and scales horizontally really well without the client having to know that. My cluster of 6 instances could index ~20k large, complex docs per second while still serving production query traffic.