4 ms·
Talk about perfect timing! I was looking for something just like this for a project in my team. We had been using this setup where a huge chunk of the data was
by FridgeSeal 8y ago
Talk about perfect timing!
I was looking for something just like this for a project in my team. We had been using this setup where a huge chunk of the data was being stored in triplicate: some of it in ES, some more of it in another database and finally the whole dataset in our data warehouse.
Hopefully I can use this to only provide the index + full text capability and just use the warehouse itself as the main db because the query performance is similar enough and the warehouse is criminally underused for what we pay for it.
- dannycastonguay 8y agoCurious to know if you considered https://www.algolia.com/ https://www.algolia.com/ as well?
- mellab 8y agoAlgolia is a beautiful product but it’s expensive. At just 1 mil items you’re already paying 500 a month
- winrid 8y agoWhat's preventing you from using ES for everything? Slow writes?
- FridgeSeal 8y agoWrite speed is fine, it’s more the fact that the dataset is reasonably large, and to run an instance with enough capacity and nodes (even with spill to disk), is silly expensive.
- mverwijs 8y agoBeing in a simular situation what do you consider "reasonably large"?
- FridgeSeal 8y ago250-300GB. Not large by absolute standards sure, but large enough to cause issues. I’m sure there’s some kind of solution that involves re-architecting the ES cluster and indices and re-architecting the data flows and stuff. But if our options are go through all that, or seriously slim down our architecture and costs by just running Sonic + our data warehouse, I’m definitely going to give it a go. After all, worst comes to worst we can go down the re-architecting ES route if Sonic doesn’t work out. ¯\_(ツ)_/¯
- Xylakant 8y agoI’d be curious what your expectations and constraints are, but from my experience of running clusters in the double digit TB-Size my ballpark figure for that amount of data would be 2 medium size data nodes and a small tiebreaker. Alternatively, if you can live with the reduced resilience and availability, even a single node might just do. Depends on the expectated churn though, ES really does not like document updates.
- FridgeSeal 8y agoIs that double digit TB on ElasticSearch?
- Xylakant 8y agoYes, certainly.
- cinbun8 8y agoThat does not sound like a good idea. You can't even maintain a quorum of 2 replicas with n=3 on a cluster like that. Losing one data node would be disastrous.
- Xylakant 8y agoThat’s really not how ES replica works. The quorum is formed on the master eligible nodes (hence a tie-breaker) and is only required to elect a master. The elected master designates a primary shard and as many replica as you configure. However, replica shards are replica and may lag. There’s no read quorum or reconciliation or anything happening. If a primary fails, an (in-sync, depending on the version of ES) replica is auto-promoted. The master keeps track of in-sync replica and you can request that writes are on a number of replica before a write returns, but still, no true quorum. You can absolutely run 2 data/master-eligible nodes plus a single master-eligible tie-breaker node as a safe configuration. The only constraint is that you should have an uneven number of master-eligibile node to avoid a split brain. You also need to understand what the resilience guarantees are for any given number of replica (roughly: each replica allows for the loss of a single random node) and how many replica you can allocate on a given cluster (at most one per data node). That would allow you to run a 2-datanode cluster in a configuration that survives the loss of one node.