3 ms·

What do you find are the tradeoffs vs. working with Elasticsearch or Solr? But great job, I've bookmarked your post to come back to and I'll give it a shot for

What do you find are the tradeoffs vs. working with Elasticsearch or Solr? But great job, I've bookmarked your post to come back to and I'll give it a shot for sure. Definitely very enticing.

simonw 9y ago
Elasticsearch is massively more powerful and flexible. The way I'm doing aggregation counts on top of PostgreSQL is very unlikely to scale to millions of documents. Elasticsearch pretty much acts as a real-time, horizontally scalable map/reduce engine so expensive aggregations get executed on multiple shards and the results are gathered back together. You may find my talk from DjangoCon this year interesting: https://speakerdeck.com/simon/the-denormalized-query-engine-design-pattern https://speakerdeck.com/simon/the-denormalized-query-engine-... / https://www.youtube.com/watch?v=NzcvewgqYog https://www.youtube.com/watch?v=NzcvewgqYog
- StavrosK 9y ago
  What type of data is ES good for? It's more than documents and full-text search, if I'm not mistaken, correct?
  - bebop 9y ago
    I have used it as a general data store, where I can accept eventual consistency. It is a very good distributed db if you are ok with the trade offs it imposes, mostly non-atomic commits to your cluster.
  - othersideofcoin 9y ago
    It also excels at storing logs for later aggregation and analysis.
    - StavrosK 9y ago
      Thank you, I was specifically wondering how good it is at data ingestion for warehousing. Good to know that's one of its strengths.
gt565k 9y ago
Generally speaking, you wouldn't want to run queries like this against the system of record (your database) in large applications where there are a lot of user queries, as that will impact performance. Hence why solutions like elasticsearch and apache solr exist. They scale very well horizontally, allowing you to spread the load between multiple nodes. Again, this comes down to what your use case is. If you have a tiny app with a relatively small number of users, using postgres for this functionality is probably ok.

What do you find are the tradeoffs vs. working with Elasticsearch or Solr? But great job, I've bookmarked your post to come back to and I'll give it a shot for sure. Definitely very enticing.

simonw 9y ago

Elasticsearch is massively more powerful and flexible. The way I'm doing aggregation counts on top of PostgreSQL is very unlikely to scale to millions of documents. Elasticsearch pretty much acts as a real-time, horizontally scalable map/reduce engine so expensive aggregations get executed on multiple shards and the results are gathered back together. You may find my talk from DjangoCon this year interesting: https://speakerdeck.com/simon/the-denormalized-query-engine-design-pattern https://speakerdeck.com/simon/the-denormalized-query-engine-... / https://www.youtube.com/watch?v=NzcvewgqYog https://www.youtube.com/watch?v=NzcvewgqYog

StavrosK 9y ago
What type of data is ES good for? It's more than documents and full-text search, if I'm not mistaken, correct?
- bebop 9y ago
  I have used it as a general data store, where I can accept eventual consistency. It is a very good distributed db if you are ok with the trade offs it imposes, mostly non-atomic commits to your cluster.
- othersideofcoin 9y ago
  It also excels at storing logs for later aggregation and analysis.
  - StavrosK 9y ago
    Thank you, I was specifically wondering how good it is at data ingestion for warehousing. Good to know that's one of its strengths.

gt565k 9y ago

Generally speaking, you wouldn't want to run queries like this against the system of record (your database) in large applications where there are a lot of user queries, as that will impact performance. Hence why solutions like elasticsearch and apache solr exist. They scale very well horizontally, allowing you to spread the load between multiple nodes. Again, this comes down to what your use case is. If you have a tiny app with a relatively small number of users, using postgres for this functionality is probably ok.