4 ms·

It seems like setting up and syncing an entire separate database for search wouldn't exactly be "trivial"...

threeseed 10y ago
Elasticsearch can be installed in seconds with most package managers and syncing is trivial depending on what you want to index. You can use a JDBC river for bulk indexing or rely on trigger's.
- nugator 10y ago
  "Syncing is trivial" Have you got any links to help me set up PostgreSQL to Elasticsearch sync?
  - yummyfajitas 10y ago
    I don't know about elasticsearch specifically, but here are postgres -> x instructions using LISTEN/NOTIFY. https://www.chrisstucchio.com/blog/2013/postgres_external_triggers.html https://www.chrisstucchio.com/blog/2013/postgres_external_tr...
- irrational 10y ago
  If it's so trivial, why don't you just put the instructions here so the rest of us can learn from your experience?
  - deleted 10y ago
    [deleted]
- djur 10y ago
  Rivers were deprecated and removed from 2.0. The syncing story isn't terrible but it's not as simple as it used to be.
  - patsplat 10y ago
    that's disappointing :-(
  - patsplat 10y ago
    Looks like the new syncing story involves logstash as an ETL platform https://www.elastic.co/guide/en/logstash/current/deploying-and-scaling.html https://www.elastic.co/guide/en/logstash/current/deploying-a...
    - ehsanu1 10y ago
      Eh, that's more for logs specifically. The "syncing story" is something that depends on your technology platform and specific situation. I adopted ES when it had rivers, but never even considered using them for syncing.
      - patsplat 10y ago
        For what it's worth Logstash is cited as a replacement for rivers in the deprecation announcement. https://www.elastic.co/blog/deprecating-rivers https://www.elastic.co/blog/deprecating-rivers
- ngrilly 10y ago
  > syncing is trivial depending on what you want to index Can you explain how you keep your Elasticsearch index consistent with your main database, in face of system failure (for example a network partition or a machine hard reboot)? This is usually the non-trivial part.
deleted 10y ago
[deleted]
eric_khun 10y ago
I've migrated last week 500M documents (100GB) of data from mongodb to ES. It's quite simple and straightforward. Used the bulk API. Just make sure to have a good machine (16vCPU) and use parallelization and just wait. after 24hours, it just works.
- mozumder 10y ago
  That's only 200 bytes per document. Can they be really considered 'documents' or are they actually database fields?
  - woah 10y ago
    That's an average.
- ngrilly 10y ago
  How do keep the ES index consistent with MongoDB?

Elasticsearch can be installed in seconds with most package managers and syncing is trivial depending on what you want to index. You can use a JDBC river for bulk indexing or rely on trigger's.

nugator 10y ago
"Syncing is trivial" Have you got any links to help me set up PostgreSQL to Elasticsearch sync?
- yummyfajitas 10y ago
  I don't know about elasticsearch specifically, but here are postgres -> x instructions using LISTEN/NOTIFY. https://www.chrisstucchio.com/blog/2013/postgres_external_triggers.html https://www.chrisstucchio.com/blog/2013/postgres_external_tr...
irrational 10y ago
If it's so trivial, why don't you just put the instructions here so the rest of us can learn from your experience?
- deleted 10y ago
  [deleted]
djur 10y ago
Rivers were deprecated and removed from 2.0. The syncing story isn't terrible but it's not as simple as it used to be.
- patsplat 10y ago
  that's disappointing :-(
- patsplat 10y ago
  Looks like the new syncing story involves logstash as an ETL platform https://www.elastic.co/guide/en/logstash/current/deploying-and-scaling.html https://www.elastic.co/guide/en/logstash/current/deploying-a...
  - ehsanu1 10y ago
    Eh, that's more for logs specifically. The "syncing story" is something that depends on your technology platform and specific situation. I adopted ES when it had rivers, but never even considered using them for syncing.
    - patsplat 10y ago
      For what it's worth Logstash is cited as a replacement for rivers in the deprecation announcement. https://www.elastic.co/blog/deprecating-rivers https://www.elastic.co/blog/deprecating-rivers
ngrilly 10y ago
> syncing is trivial depending on what you want to index Can you explain how you keep your Elasticsearch index consistent with your main database, in face of system failure (for example a network partition or a machine hard reboot)? This is usually the non-trivial part.

deleted 10y ago

[deleted]

eric_khun 10y ago

I've migrated last week 500M documents (100GB) of data from mongodb to ES. It's quite simple and straightforward. Used the bulk API. Just make sure to have a good machine (16vCPU) and use parallelization and just wait. after 24hours, it just works.

mozumder 10y ago
That's only 200 bytes per document. Can they be really considered 'documents' or are they actually database fields?
- woah 10y ago
  That's an average.
ngrilly 10y ago
How do keep the ES index consistent with MongoDB?