3 ms·
The sync issues between an RDBMS and ElasticSearch are definitely the most annoying aspect of getting something like this working. In the case of my current pr
by jcadam 9y ago
The sync issues between an RDBMS and ElasticSearch are definitely the most annoying aspect of getting something like this working.
In the case of my current project, I've got an 'indexer' service whose sole job is to await messages from RabbitMQ to create/update/delete records to/from the index.
But sometimes an update may get missed for any number of reasons. So, I've got a systemd timer that fires off a little script to go through my audit tables in Postgres and make sure all changes since the last run were actually, in fact, synced to ElasticSearch.
And I'm still wondering if what I'm doing is 'good enough.' How else could this thing fail?
- amelius 9y agoWhat if your server crashes after you commit to the database, but before you submit to elastic-search? Even if submitting is flawless, this still presents a problem.
- jcadam 9y agoDepends on which server :D The Database, ElasticSearch, RabbitMQ, and the application services are on different VMs (in my case, Linodes). Committing to the database will trigger an entry (via a trigger function) in an audit table. So there is still a record of the transaction. After the system is restored to a healthy state, the next time the sync script is run (either automatically or manually if I'm feeling anxious), it should catch up. Of course, in a worst-case scenario, I could always rebuild the index :O.
- deleted 9y ago[deleted]
- AznHisoka 9y agoA server crash should be a rare event but if you reindex data regularly, it should not be a huge worry.
- bdcravens 9y agoIn Rails, the searchkick gem seems to do a good job of solving this pain point