4 ms·
How does your writer process deal with failures / tracking state of what it has written / prune the file when it doesn't need old data anymore? You don't have
by awinder 7y ago
How does your writer process deal with failures / tracking state of what it has written / prune the file when it doesn't need old data anymore? You don't have to use SQS but this problem has tons of available options, I wouldn't resort to rolling your own (if anything just pick up something that works off a leveldb/sqlite/etc file and has already implemented all this boring stuff for you)
- madhadron 7y agoPruning the file is done the same way as logrotate, or even with logrotate. You don't track state. When the loader starts up, you spool through the whole file and use ON DUPLICATE KEY IGNORE in your INSERT statement.
- sneak 7y agoThe nice thing about the queue is that you can do fanout (eg multiple SQS subscribers to the SNS topic or similar) and have multiple parallel ingestors or change out the backend consumer/db injector without the producers knowing anything about it. You also get HA for free; your solution depends on local node disk persistence.
- madhadron 7y ago> your solution depends on local node disk persistence. We have reached a point where people think that local node disk persistence is an exotic property.
- sneak 7y agoThe Chaos Monkey is real. I don’t like losing data unnecessarily. Queues are cheap and are usually run by Other People. It’s basically free reliability. It’s also nice not having to specifically test the mode where the db is offline; this gives free maintenance windows with literally no alternate procedure on producers (and no additional testing). If we’re gonna sign our lives over to Amazon and Google anyway, we might as well get some reliability out of the exchange.