3 ms·
Pruning the file is done the same way as logrotate, or even with logrotate. You don't track state. When the loader starts up, you spool through the whole file a
by madhadron 7y ago
Pruning the file is done the same way as logrotate, or even with logrotate. You don't track state. When the loader starts up, you spool through the whole file and use ON DUPLICATE KEY IGNORE in your INSERT statement.
- sneak 7y agoThe nice thing about the queue is that you can do fanout (eg multiple SQS subscribers to the SNS topic or similar) and have multiple parallel ingestors or change out the backend consumer/db injector without the producers knowing anything about it. You also get HA for free; your solution depends on local node disk persistence.
- madhadron 7y ago> your solution depends on local node disk persistence. We have reached a point where people think that local node disk persistence is an exotic property.
- sneak 7y agoThe Chaos Monkey is real. I don’t like losing data unnecessarily. Queues are cheap and are usually run by Other People. It’s basically free reliability. It’s also nice not having to specifically test the mode where the db is offline; this gives free maintenance windows with literally no alternate procedure on producers (and no additional testing). If we’re gonna sign our lives over to Amazon and Google anyway, we might as well get some reliability out of the exchange.