7 ms·
If it's a only few thousand enterprise users, I'd actually say log to a bog standard relational database from server side like MySQL or Postgres. Think through
by sfifs 7y ago
If it's a only few thousand enterprise users, I'd actually say log to a bog standard relational database from server side like MySQL or Postgres. Think through table schemas for everything you're going to log and make sure primary keys and nomenclature for everything talk to each other. Virtually any analytics platform or software talks to standard databases. Record as much as you can because analytics use cases typically get generated following first data collection and analysis and are iterative - so you also want to build flexibility.
- sneak 7y agoSuggestion: send the events to an elastic queue (like SQS) first, and have a pair of ingestion processes do the actual INSERTs when the db is available. Then you can take the db offline for maintenance and upgrades and not lose data.
- madhadron 7y agoAlternately, log to disk and have another process ingest and write to the database. Appending to files is an amazing persistent queue.
- awinder 7y agoHow does your writer process deal with failures / tracking state of what it has written / prune the file when it doesn't need old data anymore? You don't have to use SQS but this problem has tons of available options, I wouldn't resort to rolling your own (if anything just pick up something that works off a leveldb/sqlite/etc file and has already implemented all this boring stuff for you)
- madhadron 7y agoPruning the file is done the same way as logrotate, or even with logrotate. You don't track state. When the loader starts up, you spool through the whole file and use ON DUPLICATE KEY IGNORE in your INSERT statement.
- sneak 7y agoThe nice thing about the queue is that you can do fanout (eg multiple SQS subscribers to the SNS topic or similar) and have multiple parallel ingestors or change out the backend consumer/db injector without the producers knowing anything about it. You also get HA for free; your solution depends on local node disk persistence.
- madhadron 7y ago> your solution depends on local node disk persistence. We have reached a point where people think that local node disk persistence is an exotic property.
- sneak 7y agoThe Chaos Monkey is real. I don’t like losing data unnecessarily. Queues are cheap and are usually run by Other People. It’s basically free reliability. It’s also nice not having to specifically test the mode where the db is offline; this gives free maintenance windows with literally no alternate procedure on producers (and no additional testing). If we’re gonna sign our lives over to Amazon and Google anyway, we might as well get some reliability out of the exchange.