3 ms·

Seriously, just put everything in Postgres. You have so little data, you shouldn't even be thinking about an "analytics system". I have seen so many developers

Seriously, just put everything in Postgres. You have so little data, you shouldn't even be thinking about an "analytics system". I have seen so many developers over-engineer this exact problem. Forget about Kafka, Kinesis, Redshift, Airflow, Storm, Spark, Cassandra etc. You don't need them, not even close. Unless you want to add a bunch of expensive distributed systems and operational overhead for fun/resume building, they're going to waste your time and hurt your stability.

PaulHoule 7y ago
On top of that I would say that many specialized analytics systems are SQL-based, so even if you start out with a normal RDMBS you can upgrade to something that is super-efficient for your workload.
manishsharan 7y ago
wouldn't writing events from analytics sub-system into a SQL database put load on the DB even when it is not warranted ? My point is that if you are using a SQL db and most of the database is for business transactions, wouldn't logging events which are not mission critical , unnecessarily consume db resources ? Also, assuming that SQL is being used for storing analytics events, would you not cache events in a queue and then flush them to DB in a batch?
- dougk16 7y ago
  Typically you have a separate database just for analytics, exactly for the reason you described. You don't want to hold up actual business logic. And yes typically you also have a job queue for the same reason. So e.g. API calls don't take longer due to database event writes. If your needs are similar to the original questioner (still starting out, few thousand users), I'd say even the separate database isn't necessary, just the job queue. Heck even the job queue is probably optional at that scale, but most hosting platforms make job queues easy enough to integrate so I say why not.
- gfodor 7y ago
  We put unwarranted load on computers all the time, and nobody notices. They're good at that. Now, if you think the load we are talking about will actually be so much that humans will be affected by it, then it makes sense to reconsider. But at 1000 DAU, you could probably run the database (analytics logging included) on a Linux server from the 1990s and its CPU and disk would not be the transaction bottleneck. (With proper indexing ofc.)

PaulHoule 7y ago

On top of that I would say that many specialized analytics systems are SQL-based, so even if you start out with a normal RDMBS you can upgrade to something that is super-efficient for your workload.

manishsharan 7y ago

wouldn't writing events from analytics sub-system into a SQL database put load on the DB even when it is not warranted ? My point is that if you are using a SQL db and most of the database is for business transactions, wouldn't logging events which are not mission critical , unnecessarily consume db resources ? Also, assuming that SQL is being used for storing analytics events, would you not cache events in a queue and then flush them to DB in a batch?

dougk16 7y ago
Typically you have a separate database just for analytics, exactly for the reason you described. You don't want to hold up actual business logic. And yes typically you also have a job queue for the same reason. So e.g. API calls don't take longer due to database event writes. If your needs are similar to the original questioner (still starting out, few thousand users), I'd say even the separate database isn't necessary, just the job queue. Heck even the job queue is probably optional at that scale, but most hosting platforms make job queues easy enough to integrate so I say why not.
gfodor 7y ago
We put unwarranted load on computers all the time, and nobody notices. They're good at that. Now, if you think the load we are talking about will actually be so much that humans will be affected by it, then it makes sense to reconsider. But at 1000 DAU, you could probably run the database (analytics logging included) on a Linux server from the 1990s and its CPU and disk would not be the transaction bottleneck. (With proper indexing ofc.)