2 ms·

The big problem with stuffing logs in SQL is that a log search can bring down your app. You'll be tempted to implement log search via something like SELECT * FR

The big problem with stuffing logs in SQL is that a log search can bring down your app. You'll be tempted to implement log search via something like SELECT * FROM logs WHERE message LIKE "%query%" and your DB will fall over when the log table gets big enough. It's common to ingest logs into something like elasticsearch, for performance and reliability reasons. This is a common enough problem that MongoDB Atlas has a feature that exposes searchable data through some lucene-based backend.[0] Never used it but found the concept to be interesting because it fits the convenient working pattern of "shove it all in the DB and figure it out later." 0: https://www.mongodb.com/atlas/search https://www.mongodb.com/atlas/search

tkiolp4 3y ago
> The big problem with stuffing logs in SQL is that a log search can bring down your app. Logs can be stored in a different db than the one the app is using.
kunley 3y ago
As other commenters noticed already, it is a matter of an isolation more than choosing the technology. And even if focusing on the former, the technology should be chosen wisely. Typical SQL engine is fine with the described traffic. To have more control over the usage of resources, it's good to have such logs and any other analytics in a separate database than your app's transactions. But also I saw quite big deployments where everything was in the same database, and with the right indexes, transaction isolation and some hygiene of writing the queries it was just fine - plus the code was simplified a lot. Elasticsearch is the solution that has a lot of marketing and for sure has a nice and popular query language. Still, it is a memory hog order(s) of magniture more than your average favourite RDBMS. Eating more memory hurts performance and is expensive (memory is a single most expensive factor of a price on any cloud provider). So, it is good to ask a question, "do I really need it"? Splunk is cool and fast. It's not free though, and last time I checked it has a bit complicated pricing. I don't have much clear opinion on MongoDB as it was changing its performance characteristics way too often over the last two decades. Aerospike is one of my favourite NoSQL engines. Offers great speeds and scalability. Its usage for analytics would be unorthodox, but valid. I recommend to give it a shot.

tkiolp4 3y ago

> The big problem with stuffing logs in SQL is that a log search can bring down your app. Logs can be stored in a different db than the one the app is using.

kunley 3y ago

As other commenters noticed already, it is a matter of an isolation more than choosing the technology. And even if focusing on the former, the technology should be chosen wisely. Typical SQL engine is fine with the described traffic. To have more control over the usage of resources, it's good to have such logs and any other analytics in a separate database than your app's transactions. But also I saw quite big deployments where everything was in the same database, and with the right indexes, transaction isolation and some hygiene of writing the queries it was just fine - plus the code was simplified a lot. Elasticsearch is the solution that has a lot of marketing and for sure has a nice and popular query language. Still, it is a memory hog order(s) of magniture more than your average favourite RDBMS. Eating more memory hurts performance and is expensive (memory is a single most expensive factor of a price on any cloud provider). So, it is good to ask a question, "do I really need it"? Splunk is cool and fast. It's not free though, and last time I checked it has a bit complicated pricing. I don't have much clear opinion on MongoDB as it was changing its performance characteristics way too often over the last two decades. Aerospike is one of my favourite NoSQL engines. Offers great speeds and scalability. Its usage for analytics would be unorthodox, but valid. I recommend to give it a shot.