4 ms·

Strong yes, particularly this line: > the cost of monitoring an application can easily exceed the cost of hosting the application even for simple applications

Strong yes, particularly this line: > the cost of monitoring an application can easily exceed the cost of hosting the application even for simple applications (especially if you are looking at saas monitoring for a single-box or serverless thing) I suspect SIEM is the most established use case for ingesting logs to something actionable; wonder if there are concepts there which port generally. Datomic's approach to change capture seemed very good when it came out -- I wish this were a thing I could cheaply turn on and off on SQL columns. As things are, every ORM seems to resist having a table with no primary key so 'log important things to the DB' is hard in practice. In postgres, mvcc makes it expensive to edit a scalar, so it is impractical to store counts. (There is an old uber blogpost I think about why they use mysql for this use case). Specifically for alerting, I wish I could define application logic statistics in my application rather than PromQL / datadog json. Would be much easier to express 'warn me if this ever fails', 'expect 50% failure', 'warn me if too many retries' type stuff. On the 'looking at logs' side, the hosted monitoring tools I've used are often uncomfortably slow, even when my data is small. I understand it's a hard problem but I miss graphite / whisper. (not even getting into mixpanel / posthog type product-level observability)

rjbwork 3y ago
>I wish this were a thing I could cheaply turn on and off on SQL columns. You can turn it on and off for whole tables in systems that have decided to implement SQL:2011 such as MSSQL, Oracle, MariaDB, etc. and then issue time travel queries to see what the state of the database was at any given point in time. Keyword here is "Temporal Table".
- awinter-py 3y ago
  did not know about this -- thank you for the rec
lukeasrodgers 3y ago
What do you mean "expensive to edit a scalar"? If we're thinking of the same Uber article on mysql -> postgres, IIRC it had to do with write amplification related to updates to indexed columns. I don't think postgres has any problems storing counts, but would be interested to know if I'm wrong.
- awinter-py 3y ago
  this article https://www.uber.com/blog/postgres-to-mysql-migration/ https://www.uber.com/blog/postgres-to-mysql-migration/ you're right -- I misunderstood. I thought mysql could do in-place writes to rows, saving mvcc / vacuum work let me amend my point to: a system designed for storing counters can optimize for that case, while sql DBs will do relatively more IO to incremental a counter
mst 3y ago
> every ORM seems to resist having a table with no primary key perl's DBIx::Class doesn't honestly give a shit. I understand there are many perfectly understandable reasons why people dislike perl itself, but it's really annoying to use ORMs in more popular languages and keep finding things perl developers have been able to take for granted for 15+ years that Just. Aren't. There. I'm not asking anybody to like perl who doesn't already, it's definitely an acquired taste, but I think it's understandable for me to wish people developing libraries for other languages would get around to stealing some of DBIx::Class' less commonly available features (I may eventually get around to doing so myself but I strongly suspect that for any given language there are quite a few people out there who could do a better job of it, faster, were they to decide it sounded like sufficient fun to spend time on it, so unless/until -I- decide it sounds like sufficient fun I'm going to keep mentioning it in the hopes somebody else does ;).
roflyear 3y ago
Well, "easily" - I am not totally sure, but it can happen without much trouble. I guess depends on your definition of easily.
j3s 3y ago
> I miss graphite / whisper. as someone who had to manage a system like this, i certainly don't. it doesn't scale well, it requires way too much upfront design thought (since metrics are stored in arbitrary dirs), and it's a massive pain to install and manage. not to mention, it's barely developed anymore so much needed features (like carbon cache restarts not taking a million years bc of the slow sequential flush to disk) will probably never see the light of day. for better or worse, the industry moved on. speaking personally, prometheus is so much better than graphite ever was, in every conceivable way.
- valyala 3y ago
  There are Graphite-compatible monitoring systems, which use much more efficient storage format than Whisper. They resolve the most annoying issues associated with Graphite for large-scale setups - high disk IO and high disk space usage. See [1] and [2]. [1] https://github.com/go-graphite/carbon-clickhouse https://github.com/go-graphite/carbon-clickhouse [2] https://docs.victoriametrics.com/#graphite-api-usage https://docs.victoriametrics.com/#graphite-api-usage

rjbwork 3y ago

>I wish this were a thing I could cheaply turn on and off on SQL columns. You can turn it on and off for whole tables in systems that have decided to implement SQL:2011 such as MSSQL, Oracle, MariaDB, etc. and then issue time travel queries to see what the state of the database was at any given point in time. Keyword here is "Temporal Table".

awinter-py 3y ago
did not know about this -- thank you for the rec

lukeasrodgers 3y ago

What do you mean "expensive to edit a scalar"? If we're thinking of the same Uber article on mysql -> postgres, IIRC it had to do with write amplification related to updates to indexed columns. I don't think postgres has any problems storing counts, but would be interested to know if I'm wrong.

awinter-py 3y ago
this article https://www.uber.com/blog/postgres-to-mysql-migration/ https://www.uber.com/blog/postgres-to-mysql-migration/ you're right -- I misunderstood. I thought mysql could do in-place writes to rows, saving mvcc / vacuum work let me amend my point to: a system designed for storing counters can optimize for that case, while sql DBs will do relatively more IO to incremental a counter

mst 3y ago

> every ORM seems to resist having a table with no primary key perl's DBIx::Class doesn't honestly give a shit. I understand there are many perfectly understandable reasons why people dislike perl itself, but it's really annoying to use ORMs in more popular languages and keep finding things perl developers have been able to take for granted for 15+ years that Just. Aren't. There. I'm not asking anybody to like perl who doesn't already, it's definitely an acquired taste, but I think it's understandable for me to wish people developing libraries for other languages would get around to stealing some of DBIx::Class' less commonly available features (I may eventually get around to doing so myself but I strongly suspect that for any given language there are quite a few people out there who could do a better job of it, faster, were they to decide it sounded like sufficient fun to spend time on it, so unless/until -I- decide it sounds like sufficient fun I'm going to keep mentioning it in the hopes somebody else does ;).

roflyear 3y ago

Well, "easily" - I am not totally sure, but it can happen without much trouble. I guess depends on your definition of easily.

j3s 3y ago

> I miss graphite / whisper. as someone who had to manage a system like this, i certainly don't. it doesn't scale well, it requires way too much upfront design thought (since metrics are stored in arbitrary dirs), and it's a massive pain to install and manage. not to mention, it's barely developed anymore so much needed features (like carbon cache restarts not taking a million years bc of the slow sequential flush to disk) will probably never see the light of day. for better or worse, the industry moved on. speaking personally, prometheus is so much better than graphite ever was, in every conceivable way.

valyala 3y ago
There are Graphite-compatible monitoring systems, which use much more efficient storage format than Whisper. They resolve the most annoying issues associated with Graphite for large-scale setups - high disk IO and high disk space usage. See [1] and [2]. [1] https://github.com/go-graphite/carbon-clickhouse https://github.com/go-graphite/carbon-clickhouse [2] https://docs.victoriametrics.com/#graphite-api-usage https://docs.victoriametrics.com/#graphite-api-usage