10 ms·

Scaling our observability platform by embracing wide events and replacing OTel

200 points by valyala 1y ago

henning 1y ago
Yes, this what the people who will curse you out and judge you for not using wide events omits: it will greatly increase storage costs compared to the normal metrics + traces + sample based logging that is conventional. It has both a benefit and a cost, and the cost part is always omitted.
- valyala 1y ago
  Properly implemented wide events usually reduce storage costs comparing to typical chaotic logging of everything. It is expected that a single external request leads to exactly one wide event with all the information about this request, which may be needed for further debugging and analytics. See https://jeremymorrell.dev/blog/a-practitioners-guide-to-wide-events/ https://jeremymorrell.dev/blog/a-practitioners-guide-to-wide... .
  - edzhelyov 1y ago
    How would you add an outgoing request you make to external system in the wide event? For example, I receive a request, in that request I make a HTTP call to http://example.com http://example.com. In tracing that will be a separate span, but how you manage that in a single wide event?
- behemot 1y ago
  ClickHouse is pretty good at compressing the wide events, so it's not that dramatic compared to the benefits of having high-cardinality telemetry. check this out: https://clickhouse.com/blog/optimize-clickhouse-codecs-compression-schema https://clickhouse.com/blog/optimize-clickhouse-codecs-compr...
Xcelerate 1y ago
Do wide events really have to take up this much space? I mean, observability is to a large degree basically a sampling problem where the goal is to maximize the ability to reconstruct the state of the environment at a given time using a minimal amount of storage. You can accomplish that by either reducing the number of samples taken or by improving your compression capability. For the latter, I have a very hard time believing we’ve squeezed most of the juice out of compression already. Surely there’s an absolutely massive amount of low-rank structure in all that redundant data. Yeah, I know these companies already use inverted indices and various sorts of trees, but I would have thought there are more research-y approaches (e.g. low rank tensor decomposition) that if we could figure out how to perform them efficiently would blow the existing methods out of the water. But IDK, I’m not in that industry so maybe I’m overlooking something.
- behemot 1y ago
  > Do wide events really have to take up this much space? 100PB is the total volume of the raw, uncompressed data for the full retention period (180 days). compression is what makes it cost-efficient. on this dataset, we see ~15x compression, so we only store around 6.5PB at rest.
AntonCTO 1y ago
There isn't much information about correlation. What are the state-of-the-art tools and techniques for observability in stateful use cases? Let's take the example of an SFU-based video conferencing app, where user devices go through multiple API calls to join a session. Now imagine a user reports that they cannot see video from another participant. How can such problems be effectively traced? Of course, I can manually filter logs and traces by the first user, then by the second user, and look at the signaling exchange and frontend/backend errors. But are there better approaches?
ofrzeta 1y ago
Whenever I read things like this I think: You are doing it wrong. I guess it is an amazing engineering feat for Clickhouse but I think we (as in IT or all people) should really reduce the amount of data we create. It is wasteful.
- valyala 1y ago
  This is called "progress". Humans always generate the amounts of data which can be stored and processed by the tools they have. The more data the tool can process under the given budget limit, the more data will be generated and stored.
- XorNot 1y ago
  The problem with this is generally that you have logs from years ago, but no way to get a live stream of logs which are happening now. (one of my immense frustrations with kubernetes - none of the commands for viewing logs seem to accept logical aggregates like "show me everything from this deployment").
  - shikhar 1y ago
    We have a customer using s2.dev for this capability – granular tail-able streams with granular access control (e.g. let an end user of a job tail it with a read-only access token). We'll be shipping an OTel endpoint soon to make it even easier.
  - knutzui 1y ago
    Maybe not via kubectl directly, but it is rather trivial to build this, by simply combining all log streams from pods of a deployment (or whatever else). k9s (k9scli.io) supports this directly.
  - madduci 1y ago
    And what is the sense of keeping years of logs? I could probably understand very sensitive industries, but In general, I see a pure waste of resources. At most you need 60-90 days of logs.
tjungblut 1y ago
tldr, they now do a zero (?) copy of raw bytes instead of marshaling and unmarshaling json.
the_real_cher 1y ago
What is the trick that this and dynamo use? Are they just basically large hash tables?
- valyala 1y ago
  There are two tricks used by ClickHouse and similar databases: - Smart placement of the data on disk, which allows skipping the majority of data and reading only the needed chunks (and these chunks are stored in a compressed form in order to reduce disk read IO usage even more). This includes column-oriented storage and LSM-like trees. - Brute-force optimizations all over the place, which allow processing the found data at the maximum speed by employing all the compute resources (CPU, RAM, disk IO, network bandwidth) in the most efficient way. For example, ClickHouse can process more than a billion of rows per second per every CPU core, and the scan speed scales linearly with the number of available CPU cores.
atemerev 1y ago
When I get back from Clickhouse to Postgres, I am always shocked. Like, what it is doing for some minutes importing this 20G dump? Shouldn't it take seconds?
- joshstrange 1y ago
  Every time I use Clickhouse I want blow my brains out, especially knowing that Postgres exists. I’m not saying Clickhouse doesn’t have its place or that Postgres can do everything that Clickhouse can. What I am saying is that I really dislike working in Clickhouse with all of the weird foot guns. Unless you are using it in a very specific, and in my opinion, limited way, it feels worse than Postgres in every way.
  - mdaniel 1y ago
    Anything in my life that uses Zookeeper or its dumbass etcd friend means I'm going to have a real bad time. I am thankful they're at least shipping their own ZK-ish but it seems to have fallen into the same trap as etcd, where membership has to be managed like the precious little pets that they are https://clickhouse.com/docs/guides/sre/keeper/clickhouse-keeper#recovering-after-losing-quorum https://clickhouse.com/docs/guides/sre/keeper/clickhouse-kee...
    - jiggawatts 1y ago
      Zookeeper in the only clustering product I’ve ever used that actively refused to start a cluster after an all-nodes stop/start. It blows my mind that a high availability system would purposefully prevent availability as a “feature”.
      - sciurus 1y ago
        Although this is oversimplifying things [0], in the face of partitions zookeeper emphasizes consistency over availability. [0] https://martin.kleppmann.com/2015/05/11/please-stop-calling-databases-cp-or-ap.html https://martin.kleppmann.com/2015/05/11/please-stop-calling-...
mrbluecoat 1y ago
Noteworthy point: > If a service is crash-looping or down, SysEx is unable to scrape data because the necessary system tables are unavailable. OpenTelemetry, by contrast, operates in a passive fashion. It captures logs emitted to stdout and stderr, even when the service is in a failed state. This allows us to collect logs during incidents and perform root cause analysis even if the service never became fully healthy.
- fuzzy2 1y ago
  Everything OTel I ever did was fully active. So I wouldn't say this is very noteworthy. Instead it is wrong/incomplete information.
  - behemot 1y ago
    we use k8s + otel filelog receiver. in this case you don't have to connect to the clickhouse instance to collect what it's writing to stdout/stderr, just tail /var/log/pods/*/*/*.log.
jurgenkesker 1y ago
So yeah, this is only really relevant for collecting logs from clickhouse. Not for logs from anything else. Good for them, and I really love Clickhouse, but not really relevant.
- dangoodmanUT 1y ago
  You must be fun at parties
iw7tdb2kqo9 1y ago
I haven't worked in ClickHouse level scale. Can you search log data in this volume? ElasticSearch has query capabilities for small scale log data I think. Why would I use ClickHouse instead of storing log data as json file for historical log data?
- valyala 1y ago
  > Why would I use ClickHouse instead of storing log data as json file for historical log data? There are multiple reasons: 1. Databases optimized for logs (such as ClickHouse or VictoriaLogs) store logs in a compressed form, where values per every log field are grouped and compressed individually (aka column-oriented storage). This results in smaller storage space comparing to plain files with JSON logs, even if they are compressed. 2. Databases optimized for logs perform typical queries at much faster speed comparing to grep over JSON files. Performance gains may be 1000x and more because these databases skip reading unneeded data. See https://chronicles.mad-scientist.club/tales/grepping-logs-remains-terrible/ https://chronicles.mad-scientist.club/tales/grepping-logs-re... 3. How are you going to grep 100 petabytes of JSON files? Databases optimized for logs allow querying such amounts of logs because they can scale horizontally by adding more storage nodes and storage space.
- sethammons 1y ago
  Scale and costs. We are faced with logging scale at my work. A naive "push json into splunk" will cost us over $6M/year, but I can only get maybe 5-10% of that approved. In the article, they talk about needing 8k cpu to process their json logs, but only 90 cpu afterward.
- munchbunny 1y ago
  > Can you search log data in this volume? (Context: I work at this scale) Yes. However, as you can imagine, the processing costs can be potentially enormous. If your indexing/ordering/clustering strategy isn't set up well, a single query can easily end up costing you on the order of $1-$10 to do something as simple as "look for records containing this string". My experiences line up with theirs: at the scale where you are moving petabytes of data, the best optimizations are, unsurprisingly, "touch as little data as few times as possible" and "move as little data as possible". Every time you have to serialize/de-serialize, and every time you have to perform disk/network I/O, you introduce a lot of performance cost and therefore overall cost to your wallet. Naturally, this can put OTel directly at odds with efficiency because the OTel collector is an extra I/O and serialization hop. But then again, if you operate at the petabyte scale, the amount of money you save by throwing away a single hop can more than pay for an engineer whose only job is to write serializer/deserializer logic.
revskill 1y ago
THis industry is mostly filled with half-baked or in-progress standards which leads to segmentation of the ecosystems. From graphql, to openapi, to mcp,... to everything, nothing is perfect and it's fine. The problem is, people who created spec is just following trial and error approach, which is insane.
Thaxll 1y ago
I mean if you don´t get the logs when the serivce is down the entire solution is useless.
b0a04gl 1y ago
tbh that's not the flex. storing 100PB of logs just means we haven't figured out what's actually worth logging. metrics + structured events can usually tell 90% of the story. the rest? trace level chaos no one reads unless prod's on fire. what'd could've done better be: auto pruning logs that no alert ever looked at. or logs that never hit a search query in 3 months. call it attention weighted retention. until then this is just high end digital landfill with compression
- Spivak 1y ago
  > trace level chaos no one reads unless prod's on fire God why do we keep these fire extinguishers around, they sit unused 99.999% of the time.
  - jiggawatts 1y ago
    “Just go back in time and turn on the specific log you will need!”
- hinkley 1y ago
  That logging isn’t even free on the sending side, especially in languages where they are eager to get the logs to disk in case the final message reveals why the program crashed. And there’s a lot of scanning blindness out there. Too much extraneous data can hide correlations between other logs entries. And there’s half life in value of logs written for bugs that are already closed, and it’s fairly short. I prefer stats because of the way they get aggregated. Though for GIL languages some models like OTEL have higher overhead than they should.
  - nijave 1y ago
    In fairness, I think a lot of GIL languages already have high overload and I've never been under the impression OTEL was optimized for performance and efficiency.
    - hinkley 1y ago
jappgar 1y ago
Observability maximalism is a cult. A very rich one.
- hinkley 1y ago
  Funny how they give you a problem and solve it for you for a small monthly fee.
- k__ 1y ago
  Well, if you wanna investigate unknown unknowns, there isn't much alternative.
the_arun 1y ago
I didn’t see how long logs are kept - retention time. After x months you may need summary/aggregated data but not sure about raw data.
- behemot 1y ago
  we keep it for 180 days.

200 points by valyala 1y ago

Yes, this what the people who will curse you out and judge you for not using wide events omits: it will greatly increase storage costs compared to the normal metrics + traces + sample based logging that is conventional. It has both a benefit and a cost, and the cost part is always omitted.

valyala 1y ago
Properly implemented wide events usually reduce storage costs comparing to typical chaotic logging of everything. It is expected that a single external request leads to exactly one wide event with all the information about this request, which may be needed for further debugging and analytics. See https://jeremymorrell.dev/blog/a-practitioners-guide-to-wide-events/ https://jeremymorrell.dev/blog/a-practitioners-guide-to-wide... .
- edzhelyov 1y ago
  How would you add an outgoing request you make to external system in the wide event? For example, I receive a request, in that request I make a HTTP call to http://example.com http://example.com. In tracing that will be a separate span, but how you manage that in a single wide event?
behemot 1y ago
ClickHouse is pretty good at compressing the wide events, so it's not that dramatic compared to the benefits of having high-cardinality telemetry. check this out: https://clickhouse.com/blog/optimize-clickhouse-codecs-compression-schema https://clickhouse.com/blog/optimize-clickhouse-codecs-compr...

Xcelerate 1y ago

Do wide events really have to take up this much space? I mean, observability is to a large degree basically a sampling problem where the goal is to maximize the ability to reconstruct the state of the environment at a given time using a minimal amount of storage. You can accomplish that by either reducing the number of samples taken or by improving your compression capability. For the latter, I have a very hard time believing we’ve squeezed most of the juice out of compression already. Surely there’s an absolutely massive amount of low-rank structure in all that redundant data. Yeah, I know these companies already use inverted indices and various sorts of trees, but I would have thought there are more research-y approaches (e.g. low rank tensor decomposition) that if we could figure out how to perform them efficiently would blow the existing methods out of the water. But IDK, I’m not in that industry so maybe I’m overlooking something.

behemot 1y ago
> Do wide events really have to take up this much space? 100PB is the total volume of the raw, uncompressed data for the full retention period (180 days). compression is what makes it cost-efficient. on this dataset, we see ~15x compression, so we only store around 6.5PB at rest.

AntonCTO 1y ago

There isn't much information about correlation. What are the state-of-the-art tools and techniques for observability in stateful use cases? Let's take the example of an SFU-based video conferencing app, where user devices go through multiple API calls to join a session. Now imagine a user reports that they cannot see video from another participant. How can such problems be effectively traced? Of course, I can manually filter logs and traces by the first user, then by the second user, and look at the signaling exchange and frontend/backend errors. But are there better approaches?

ofrzeta 1y ago

Whenever I read things like this I think: You are doing it wrong. I guess it is an amazing engineering feat for Clickhouse but I think we (as in IT or all people) should really reduce the amount of data we create. It is wasteful.

valyala 1y ago
This is called "progress". Humans always generate the amounts of data which can be stored and processed by the tools they have. The more data the tool can process under the given budget limit, the more data will be generated and stored.
XorNot 1y ago
The problem with this is generally that you have logs from years ago, but no way to get a live stream of logs which are happening now. (one of my immense frustrations with kubernetes - none of the commands for viewing logs seem to accept logical aggregates like "show me everything from this deployment").
- shikhar 1y ago
  We have a customer using s2.dev for this capability – granular tail-able streams with granular access control (e.g. let an end user of a job tail it with a read-only access token). We'll be shipping an OTel endpoint soon to make it even easier.
- knutzui 1y ago
  Maybe not via kubectl directly, but it is rather trivial to build this, by simply combining all log streams from pods of a deployment (or whatever else). k9s (k9scli.io) supports this directly.
- madduci 1y ago
  And what is the sense of keeping years of logs? I could probably understand very sensitive industries, but In general, I see a pure waste of resources. At most you need 60-90 days of logs.

tjungblut 1y ago

tldr, they now do a zero (?) copy of raw bytes instead of marshaling and unmarshaling json.

the_real_cher 1y ago

What is the trick that this and dynamo use? Are they just basically large hash tables?

valyala 1y ago
There are two tricks used by ClickHouse and similar databases: - Smart placement of the data on disk, which allows skipping the majority of data and reading only the needed chunks (and these chunks are stored in a compressed form in order to reduce disk read IO usage even more). This includes column-oriented storage and LSM-like trees. - Brute-force optimizations all over the place, which allow processing the found data at the maximum speed by employing all the compute resources (CPU, RAM, disk IO, network bandwidth) in the most efficient way. For example, ClickHouse can process more than a billion of rows per second per every CPU core, and the scan speed scales linearly with the number of available CPU cores.

atemerev 1y ago

When I get back from Clickhouse to Postgres, I am always shocked. Like, what it is doing for some minutes importing this 20G dump? Shouldn't it take seconds?

joshstrange 1y ago
Every time I use Clickhouse I want blow my brains out, especially knowing that Postgres exists. I’m not saying Clickhouse doesn’t have its place or that Postgres can do everything that Clickhouse can. What I am saying is that I really dislike working in Clickhouse with all of the weird foot guns. Unless you are using it in a very specific, and in my opinion, limited way, it feels worse than Postgres in every way.
- mdaniel 1y ago
  Anything in my life that uses Zookeeper or its dumbass etcd friend means I'm going to have a real bad time. I am thankful they're at least shipping their own ZK-ish but it seems to have fallen into the same trap as etcd, where membership has to be managed like the precious little pets that they are https://clickhouse.com/docs/guides/sre/keeper/clickhouse-keeper#recovering-after-losing-quorum https://clickhouse.com/docs/guides/sre/keeper/clickhouse-kee...
  - jiggawatts 1y ago
    Zookeeper in the only clustering product I’ve ever used that actively refused to start a cluster after an all-nodes stop/start. It blows my mind that a high availability system would purposefully prevent availability as a “feature”.
    - sciurus 1y ago
      Although this is oversimplifying things [0], in the face of partitions zookeeper emphasizes consistency over availability. [0] https://martin.kleppmann.com/2015/05/11/please-stop-calling-databases-cp-or-ap.html https://martin.kleppmann.com/2015/05/11/please-stop-calling-...

mrbluecoat 1y ago

Noteworthy point: > If a service is crash-looping or down, SysEx is unable to scrape data because the necessary system tables are unavailable. OpenTelemetry, by contrast, operates in a passive fashion. It captures logs emitted to stdout and stderr, even when the service is in a failed state. This allows us to collect logs during incidents and perform root cause analysis even if the service never became fully healthy.

fuzzy2 1y ago
Everything OTel I ever did was fully active. So I wouldn't say this is very noteworthy. Instead it is wrong/incomplete information.
- behemot 1y ago
  we use k8s + otel filelog receiver. in this case you don't have to connect to the clickhouse instance to collect what it's writing to stdout/stderr, just tail /var/log/pods/*/*/*.log.

jurgenkesker 1y ago

So yeah, this is only really relevant for collecting logs from clickhouse. Not for logs from anything else. Good for them, and I really love Clickhouse, but not really relevant.

dangoodmanUT 1y ago
You must be fun at parties

iw7tdb2kqo9 1y ago

I haven't worked in ClickHouse level scale. Can you search log data in this volume? ElasticSearch has query capabilities for small scale log data I think. Why would I use ClickHouse instead of storing log data as json file for historical log data?

valyala 1y ago
> Why would I use ClickHouse instead of storing log data as json file for historical log data? There are multiple reasons: 1. Databases optimized for logs (such as ClickHouse or VictoriaLogs) store logs in a compressed form, where values per every log field are grouped and compressed individually (aka column-oriented storage). This results in smaller storage space comparing to plain files with JSON logs, even if they are compressed. 2. Databases optimized for logs perform typical queries at much faster speed comparing to grep over JSON files. Performance gains may be 1000x and more because these databases skip reading unneeded data. See https://chronicles.mad-scientist.club/tales/grepping-logs-remains-terrible/ https://chronicles.mad-scientist.club/tales/grepping-logs-re... 3. How are you going to grep 100 petabytes of JSON files? Databases optimized for logs allow querying such amounts of logs because they can scale horizontally by adding more storage nodes and storage space.
sethammons 1y ago
Scale and costs. We are faced with logging scale at my work. A naive "push json into splunk" will cost us over $6M/year, but I can only get maybe 5-10% of that approved. In the article, they talk about needing 8k cpu to process their json logs, but only 90 cpu afterward.
munchbunny 1y ago
> Can you search log data in this volume? (Context: I work at this scale) Yes. However, as you can imagine, the processing costs can be potentially enormous. If your indexing/ordering/clustering strategy isn't set up well, a single query can easily end up costing you on the order of $1-$10 to do something as simple as "look for records containing this string". My experiences line up with theirs: at the scale where you are moving petabytes of data, the best optimizations are, unsurprisingly, "touch as little data as few times as possible" and "move as little data as possible". Every time you have to serialize/de-serialize, and every time you have to perform disk/network I/O, you introduce a lot of performance cost and therefore overall cost to your wallet. Naturally, this can put OTel directly at odds with efficiency because the OTel collector is an extra I/O and serialization hop. But then again, if you operate at the petabyte scale, the amount of money you save by throwing away a single hop can more than pay for an engineer whose only job is to write serializer/deserializer logic.

revskill 1y ago

THis industry is mostly filled with half-baked or in-progress standards which leads to segmentation of the ecosystems. From graphql, to openapi, to mcp,... to everything, nothing is perfect and it's fine. The problem is, people who created spec is just following trial and error approach, which is insane.

Thaxll 1y ago

I mean if you don´t get the logs when the serivce is down the entire solution is useless.

b0a04gl 1y ago

tbh that's not the flex. storing 100PB of logs just means we haven't figured out what's actually worth logging. metrics + structured events can usually tell 90% of the story. the rest? trace level chaos no one reads unless prod's on fire. what'd could've done better be: auto pruning logs that no alert ever looked at. or logs that never hit a search query in 3 months. call it attention weighted retention. until then this is just high end digital landfill with compression

Spivak 1y ago
> trace level chaos no one reads unless prod's on fire God why do we keep these fire extinguishers around, they sit unused 99.999% of the time.
- jiggawatts 1y ago
  “Just go back in time and turn on the specific log you will need!”
hinkley 1y ago
That logging isn’t even free on the sending side, especially in languages where they are eager to get the logs to disk in case the final message reveals why the program crashed. And there’s a lot of scanning blindness out there. Too much extraneous data can hide correlations between other logs entries. And there’s half life in value of logs written for bugs that are already closed, and it’s fairly short. I prefer stats because of the way they get aggregated. Though for GIL languages some models like OTEL have higher overhead than they should.
- nijave 1y ago
  In fairness, I think a lot of GIL languages already have high overload and I've never been under the impression OTEL was optimized for performance and efficiency.
  - hinkley 1y ago

jappgar 1y ago

Observability maximalism is a cult. A very rich one.

hinkley 1y ago
Funny how they give you a problem and solve it for you for a small monthly fee.
k__ 1y ago
Well, if you wanna investigate unknown unknowns, there isn't much alternative.

the_arun 1y ago

I didn’t see how long logs are kept - retention time. After x months you may need summary/aggregated data but not sure about raw data.

behemot 1y ago
we keep it for 180 days.