3 ms·

A few of my colleagues and I had the silly (?) idea that you don't really need logs anymore. Instead of log messages you just attach span events [0]. You then j

by renke1 3y ago

A few of my colleagues and I had the silly (?) idea that you don't really need logs anymore. Instead of log messages you just attach span events [0]. You then just log the span title and a link to that span in Jaeger; something like [1]. I've only really tried that in my private project, but it felt pretty good. The UI of Jaeger could be a bit better to support that usage, though. Edit: Actually, those colleagues are doing a talk about that topic. So, if you are in Germany and Hannover area, have a look at [2] and search for "Nie wieder Log-Files!". [0]: https://opentelemetry.io/docs/instrumentation/ruby/manual/#add-span-events https://opentelemetry.io/docs/instrumentation/ruby/manual/#a... [1]: tracing.ts:38 Usecase: Handle Auth tracing.ts:47 http://localhost:16686/trace/ec7ffb1e23ddbb8dd770a3f08028666b tracing.ts:38 Adapter: Find Personal Board tracing.ts:47 http://localhost:16686/trace/e22d342316ab0d7d23230864008e27bc tracing.ts:38 Adapter: Find Starred Board List tracing.ts:47 http://localhost:16686/trace/129f89cee26d54cfdc38abea368d9b4e tracing.ts:38 Adapter: Find Personal Board List tracing.ts:47 http://localhost:16686/trace/97948127d77501ff0c65a5db21b21b5a [2]: https://javaforumnord.de/2023/programm/ https://javaforumnord.de/2023/programm/

phillipcarter 3y ago
Depending on how greenfield a project is, you don't even need span events unless you absolutely require a timestamp for a specific operation with no duration. Just using spans for every meaningful operation is like having more powerful structured logs. This approach isn't possible for a lot of systems that have existing logs they need to bring along, but if you're greenfield enough, I'd recommend it.
- glenjamin 3y ago
  Using only spans is surprisingly effective! I did a talk about this at QCon London last year https://www.infoq.com/presentations/event-tracing-monitoring/ https://www.infoq.com/presentations/event-tracing-monitoring...
- paulddraper 3y ago
  Agreed. Same is true for metrics derived from spans. (Though for metrics, you don't need to sample, and for spans you might. So keep in mind.)
klysm 3y ago
I've thought about doing this as well, but I like being able to use dumb tools to get an idea of what's going on. There's a lot that has to be working correctly to use traces. Or maybe I'm just scared of the tooling because I don't have enough experience with it yet idk
- pquerna 3y ago
  You can also just, Log the spans as they are being created to stderr/stdout -- I've done this on a previous project with this approach of "spans first". It made it debuggable via output if needed, but the primary consumption became span oriented.
  - klysm 3y ago
    Good idea yeah, but do the same notions of log level apply?
kiitos 3y ago
You don't even really need to ship traces anywhere. You can just keep them in-process, and build an API on top of that in-memory trace data.
morelisp 3y ago
This was the idea behind Stripe's Veneur project - spans, logs, and metrics all in the same format, "automatically" rolling up cardinality as needed - which I thought was cool but also that it would be very hard to get non-SRE developers on board with when I saw a talk about it a few years ago. https://github.com/stripe/veneur https://github.com/stripe/veneur
nijave 3y ago
We tried this with Grafana Cloud/Tempo but hit trace size limits. Logs are still good for "events"
mman0114 3y ago
I’ve been talking about the same thing at my place. I think it makes a ton of sense and then can kill logs almost completely

by renke1 3y ago

phillipcarter 3y ago

Depending on how greenfield a project is, you don't even need span events unless you absolutely require a timestamp for a specific operation with no duration. Just using spans for every meaningful operation is like having more powerful structured logs. This approach isn't possible for a lot of systems that have existing logs they need to bring along, but if you're greenfield enough, I'd recommend it.

glenjamin 3y ago
Using only spans is surprisingly effective! I did a talk about this at QCon London last year https://www.infoq.com/presentations/event-tracing-monitoring/ https://www.infoq.com/presentations/event-tracing-monitoring...
paulddraper 3y ago
Agreed. Same is true for metrics derived from spans. (Though for metrics, you don't need to sample, and for spans you might. So keep in mind.)

klysm 3y ago

I've thought about doing this as well, but I like being able to use dumb tools to get an idea of what's going on. There's a lot that has to be working correctly to use traces. Or maybe I'm just scared of the tooling because I don't have enough experience with it yet idk

pquerna 3y ago
You can also just, Log the spans as they are being created to stderr/stdout -- I've done this on a previous project with this approach of "spans first". It made it debuggable via output if needed, but the primary consumption became span oriented.
- klysm 3y ago
  Good idea yeah, but do the same notions of log level apply?

kiitos 3y ago

You don't even really need to ship traces anywhere. You can just keep them in-process, and build an API on top of that in-memory trace data.

morelisp 3y ago

This was the idea behind Stripe's Veneur project - spans, logs, and metrics all in the same format, "automatically" rolling up cardinality as needed - which I thought was cool but also that it would be very hard to get non-SRE developers on board with when I saw a talk about it a few years ago. https://github.com/stripe/veneur https://github.com/stripe/veneur

nijave 3y ago

We tried this with Grafana Cloud/Tempo but hit trace size limits. Logs are still good for "events"

mman0114 3y ago

I’ve been talking about the same thing at my place. I think it makes a ton of sense and then can kill logs almost completely