5 ms·

I've just started down a path for us and been pretty happy with Loggly so far. Just by adding a UUID to the requests as they're initially received I'm able to

I've just started down a path for us and been pretty happy with Loggly so far. Just by adding a UUID to the requests as they're initially received I'm able to follow the chain of events across 5 different services, a message queue, and reconcile with any code exception reports from Bugsnag. We don't have a lot of unique clients, but we do have a lot of requests. Loggly can be set to parse logs (or have the data sent directly to it) and then tag messages with keys/values that can let you watch your request through every server.

ktamura 10y ago
Now that I understand the parent comment, I don't think this quite solves the problem he's asking unless all of your services are perfectly synchronized. No?
- manyxcxi 10y ago
  For us, messaging between services go through queues and a client request might start in a particular service and wind it's way through a few others to finish the work. You wouldn't have the same client requesting things from two different services simultaneously, though in fairly short order they come through. Our use case might be just simple enough that this wouldn't work for a lot of people. Also, if I'm looking at access logs I'm not going to have that ID unless I start putting it in request headers. Right now the application logs and other monitoring tools pick it up though. Edit: Didn't see mention of the clock on my first post, but it still stands for us that we're not in the millisecond timings between requests going in and out of services so we haven't had a problem with the timestamps from the systems they're on. Requests generally take a few seconds to process (handle a file upload of decent size, ETL some data, pump out some output document, etc) so each step through a service buys enough time before moving on to the next thing that we're not getting out of order log statements across each service.
  - vardump 10y ago
    > Edit: Didn't see mention of the clock on my first post, but it still stands for us that we're not in the millisecond timings Vector clocks don't necessarily have much to do with clocks. It's an algorithm for generating order in a distributed system. https://en.wikipedia.org/wiki/Vector_clock https://en.wikipedia.org/wiki/Vector_clock
rjbwork 10y ago
Yeah, Loggly is very solid. I've set up logging infrastructure for 2 companies using it now. Their structured logging parsing, searching, and alerting is fantastic.
ben_jones 10y ago
Any tips for a beginner on optimizing usage of the dashboard? Not using it for much right now, and I feel like the interface is going to take awhile to optimize/learn.
organsnyder 10y ago
How are you passing that UUID along the request chain? For instance, when one microservice calls another, does it have to be manually passed in?
- jacques_chester 10y ago
  Some platforms will do this for you -- Cloud Foundry will show `x-cf-requestid` for anything that passed through the gorouter and back.
- eddieroger 10y ago
  My web stack checks for a specific header at each potential level of entry, and in the absence of a value, generates one. Then each layer's logger will log it back out. It's pretty straightforward for web requests - just use the plumbing that's already there, headers.

ktamura 10y ago

Now that I understand the parent comment, I don't think this quite solves the problem he's asking unless all of your services are perfectly synchronized. No?

manyxcxi 10y ago
For us, messaging between services go through queues and a client request might start in a particular service and wind it's way through a few others to finish the work. You wouldn't have the same client requesting things from two different services simultaneously, though in fairly short order they come through. Our use case might be just simple enough that this wouldn't work for a lot of people. Also, if I'm looking at access logs I'm not going to have that ID unless I start putting it in request headers. Right now the application logs and other monitoring tools pick it up though. Edit: Didn't see mention of the clock on my first post, but it still stands for us that we're not in the millisecond timings between requests going in and out of services so we haven't had a problem with the timestamps from the systems they're on. Requests generally take a few seconds to process (handle a file upload of decent size, ETL some data, pump out some output document, etc) so each step through a service buys enough time before moving on to the next thing that we're not getting out of order log statements across each service.
- vardump 10y ago
  > Edit: Didn't see mention of the clock on my first post, but it still stands for us that we're not in the millisecond timings Vector clocks don't necessarily have much to do with clocks. It's an algorithm for generating order in a distributed system. https://en.wikipedia.org/wiki/Vector_clock https://en.wikipedia.org/wiki/Vector_clock

rjbwork 10y ago

Yeah, Loggly is very solid. I've set up logging infrastructure for 2 companies using it now. Their structured logging parsing, searching, and alerting is fantastic.

ben_jones 10y ago

Any tips for a beginner on optimizing usage of the dashboard? Not using it for much right now, and I feel like the interface is going to take awhile to optimize/learn.

organsnyder 10y ago

How are you passing that UUID along the request chain? For instance, when one microservice calls another, does it have to be manually passed in?

jacques_chester 10y ago
Some platforms will do this for you -- Cloud Foundry will show `x-cf-requestid` for anything that passed through the gorouter and back.
eddieroger 10y ago
My web stack checks for a specific header at each potential level of entry, and in the absence of a value, generates one. Then each layer's logger will log it back out. It's pretty straightforward for web requests - just use the plumbing that's already there, headers.