4 ms·

Since others have answered with specific tech stacks, I'll give a more generalized/abstracted answer. While getting started, here are a few high-level principle

Since others have answered with specific tech stacks, I'll give a more generalized/abstracted answer. While getting started, here are a few high-level principles I found useful to adhere that will make your life easier later: Think of a multi-stage pipeline for getting raw data from your transactional/interaction systems and extracting insights and intelligence out of them. Stage-1: Ingestion – Keep this simple. Don't mess this up. Its a serious headache if you do. 1. Generate a request-id or message-id at the genesis of the request and propagate it throughout your call graph from client to servers (across any number of api call graph hops). 2. At each node in the call graph, emit whatever logs you want to emit, include this id. 3. Use whatever format you find natural and easy to use in each tech stack environment. Key is to make the logging instrumentation very natural and normal to each codebase such that the instrumentation does not get accidentally broken while adding new features. 4. Build a plumbing layer (agnostic of what is being logged) that can locally buffer these log messages, periodically compress and package them with added sequence and integrity verification mechanisms, and reliability transmit them to a central warehouse. Use this across all your server-side nodes. Build a similar one for each of your client side platforms. 5. At the central warehouse, immediately persist these log packages durably and then only respond to client indicating it is safe to purge those packages on their local nodes. Stage-2: Use-case driven ETLs. 6. Come up with use-cases to consume this data. Define data tables (facts and dimensions) needed to support these consumption use case. 7. Build a high-performance stream processing system that can process the raw log packages for doing ETL (extract, transform and load) on the raw data in different formats to the defined consumable data tables. Stage-3: Actual Use-case data applications. Run your analytics and machine learning systems on top of these stable consumable data formats. Keep the stages separate and decoupled in code and systems. Don't do end-to-end optimizations and break the boundaries. Recognize that the actors/stakeholders involved in each stage are different. The job of data team is to be the guardian of these stages and run the systems and org processes to support it.

mustardo 7y ago
Such a HN answer, The dudes... >the backend developer at a mobile app startup, How about syslog, ELK stack or something and focus on building the app Some good points in there like correlation IDs etc all the same
alex-bender 7y ago
> and propagate it throughout your call graph from Have you tried something like opentracing.io ?
stareatgoats 7y ago
Having battled the curse of spurious errors in both large and small systems without centralized logging I'd say looks like great tips to me. Even for smaller setups, who might need something like this more than anyone. (A simplified analytics setup in that case and the machine learning excluded I guess).
meowface 7y ago
This is true if you plan to develop a SIEM or ELK completely from scratch. Interesting as general background info, but I can't see this information being practically useful to anyone who just wants to log stuff. It'd be like building a washing machine and drying machine from scratch because you want to wash your clothes. You seem to be describing low level principles, not high level ones. A high level principle would be "forward your logs to a centralized logging service and let the logging library and the service do 100% of the work for you", which I think is what nearly everyone should do (and which most are already doing).
- zawerf 7y ago
  His first bullet point is probably the most practical thing I know about logging. Log lines are useless without context especially when the lines are interleaved. But this is easily fixed just by prefixing them with a context id that you filter on later. This requires no frameworks and even works with unstructured text logging. Super simple, super useful, not everyone does it.
  - meowface 7y ago
    True, but this is semi-automatically handled by any structured logging library, though. No need to reinvent the wheel or force yourself to remember to prefix or postfix every log message with one or more "%s"'s (or equivalent) and the ID(s) to interpolate. I think that's one of the main reasons to use a structured logging library in the first place, and maybe the main purpose of structured logging. A simple example for a Python Flask app: http://www.structlog.org/en/stable/examples.html http://www.structlog.org/en/stable/examples.html log = logger.new(request_id=str(uuid.uuid4())) log.info("user logged in", user="test-user") # gives you: # event='user logged in' request_id='ffcdc44f-b952-4b5f-95e6-0f1f3a9ee5fd' user='test-user' Sure, if you absolutely must use unstructured logging, you need to remember to do the format string prefixing or postfixing for every single message. But why put yourself in that position if you don't need to? Other than maybe when maintaining large legacy apps that aren't worth the effort to add structured logging to.
    - 7y ago
hooch 7y ago
Thanks. This provides a very useful architectural map to keep in mind whilst investigating the various implementations.

mustardo 7y ago

Such a HN answer, The dudes... >the backend developer at a mobile app startup, How about syslog, ELK stack or something and focus on building the app Some good points in there like correlation IDs etc all the same

alex-bender 7y ago

> and propagate it throughout your call graph from Have you tried something like opentracing.io ?

stareatgoats 7y ago

Having battled the curse of spurious errors in both large and small systems without centralized logging I'd say looks like great tips to me. Even for smaller setups, who might need something like this more than anyone. (A simplified analytics setup in that case and the machine learning excluded I guess).

meowface 7y ago

This is true if you plan to develop a SIEM or ELK completely from scratch. Interesting as general background info, but I can't see this information being practically useful to anyone who just wants to log stuff. It'd be like building a washing machine and drying machine from scratch because you want to wash your clothes. You seem to be describing low level principles, not high level ones. A high level principle would be "forward your logs to a centralized logging service and let the logging library and the service do 100% of the work for you", which I think is what nearly everyone should do (and which most are already doing).

zawerf 7y ago
His first bullet point is probably the most practical thing I know about logging. Log lines are useless without context especially when the lines are interleaved. But this is easily fixed just by prefixing them with a context id that you filter on later. This requires no frameworks and even works with unstructured text logging. Super simple, super useful, not everyone does it.
- meowface 7y ago
  True, but this is semi-automatically handled by any structured logging library, though. No need to reinvent the wheel or force yourself to remember to prefix or postfix every log message with one or more "%s"'s (or equivalent) and the ID(s) to interpolate. I think that's one of the main reasons to use a structured logging library in the first place, and maybe the main purpose of structured logging. A simple example for a Python Flask app: http://www.structlog.org/en/stable/examples.html http://www.structlog.org/en/stable/examples.html log = logger.new(request_id=str(uuid.uuid4())) log.info("user logged in", user="test-user") # gives you: # event='user logged in' request_id='ffcdc44f-b952-4b5f-95e6-0f1f3a9ee5fd' user='test-user' Sure, if you absolutely must use unstructured logging, you need to remember to do the format string prefixing or postfixing for every single message. But why put yourself in that position if you don't need to? Other than maybe when maintaining large legacy apps that aren't worth the effort to add structured logging to.
  - 7y ago

hooch 7y ago

Thanks. This provides a very useful architectural map to keep in mind whilst investigating the various implementations.