6 ms·

Metrics are only good when you can disregard some amount of errors without investigation. But they're a financial organization, they have a certain amount of li

• pavlov 2y ago
  > "But they're a financial organization, they have a certain amount of liability." In the loosest possible sense. Binance is an organization that pretended it doesn't have any physical location in any jurisdiction. Its founder is currently in jail in the United States.
• fells 2y ago
  It's always struck me that these are two wildly different concerns though. Use metrics & SLOs to help diagnose the health of your systems. Derive those directly from logs/traces, keep a sample of the raw data, and now you can point any alert to the sampled data to help go about understanding a client-facing issue. But, for auditing of a particular transaction, you don't need full indexing of the events? You need a transactional journal for every account/user, likely with a well-defined schema to describe successful changes and failed attempts. Perhaps these come from the same stream of data as the observability tooling, but I can only imagine it must be a much smaller subset of the 100PB that you can avoid doing full inverse indexes on this, because your search pattern is simply answering "what happened to this transaction?"
  • alexchantavy 2y ago
    > You need a transactional journal for every account/user, likely with a well-defined schema to describe successful changes and failed attempts. Sounds like a row in a database to me. Dumb question, but is that how structured log systems are implemented?
  • renewiltord 2y ago
    The reality is that when their service delays something they owe us tens to hundreds of thousands of dollars. This is the tool they’re using but if they can’t even get a precise notion of when a specific request arrived at their gateway they’re in trouble.
• KaiserPro 2y ago
  You can still have logs. What I'm suggesting is that vast amounts of unstructured logs, are worse than useless. Metics tell you where and when something when wrong. Logs tell you why. However, a logging framework, which is generally lossy, and has the lowest level of priority in terms of deliverability is not an audit mechanism. especially as nowhere are ACLs or verifiability is mentioned. How do they prove that those logs originates from that machine? If you're going to have an audit mechanism, some generic logging framework is almost certainly a bad fit.
  • nostrebored 2y ago
    Why would you assume they're unstructured? Even at very immature organizations, log data within a service is usually structured. Even in my personal projects if I'm doing anything parallel structured logging is the first helper function I write. I don't think I'm unrepresentative here.
    • KaiserPro 2y ago
      Because most logs are unstructured. > Even at very immature organizations, log data within a service is usually structured. unless the framework provides it by default, I've never seen this actually happen in real life. Sure I've seen a lot of custom telegraf configs, status end points and the like, but never actual working structured logging. When I have seen structure logs, each team did it differently, The "ontology" was different (protip: if you're ever discussing ontology in logging then you might as well scream and run away.)
      • quantified 2y ago
        I suspect you and the parent are using different meanings of the word "structured". They're not totally random or they wouldn't be usable. It's a question of what the structuring principle is.
        • KaiserPro 2y ago
          How many unique message formats do you think exist in your org? actually, how many of your messages include the time and date, and how many different ways of displaying timestamp exist in those messages? That is why I say logs are unstructured, because all but a very few places actually have the discipline to enforce a single log structure.