5 ms·

discussed previously at: https://news.ycombinator.com/item?id=6589508 https://news.ycombinator.com/item?id=6589508 I remember the week after this. Everyone I

discussed previously at: https://news.ycombinator.com/item?id=6589508 https://news.ycombinator.com/item?id=6589508 I remember the week after this. Everyone I knew who worked at a fund was going over their code and also updating their Compliance documents covering testing and deployment of automated code. As a side note one of hte biggest ways funds tend to get in trouble from their regulators is to not follow the steps outlined in their compliance manual. Its been my experience that regulators care more that you follow the steps in your manual than those steps necessary being the best way to do something. I came away from this thinking the worst part of this was that their system did send them errors, its just that when you deal with billions of events emailing errors just tend to get ignored as at that scale you generate so many false positives with logging. I still don't know the best way to monitor and alert users for large distributed systems. The other take away was that this wasn't just a software issue but a deployment issue as well. It wasn't just one root cause but a number of issues that built up to cause the issue. 1) New exchange feature going live so this is the first day you are actually running live with this feature 2) old code left in the system long after it was done being used 3) re-purposed command flag that used to call the old code, but now is used in the new code 4) only a partial deployment leaving both old and new code working together. 5) inability to quickly diagnose where the problem was 6) you are also managing client orders and have the equivalent of an SLA with them so you don't want to go nuclear and shut down everything

LeifCarrotson 8y ago
> I came away from this thinking the worst part of this was that their system did send them errors, its just that when you deal with billions of events emailing errors just tend to get ignored as at that scale you generate so many false positives with logging. I write apps that generate lots of logs too...I think an improvement lies in some form of automated algorithmic/machine learning (to incorporate a buzzword in your pitch) log analysis. When I page through the log in a text editor, or watch `tail` if it's live, there's a lot of stuff that looks like TRACE: 2019-04-01 09:45:03 ID A1D65F19: Request 1234 initiated ERROR: 2019-04-01 09:45:04 ID A1D65F19: NumberFormatException: '' is not a valid number in ProfileParser, line 127 WARN : 2019-04-01 09:45:04 ID A1D65F19: Profile incomplete, default values used WARN : 2019-04-01 09:45:14 ID A1D65F19: Timeout: Service did not respond within 10 seconds TRACE: 2019-04-01 09:45:14 ID A1D65F19: Request 1234 completed. A = 187263, B = 1.8423, C = $-85.12, T = 11.15s Visually (or through regex), you can filter out all the "Request initiated" noise. Maybe the default value warning occurs 10% of the time, and is usually accompanied by that number format exception (which somebody should address, but it still functions, and there's other stuff to fix). But maybe the "Timeout error" hasn't been seen in weeks, and the value of C has always been positive - that is useful information! Don't email me when there's a profile incomplete warning. Don't email me any time there's an "ERROR" entry, because that just makes people reluctant to use error level logging. Definitely don't email me when there's a unique request complete string, that's trivially different every time. But do let me know when something weird is going on!
- nradov 8y ago
  There's nothing particularly wrong with e-mailing (or using other push notification mechanisms) when a warning or error occurs. When those logging levels are properly used it means some sort of administrator intervention is needed. But send the notification only once. Don't keep spamming out the same notification over and over.
- tracker1 8y ago
  I once worked in an envrionment like this that saw about 1k emails a day when I started... What I started doing was trying to resolve one bug a day... I'd look at an error, do a search in my email filtered folder for that error, and manually count them, whichever had the most emails, I took care of it. About 2/3 just came down to filtering certain classes of errors. 4xx errors, I stopped email notification altogether, since they were already being trapped/handled by the system. Others were a little more specific. Ironically, .Net tends to handle some things that should be 4xx errors as 500, so reclassifying those took out a lot as well. In the end, within about a month, the emails were down to a manageable 20 or so a day and got more visibility as a result.
- exelius 8y ago
  Don’t mean to sound snarky, but there are tools that do this and have been for years. If you’ve been grepping through logs for the last 3 years, you’re doing it wrong for the cloud era. Often times the answer is writing better alert triggers that take historical activity into account to cut down on false positives. Other times it’s simply to reduce the number of alerts. In every case you need an alerting strategy that takes balances stakeholder needs, and you need to realign on that strategy quarterly. It’s ultimately an operational problem, not a technical one. Alas, back in the real world, logging is always the last thing teams have time to think about...
  - mattdodge 8y ago
    Care to share what types of tools do this? I'm genuinely interested. I haven't come across a log management tool that uses AI to detect abnormal conditions based on the log message contents like the OP describes. I stick to Papertrail for the most part though so I'm likely out of the loop.
BonesJustice 8y ago
Not only did they leave old code in the system, but they made changes to it without reviewing whether the old code would still work correctly, despite still being callable: In 2003, Knight ceased using the Power Peg functionality. In 2005, Knight moved the tracking of cumulative shares function in the Power Peg code to an earlier point in the SMARS code sequence. Knight did not retest the Power Peg code after moving the cumulative quantity function to determine whether Power Peg would still function correctly if called. If you want to leave the old code in, fine, but then it still needs to be tested.
prokes 8y ago
Reading this I was considering point #6 and shutting down the system. If the proper alerts had shown a system improperly taking on massive positions, and the related risk in dollar terms, wouldn't shutting down be a better route than bankruptcy? In retrospect, would (could?) they have chosen that? Of course, violating those SLA's could cause bankruptcy through client dissatisfaction but that seems less certain than bleeding out money.
zeveb 8y ago
> I still don't know the best way to monitor and alert users for large distributed systems. I'd imagine that this is somewhere statistical process control would apply: if you're dealing with a system in which errors are expected, then monitor frequency & magnitude, and alert when they fall outside of one or two standard deviations from the mean. For a financial company, you'd probably want a graph of net worth or somesuch, and alert when it falls outside of one standard deviation. If you don't have the IT do calculate net worth to at least hourly granularity, then get there, and aim for to-the-minute granularity. This shouldn't be hard, but it might be, and if it is then it's worth fixing.
deleted 8y ago
[deleted]
csomar 8y ago
> Its been my experience that regulators care more that you follow the steps in your manual than those steps necessary being the best way to do something. Not a joke. Bitfinex lost its Bitcoins by being compliant (regulators required that users wallets are displayed on the blockchain). It'd have been safer to just have a single cold wallet.

LeifCarrotson 8y ago

> I came away from this thinking the worst part of this was that their system did send them errors, its just that when you deal with billions of events emailing errors just tend to get ignored as at that scale you generate so many false positives with logging. I write apps that generate lots of logs too...I think an improvement lies in some form of automated algorithmic/machine learning (to incorporate a buzzword in your pitch) log analysis. When I page through the log in a text editor, or watch `tail` if it's live, there's a lot of stuff that looks like TRACE: 2019-04-01 09:45:03 ID A1D65F19: Request 1234 initiated ERROR: 2019-04-01 09:45:04 ID A1D65F19: NumberFormatException: '' is not a valid number in ProfileParser, line 127 WARN : 2019-04-01 09:45:04 ID A1D65F19: Profile incomplete, default values used WARN : 2019-04-01 09:45:14 ID A1D65F19: Timeout: Service did not respond within 10 seconds TRACE: 2019-04-01 09:45:14 ID A1D65F19: Request 1234 completed. A = 187263, B = 1.8423, C = $-85.12, T = 11.15s Visually (or through regex), you can filter out all the "Request initiated" noise. Maybe the default value warning occurs 10% of the time, and is usually accompanied by that number format exception (which somebody should address, but it still functions, and there's other stuff to fix). But maybe the "Timeout error" hasn't been seen in weeks, and the value of C has always been positive - that is useful information! Don't email me when there's a profile incomplete warning. Don't email me any time there's an "ERROR" entry, because that just makes people reluctant to use error level logging. Definitely don't email me when there's a unique request complete string, that's trivially different every time. But do let me know when something weird is going on!

nradov 8y ago
There's nothing particularly wrong with e-mailing (or using other push notification mechanisms) when a warning or error occurs. When those logging levels are properly used it means some sort of administrator intervention is needed. But send the notification only once. Don't keep spamming out the same notification over and over.
tracker1 8y ago
I once worked in an envrionment like this that saw about 1k emails a day when I started... What I started doing was trying to resolve one bug a day... I'd look at an error, do a search in my email filtered folder for that error, and manually count them, whichever had the most emails, I took care of it. About 2/3 just came down to filtering certain classes of errors. 4xx errors, I stopped email notification altogether, since they were already being trapped/handled by the system. Others were a little more specific. Ironically, .Net tends to handle some things that should be 4xx errors as 500, so reclassifying those took out a lot as well. In the end, within about a month, the emails were down to a manageable 20 or so a day and got more visibility as a result.
exelius 8y ago
Don’t mean to sound snarky, but there are tools that do this and have been for years. If you’ve been grepping through logs for the last 3 years, you’re doing it wrong for the cloud era. Often times the answer is writing better alert triggers that take historical activity into account to cut down on false positives. Other times it’s simply to reduce the number of alerts. In every case you need an alerting strategy that takes balances stakeholder needs, and you need to realign on that strategy quarterly. It’s ultimately an operational problem, not a technical one. Alas, back in the real world, logging is always the last thing teams have time to think about...
- mattdodge 8y ago
  Care to share what types of tools do this? I'm genuinely interested. I haven't come across a log management tool that uses AI to detect abnormal conditions based on the log message contents like the OP describes. I stick to Papertrail for the most part though so I'm likely out of the loop.

BonesJustice 8y ago

Not only did they leave old code in the system, but they made changes to it without reviewing whether the old code would still work correctly, despite still being callable: In 2003, Knight ceased using the Power Peg functionality. In 2005, Knight moved the tracking of cumulative shares function in the Power Peg code to an earlier point in the SMARS code sequence. Knight did not retest the Power Peg code after moving the cumulative quantity function to determine whether Power Peg would still function correctly if called. If you want to leave the old code in, fine, but then it still needs to be tested.

prokes 8y ago

Reading this I was considering point #6 and shutting down the system. If the proper alerts had shown a system improperly taking on massive positions, and the related risk in dollar terms, wouldn't shutting down be a better route than bankruptcy? In retrospect, would (could?) they have chosen that? Of course, violating those SLA's could cause bankruptcy through client dissatisfaction but that seems less certain than bleeding out money.

zeveb 8y ago

> I still don't know the best way to monitor and alert users for large distributed systems. I'd imagine that this is somewhere statistical process control would apply: if you're dealing with a system in which errors are expected, then monitor frequency & magnitude, and alert when they fall outside of one or two standard deviations from the mean. For a financial company, you'd probably want a graph of net worth or somesuch, and alert when it falls outside of one standard deviation. If you don't have the IT do calculate net worth to at least hourly granularity, then get there, and aim for to-the-minute granularity. This shouldn't be hard, but it might be, and if it is then it's worth fixing.

deleted 8y ago

[deleted]

csomar 8y ago

> Its been my experience that regulators care more that you follow the steps in your manual than those steps necessary being the best way to do something. Not a joke. Bitfinex lost its Bitcoins by being compliant (regulators required that users wallets are displayed on the blockchain). It'd have been safer to just have a single cold wallet.