6 ms·

> Things like logging just could not be done. Had that problem at an ad tech company before. We just did sampling so that 1 in 10 or 1 in 20 requests would get

> Things like logging just could not be done. Had that problem at an ad tech company before. We just did sampling so that 1 in 10 or 1 in 20 requests would get logging. It definitely made me wonder how many web servers out there have their performance determined by the rate at which logs can be written to disk.

throwaway0a5e 6y ago
>It definitely made me wonder how many web servers out there have their performance determined by the rate at which logs can be written to disk. As someone in the HFT space I suspect a lot of traders are performance bound by this. Wouldn't surprise me at all if a couple darkpools were this way.
- dmurray 6y ago
  As someone formerly in that space I don't think so, assuming by "traders" you mean the prop firms who actually need the speed advantage. You can always arrange for the logging to happen outside the critical path, which is a tiny minority of the code you run. And you will do that if its necessary to gain a fraction of a microsecond on a competitive trade. Even if it's just being able to run a build with logging off for special cases. For a looser definition of "traders" that includes dark pool operators and exchanges, then sure. Writing logs was definitely the bottleneck at one major European exchange for some years. They may not be able to turn off logging for regulatory reasons, but they also don't care so much about the microseconds. In this case they could definitely have improved performance with horizontal scaling (have each computer handle fewer stock symbols) but didn't consider that economically justified.
  - throwaway0a5e 6y ago
    By traders I mean "anyone who doesn't sit on the market side of my network card", so a wide variety of firms. My comment about darkpools was tangential but there's a couple of them who aren't exactly known for running a tight ship. >You can always arrange for the logging to happen outside the critical path, which is a tiny minority of the code you run. And you will do that if its necessary to gain a fraction of a microsecond on a competitive trade. Even if it's just being able to run a build with logging off for special cases. Yes you can do that. Yes you'd think that anyone for whom it would be beneficial would do that... but you'd be surprised how often people run with logs cranked to 11 in prod.
- tyingq 6y ago
  I've had the opposite. Where we couldn't turn logging off because the writes prevented a weird race condition that would crash the application.
  - colinmhayes 6y ago
    Lmao why use data synchronization when you can just write to disk?
    - tyingq 6y ago
      Heh. It was a bit hard to pin down, but we did eventually fix it. One of those Java super deep dependency things where the 3rd party thing at the bottom is broken. I think it was cxf.
    - throwaway894345 6y ago
      Why write to disk when you can just sleep() sporadically?
- ashtonkem 6y ago
  Trading systems often record every single packet going through the network using fiber optic splitters, which makes it pretty easy to replay traffic to a debugging instance.
sleepybrett 6y ago
Most teams don't understand the cost of logging on their infrastructure, they just seem to thin writing logs is 'free'.

throwaway0a5e 6y ago

>It definitely made me wonder how many web servers out there have their performance determined by the rate at which logs can be written to disk. As someone in the HFT space I suspect a lot of traders are performance bound by this. Wouldn't surprise me at all if a couple darkpools were this way.

dmurray 6y ago
As someone formerly in that space I don't think so, assuming by "traders" you mean the prop firms who actually need the speed advantage. You can always arrange for the logging to happen outside the critical path, which is a tiny minority of the code you run. And you will do that if its necessary to gain a fraction of a microsecond on a competitive trade. Even if it's just being able to run a build with logging off for special cases. For a looser definition of "traders" that includes dark pool operators and exchanges, then sure. Writing logs was definitely the bottleneck at one major European exchange for some years. They may not be able to turn off logging for regulatory reasons, but they also don't care so much about the microseconds. In this case they could definitely have improved performance with horizontal scaling (have each computer handle fewer stock symbols) but didn't consider that economically justified.
- throwaway0a5e 6y ago
  By traders I mean "anyone who doesn't sit on the market side of my network card", so a wide variety of firms. My comment about darkpools was tangential but there's a couple of them who aren't exactly known for running a tight ship. >You can always arrange for the logging to happen outside the critical path, which is a tiny minority of the code you run. And you will do that if its necessary to gain a fraction of a microsecond on a competitive trade. Even if it's just being able to run a build with logging off for special cases. Yes you can do that. Yes you'd think that anyone for whom it would be beneficial would do that... but you'd be surprised how often people run with logs cranked to 11 in prod.
tyingq 6y ago
I've had the opposite. Where we couldn't turn logging off because the writes prevented a weird race condition that would crash the application.
- colinmhayes 6y ago
  Lmao why use data synchronization when you can just write to disk?
  - tyingq 6y ago
    Heh. It was a bit hard to pin down, but we did eventually fix it. One of those Java super deep dependency things where the 3rd party thing at the bottom is broken. I think it was cxf.
  - throwaway894345 6y ago
    Why write to disk when you can just sleep() sporadically?
ashtonkem 6y ago
Trading systems often record every single packet going through the network using fiber optic splitters, which makes it pretty easy to replay traffic to a debugging instance.

sleepybrett 6y ago

Most teams don't understand the cost of logging on their infrastructure, they just seem to thin writing logs is 'free'.