4 ms·

Wait, pcap as in wireshark packet capture?

by ajb 2y ago

Loic 2y ago
I suppose Pre-Calculated Aggregated Positions, but I am not an expert in the field.
w23j 2y ago
I would also really like to know that! It generally seems to be a thing in trading: https://databento.com/pcaps https://databento.com/pcaps There is also this (though this page does not specify what pcap means): https://www.lseg.com/en/data-analytics/market-data/data-feeds/tick-history/tick-history-pcap https://www.lseg.com/en/data-analytics/market-data/data-feed...
- hnbear 2y ago
  Look up Corvil devices by Pico. Commonly used in finance. https://www.pico.net/corvil-analytics/ https://www.pico.net/corvil-analytics/
tnlnbn 2y ago
I'm not the commenter, but yes, often trading firms record all order gateway traffic to from brokers or exchanges at the TCP/IP packet level, in what are referred to as "pcap files". Awkwardly low-level to work with, but it means you know for sure what you sent, not what your software thought it was sending!
- pclmulqdq 2y ago
  The ultimate source of truth about what orders you sent to the exchange is the exact set of bits sent to the exchange. This is very important because your software can have bugs (and so can theirs), so using the packet captures from that wire directly is the only real way to know what really happened.
  - generic92034 2y ago
    But then the software capturing, storing and displaying the packets can also have bugs.
    - bostik 2y ago
      Among all the software installed in a reputable Linux system, tcpdump and libpcap are some of the most battle tested pieces one can find. Wireshark has bugs, yes. Mostly in the dissectors and in the UI. But the packet capture itself is through libpcap. Also, to point out the obvious: pcap viewers in turn are auditable if and when necessary.
      - wjholden 2y ago
        Cisco switches can mirror ports with a feature called Switch Port Analyzer (SPAN). For a monitored port, one can specify the direction (frames in, out, or both), and the destination port or VLAN. SPAN ports are great for network troubleshooting. They're also nice for security monitors, such as an intrusion detection system. The IDS logically sees traffic "on-line," but completely transparent to users. If the IDS fails, traffic fails open (which wouldn't be acceptable in some circumstances, but it all depends on your priorities).
        generic92034 2y ago
        When I think "Cisco" I think error-free. /s No, really, I get where you and your parent are coming from. It is a low probability. But occasionally there is also thoroughly verified application code out there. That is when you are asking yourself where the error really is. It could be any layer.
    - kortilla 2y ago
      They can, but it’s far less likely to be incorrect on the capture side. They are just moving bytes, not really doing anything with structured data. Parsing the pcaps is much more prone to bugs than capturing and storing, but that’s easier to verify with deserialize/serialize equality checks.
- baq 2y ago
  That makes sense - but it's still somewhat surprising that there's nothing better. I guess that's the equivalent of the modern paper trail.
  - HolyLampshade 2y ago
    It’s the closest to truth you can find (the network capture, not the drop copy). If it wasn’t on the network outbound, you didn’t send it, and it’s pretty damn close to an immutable record.
    - ajb 2y ago
      It makes sense. I'm a little surprised that they'd do the day to day reconciliation from it but I suppose if you had to write the code to decode them anyway for some exceptional purpose, you might as well use it day to day as well.
- thomasjudge 2y ago
  The storage requirements of this must be impressive
  - bostik 2y ago
    Storage is cheap, and the overall figures are not that outlandish. If we look at a suitable first page search result[0], and round figures up we get to about 700 GB per day. And how did I get that figure? I'm going to fold pcap overhead into the per-message size estimate. Let's assume a trading day at an exchange, including after hours activity, is 14 hours. (~50k seconds) If we estimate that during the highest peaks of trading activity the exchange receives about 2M messages per second, then during more serene hours the average could be about 500k messages per second. Let's guess that the average rate applies 95% of the time and the peak rate the remaining 5% of the time. That gives us an average rate of about 575k messages per second. Round that up to 600k. If we assume that an average FIX message is about 200 bytes of data, and add 50 bytes of IP + pcap framing overhead, we get to ~250 bytes of transmitted data per message. At 600k messages per second, 14 hours a day, the total amount of trading data received by an exchange would then be slightly less than 710GB per day. Before compression for longer-term storage. Whether you consider the aggregate storage requirements impressive or merely slightly inconvenient is a more personal matter. 0: https://robertwray.co.uk/blog/the-anatomy-of-a-fix-message https://robertwray.co.uk/blog/the-anatomy-of-a-fix-message
    - tetha 2y ago
      And compression and deduplication should be very happy with this. A lot of the message contents and the IP/pcap framing overheads should be pretty low-entropy and have enough patterns to deduplicate. It could be funny though because you could be able to bump up your archive storage requirements by changing an IP address, or have someone else do that. But that's life.
  - oblio 2y ago
    Why? They're not streaming 4k video, it's either text protocol or efficient binary protocols.
- chairmansteve 2y ago
  The result of bitter lessons learnt I'm sure. Lessons the fintechs have not learned.
hnbear 2y ago
Typically not a literal pcap. Not just wireshsrk running persistently everywhere. There are systems you can buy (eg by Pico) that you mirror all traffic to and they store it, index it, and have pre-configured parsers for a lot of protocols to make querying easier. Think Splunk/ELK for network traffic by packet.
- hn_version_0023 2y ago
  The Corvil devices used by Pico have IME largely been replaced by Arista 7130 Metamux platforms at the capture “edge”
  - HolyLampshade 2y ago
    Which is great for the companies that have made the switch because those corvils were truly terrible.
- cjalmeida 2y ago
  Except it is literal “pcap” as they capture all packets at layer 3. I don’t know the exact specifications of Pico appliances, but it would not surprise me they’re running Linux + libpcap + some sort of timeseries DB
  - hnbear 2y ago
    Well, probably, but I meant more like it's not typically someone running tcpdump everywhere and someone analyzing with Wireshark, rather than a systems configured to do this at scale across the desktop.
    - kelnos 2y ago
      I don't think that's what anyone was assuming. A "pcap" is a file format for serialized network packets, not a particular application that generates them.
HolyLampshade 2y ago
Looks like tnlnbn already answered, but the other benefit to having a raw network capture is often this is performed on devices (pico and exablaze just to name two) that provide very precise timestamping on a packet by packet basis, typically as some additional bytes prepended to the header. Most modern trading systems performing competitive high frequency or event trades have performance thesholds in the tens of nanos, and the only place to land at that sort of precision is running analysis on a stable hardware clock.

Wait, pcap as in wireshark packet capture?

by ajb 2y ago

Wait, pcap as in wireshark packet capture?

Loic 2y ago

I suppose Pre-Calculated Aggregated Positions, but I am not an expert in the field.

w23j 2y ago

I would also really like to know that! It generally seems to be a thing in trading: https://databento.com/pcaps https://databento.com/pcaps There is also this (though this page does not specify what pcap means): https://www.lseg.com/en/data-analytics/market-data/data-feeds/tick-history/tick-history-pcap https://www.lseg.com/en/data-analytics/market-data/data-feed...

hnbear 2y ago
Look up Corvil devices by Pico. Commonly used in finance. https://www.pico.net/corvil-analytics/ https://www.pico.net/corvil-analytics/

tnlnbn 2y ago

I'm not the commenter, but yes, often trading firms record all order gateway traffic to from brokers or exchanges at the TCP/IP packet level, in what are referred to as "pcap files". Awkwardly low-level to work with, but it means you know for sure what you sent, not what your software thought it was sending!

pclmulqdq 2y ago
The ultimate source of truth about what orders you sent to the exchange is the exact set of bits sent to the exchange. This is very important because your software can have bugs (and so can theirs), so using the packet captures from that wire directly is the only real way to know what really happened.
- generic92034 2y ago
  But then the software capturing, storing and displaying the packets can also have bugs.
  - bostik 2y ago
    Among all the software installed in a reputable Linux system, tcpdump and libpcap are some of the most battle tested pieces one can find. Wireshark has bugs, yes. Mostly in the dissectors and in the UI. But the packet capture itself is through libpcap. Also, to point out the obvious: pcap viewers in turn are auditable if and when necessary.
    - wjholden 2y ago
      Cisco switches can mirror ports with a feature called Switch Port Analyzer (SPAN). For a monitored port, one can specify the direction (frames in, out, or both), and the destination port or VLAN. SPAN ports are great for network troubleshooting. They're also nice for security monitors, such as an intrusion detection system. The IDS logically sees traffic "on-line," but completely transparent to users. If the IDS fails, traffic fails open (which wouldn't be acceptable in some circumstances, but it all depends on your priorities).
      - generic92034 2y ago
        When I think "Cisco" I think error-free. /s No, really, I get where you and your parent are coming from. It is a low probability. But occasionally there is also thoroughly verified application code out there. That is when you are asking yourself where the error really is. It could be any layer.
  - kortilla 2y ago
    They can, but it’s far less likely to be incorrect on the capture side. They are just moving bytes, not really doing anything with structured data. Parsing the pcaps is much more prone to bugs than capturing and storing, but that’s easier to verify with deserialize/serialize equality checks.
baq 2y ago
That makes sense - but it's still somewhat surprising that there's nothing better. I guess that's the equivalent of the modern paper trail.
- HolyLampshade 2y ago
  It’s the closest to truth you can find (the network capture, not the drop copy). If it wasn’t on the network outbound, you didn’t send it, and it’s pretty damn close to an immutable record.
  - ajb 2y ago
    It makes sense. I'm a little surprised that they'd do the day to day reconciliation from it but I suppose if you had to write the code to decode them anyway for some exceptional purpose, you might as well use it day to day as well.
thomasjudge 2y ago
The storage requirements of this must be impressive
- bostik 2y ago
  Storage is cheap, and the overall figures are not that outlandish. If we look at a suitable first page search result[0], and round figures up we get to about 700 GB per day. And how did I get that figure? I'm going to fold pcap overhead into the per-message size estimate. Let's assume a trading day at an exchange, including after hours activity, is 14 hours. (~50k seconds) If we estimate that during the highest peaks of trading activity the exchange receives about 2M messages per second, then during more serene hours the average could be about 500k messages per second. Let's guess that the average rate applies 95% of the time and the peak rate the remaining 5% of the time. That gives us an average rate of about 575k messages per second. Round that up to 600k. If we assume that an average FIX message is about 200 bytes of data, and add 50 bytes of IP + pcap framing overhead, we get to ~250 bytes of transmitted data per message. At 600k messages per second, 14 hours a day, the total amount of trading data received by an exchange would then be slightly less than 710GB per day. Before compression for longer-term storage. Whether you consider the aggregate storage requirements impressive or merely slightly inconvenient is a more personal matter. 0: https://robertwray.co.uk/blog/the-anatomy-of-a-fix-message https://robertwray.co.uk/blog/the-anatomy-of-a-fix-message
  - tetha 2y ago
    And compression and deduplication should be very happy with this. A lot of the message contents and the IP/pcap framing overheads should be pretty low-entropy and have enough patterns to deduplicate. It could be funny though because you could be able to bump up your archive storage requirements by changing an IP address, or have someone else do that. But that's life.
- oblio 2y ago
  Why? They're not streaming 4k video, it's either text protocol or efficient binary protocols.
chairmansteve 2y ago
The result of bitter lessons learnt I'm sure. Lessons the fintechs have not learned.

hnbear 2y ago

Typically not a literal pcap. Not just wireshsrk running persistently everywhere. There are systems you can buy (eg by Pico) that you mirror all traffic to and they store it, index it, and have pre-configured parsers for a lot of protocols to make querying easier. Think Splunk/ELK for network traffic by packet.

hn_version_0023 2y ago
The Corvil devices used by Pico have IME largely been replaced by Arista 7130 Metamux platforms at the capture “edge”
- HolyLampshade 2y ago
  Which is great for the companies that have made the switch because those corvils were truly terrible.
cjalmeida 2y ago
Except it is literal “pcap” as they capture all packets at layer 3. I don’t know the exact specifications of Pico appliances, but it would not surprise me they’re running Linux + libpcap + some sort of timeseries DB
- hnbear 2y ago
  Well, probably, but I meant more like it's not typically someone running tcpdump everywhere and someone analyzing with Wireshark, rather than a systems configured to do this at scale across the desktop.
  - kelnos 2y ago
    I don't think that's what anyone was assuming. A "pcap" is a file format for serialized network packets, not a particular application that generates them.

HolyLampshade 2y ago

Looks like tnlnbn already answered, but the other benefit to having a raw network capture is often this is performed on devices (pico and exablaze just to name two) that provide very precise timestamping on a packet by packet basis, typically as some additional bytes prepended to the header. Most modern trading systems performing competitive high frequency or event trades have performance thesholds in the tens of nanos, and the only place to land at that sort of precision is running analysis on a stable hardware clock.