5 ms·
Pcaps are remarkably versatile. The choice to store two lengths in the packet header means that you can record both the actual wire length--the number of bytes
by ncmncm 7y ago
Pcaps are remarkably versatile. The choice to store two lengths in the packet header means that you can record both the actual wire length--the number of bytes that came in--and a bigger number that says where the next record starts, and any metadata you like in the gap.
(This wasn't the original purpose of the second number: the original thought was that you would not be able to afford to store all the bytes of every packet, so the second number would say where you cut it off.)
The 32-bit seconds / 32-bit microseconds timestamp in the packet header has evolved to use 32-bit nanoseconds as we got better at timekeeping, with just a change of magic number. Meanwhile, the 32-bit unsigned seconds count (since 1970, UTC) takes us all the way to C.E.2106 while the time_t crybabies are running in circles about 2038.
The whole fintech industry rides on lz4 pcaps of multicast packet streams, with essential file metadata recorded in the file names. (There is a very complicated "NG" pcap format, but nobody uses it.)
Just for the record, a single 1U server with a disk array can record the complete activity of all the New York exchanges plus CME, CFE, Impact, and OPRA with enough headroom to handle >2x peak daily traffic of previous years, as happened in each of the last three weeks: but only if programmed with C++, ring buffers, and kernel bypass network drivers.
This has been a PSA.
- lmeyerov 7y agoInteresting, thanks! RE:NYSE, any sense of avg+peak bytes/s? (Context being we work on interactive GPU visual analytics for related stuff, so always love fun targets to drive benchmarks!)
- ncmncm 7y agoDaily OPRA traffic has lately exceeded 6 TB, compressed. So, probably 15 TB. Figure 58 bytes of header + ~50 of payload per packet. Note the "all NY exchanges" includes a whole hell of a lot more than NYSE.
- lmeyerov 7y agoAh neat -- 15TB/s or /d? Single GPUs are at TB/s levels, so interesting locality + pricing implications here!
- ncmncm 7y agoPer day. But OPRA is only a (large) part of the traffic. It is hard to see how GPUs figure here. This is a wide-area, latency-sensitive data distribution system. In fintech we say a microsecond is an eon, a millisecond an eternity. Musk's LEO satellite network will cut several ms from the Europe - US transmission delay, which will be worth billions to certain organizations. Musk might collect additional billions for denying it to certain others.
- lmeyerov 7y agoMost relevant to the thread, if someone wants to do easy analytics on pcap, enabling regular dataframe / sql / dashboarding code to work on that. Not interesting to traders, but of it/sec/ops, seems so! Both for forensic, monitoring, & controls. For financial use cases, the most sensitive autotrading, yeah, I'd expect ASICS for serving simple models to win for stable algs, and I'm sure all sorts of other interesting things. At the same time, I'd expect others would enjoy something more manageable when less sensitive. Improving the whole Excel / Bloomberg / Factset etc. experience is probably a saner starting point though. I'd turn it around - where would you expect the most useful starting points to be?
- viraptor 7y ago> the original thought was that you would not be able to afford to store all the bytes of every packet I would phrase that differently. The idea was that you can explicitly capture shorter packets, because often you're interested in the flow/metadata only. That means you can still capture all the protocol and application headers/commands without storing the whole 1.5KB packet. This is very handy when you capture on a non-dedicated hardware for ad-hoc analysis. Especially if you only have a vanilla kernel and capture using tcpdump.
- p0cc 7y ago> The choice to store two lengths in the packet header means that you can record both the actual wire length--the number of bytes that came in--and a bigger number that says where the next record starts, and any metadata you like in the gap. > There is a very complicated "NG" pcap format, but nobody uses it. This is the reason that pcapng was created - to formalize the structure of this metadata. pcapng "Pcap Next Generation" (2006) is an update to the pcap format (which originated with libpcap/tcpdump in the 90s). It's used in the networking industry as it's the default output file of wireshark, tshark, and tcpdump to capture traffic. If you use these tools, chances are that you have pcapng files. The biggest reason to use it over pcap (in networking) is that you can save packets from multiple network interfaces. If you have a wireless access point and want to capture both ethernet and 802.11 traffic in the same capture, pcapng would be necessary.
- ncmncm 7y agoThe reason it is not used in fintech is that it is extremely complex. The greatest failing of wireshark et al today is not understanding lz4 and zstd compression.
- touisteur 7y agoOh come on. Ported my pcap parser in 3 hours, and that included reading the docs and looking up a example capture. SDB is the 'file header' packet. IDB is the 'interface description' packet. EPB is the 'data' packet. The rest you should ignore except if you need to look through other block kinds, and in this case you're quite happy to have pcap-ng. No custom block in pcap, except forking wireshark... All the ethernet/ppp/802.11 after that is the same as with pcap. What do people find so complicated there ? Also pcap-ng is quite the generic data-recording format. Merging internal logs, network capture, interface stats, system configuration state, whatever you might want... In a /simple/ unique format. Easy to write, easy to have it read by wireshark... Even if it's not only network related. The real strength of pcap-ng is that you can read it in both directions. Size is at the start /and/ end of each block. Seems stupid but it's very useful for some kind of analysis. Used to build indexes for pcap files... Not so useful now. And anyway, if you want to build an index, you can just add it as custom packet at the end.