4 ms·
Had a VoIP network with TB+ of SIP packets each day. Customers demand resolution to problems from days ago, so having a time-travelling, content-aware PCAP was
by MichaelGG 13y ago
Had a VoIP network with TB+ of SIP packets each day. Customers demand resolution to problems from days ago, so having a time-travelling, content-aware PCAP was necessary. At 50K packets/sec, piping tshark to mysql and 11 indexes simply wouldn't cut it. We spent $$$$$$ on a commercial system that didn't work so well.
I slowly reinvented the basics of an information retrieval system (the curse of not having taken CS classes). Came up with the idea of a log-structured merge tree, made easier by this being a write-once database. Got some inspiration from the original Google paper. But most of it was just figuring out the least number of actions needed to retrieve info.
I published the core DB part, which maps an int64 (index hash value) to an int64 (docId) and stores in an efficient format (on our data, ~2.3 bits per packet). http://github.com/michaelgg/cidb http://github.com/michaelgg/cidb - I couldn't find an existing library that has zero/low per-record overhead.
On a Q6600 and a single 7200RPM platter, I was able to index a TB of SIP a day and provide fairly quick flow reconstructions going back as far as disk space allowed. On a quad-core i7 parsing+indexing was over 1Gbps.
Company impact was huge, because we could suddenly troubleshoot things in minutes instead of hours. A few years later, after I was gone, I heard they were still using it. Neat.
This was all in F#, which presented fun challenges regarding optimization. Lots of unsafe code and manual memory management. SSE would improve varint encoding - the CLR generated code is a joke in comparison.
Last month, I dropped this lib in as a replacement for storing routing information in a telecom app and dropped RAM requirements from 6GB to 1GB.
On the downside, I'm sure any compsci student could build a similar thing in a week, and probably they do so for school projects. But to a lot of app-level developers, this kind of algorithmic work is sorta magic for some reason.
- smikhanov 13y agoSounds like a great tool -- I would certainly use something like this during my telecom days. At the end, did you still use tshark to pipe input data to your database?
- MichaelGG 13y agoNo, I wrote a fully custom SIP parser and database format. If I was doing it again, I'd look at stealing a SIP parser from another project. Scale out and don't worry about every CPU cycle. I thought about commercializing it but interest seemed lukewarm.
- smikhanov 13y agoBut how did you capture the actual bytes on the network (that you would later parse as SIP)?
- MichaelGG 13y agolibpcap. For my first pass I just setup a mirror port (SPAN) port from a core switch. libpcap has efficient filters so non-SIP (non port 5060) traffic was just dropped. Looking forward: Intel 10G NICs have little processor in them, and there's a library called DNA (Direct NIC Access) that bypasses kernel mode for packet capture/transmit. Makes it super efficient and possible to do line-rate 10G apps on commodity hardware. Those processors can also distribute packets to specific cores, which helps solves the next problem of scaling the capture part beyond a single core. Looking beyond 10G to 40G and beyond, I'd imagine something similar to Etherchannelling would be a cheap way to leverage existing hardware into breaking up the load. Unfortunately a low-level "dumb" load balancing system will split up dialogs so your indexing and compression gets a bit less efficient. One of my semi-open research projects is to do line-rate SIP DDoS protection. So far, no one (not commercially, not academically) has any magic bullet and it would appear that almost every SIP network out there is trivially DoS-able. State-of-the-art is along the lines of "we push a whitelist into our switches" (which, actually, technically doesn't even work with SIP since multiple IPs can get involved with the signalling, IPs unrelated to original hosts - although I've only seen this once or twice in production). As far as I can tell, it's mainly an engineering problem - just do the grunt work of writing smart code. But there's no market for it until VoIP providers start suffering real serious pain - until then, I don't think telecom cares about security, DoS or otherwise.
- jonmb 13y agoI'm a senior compsci student at one of the largest (by # of students) universities in the nation. I assure you, most of my classmates could not do such a thing in a week!