17 ms·

What happens if you write a TCP stack in Python?

408 points by jvns 12y ago

latiera 12y ago
It's not Python that's slow, but scapy, which is dog slow. In fact, it is so slow that it should come with big WARNINGs that it's only really meant for interactive use. Do the dissection yourself or use something built for that purpose. It's really surprising to me that lots of ppl are using scapy for things that require performance but then again if you look at the scapy website or the docs, it's not immediately apparent that their tool is not meant for this. Which I guess says a lot about the scapy developers rather than the scapy users. tl;dr Scapy is a joke, performance-wise.
philangist 12y ago
Fun read. Does anyone here know how to deal with the Python being slow at sending ACK packets problem? Or is it a built-in limitation that comes with dealing with high level languages?
- crazypyro 12y ago
  Implement it in C. Also its likely more a python problem than a "high level" language problem.
  - fastball 12y ago
    I blame all networking issues on the GIL.
  - jnazario 12y ago
    there's a userland stack written in C in honeyd. http://honeyd.org/ http://honeyd.org/ (slow to respond) and http://en.wikipedia.org/wiki/Honeyd http://en.wikipedia.org/wiki/Honeyd
    - crazypyro 12y ago
      Except that was written for an explicit purpose that wasn't speed. I fail to see how my suggestion of implementing low level networking in C and linking that to python code is considered a poor solution.
  - mkonecny 12y ago
    This has nothing to do with Python or C. More likely he has a bug in his code.
    - crazypyro 12y ago
      Python being magnitudes slower than C for networking code has a lot to do with Python, actually. Other smarter people have already explained it better than I can up above.
- lmm 12y ago
  The answer to questions about performance is always "profile it". I mean, sure, you could probably get a crude speedup by just running it under pypy or some such, but the real answer is to profile it, see where the bottleneck is, and fix it. As the other post suggests, this is not a problem in all high-level languages; it's possible to write very high-performance code in Haskell or Ocaml, for example. But python's semantics are e.g. that every object has its own methods that can be overridden arbitrarily, which means that every single method call pretty much has to involve a hash table lookup. 99% of Python code never uses this extreme dynamism (usually you define methods on classes, and if you want an instance with a different implementation for a particular method then you make a subclass), but it's there, and a Python interpreter that didn't respect that would behave incorrectly.
  - kyllo 12y ago
    So you're saying that late binding is the most significant reason why Python is slow? Is slowness just an inherent tradeoff in using a language that supports this powerful feature?
    - lmm 12y ago
      I'm not an expert but that's my understanding. Remember that it applies to some scopes as well as to object properties. In benchmarks the big divide I've seen is between languages that allow this and languages that don't. You can do sophisticated things where you compile objects assuming they won't be overridden and then back out the compile if something touches an object (the JVM does similar things where it will compile a never-overridden method as non-virtual and then detect when the class hierarchy changes), but that requires a lot of complexity that goes against the goals of CPython. Nowadays I mostly work in Scala, and anywhere where I would have used such a technique I find there's a way to do it "statically". So I'd be interested if there are good examples of what makes this a "powerful" technique, and to see if I can't replicate them "statically" with enough typeclasses etc.
    - marcosdumay 12y ago
      Not only late biding, but all the Python's powerfull reflection (not only hashes, but it's string hashes all the way down) makes it hard to write a fast compiler. Not impossible, as lmm said, just hard.
- mbell 12y ago
  I'd start with the question of 'why so much ACKing?'. In TCP you don't normally ACK every packet. You really only need one ACK per receive window, not one ACK per packet which is what the op's code appears to be trying to do. It also doesn't look like op's code is setting the window size anywhere so in effect she's saying to google's server 'ok got that one, send me a bunch more' and trying to do that for every packet at fast as possible is creating a runaway congestion problem. The best solution doesn't have anything to do with python, it's to implement window size (flow control) and only ack when you need to. ps - It's been several years since I've worked on a TCP stack so please correct anything I'm remembered incorrectly
mholt 12y ago
ARP spoofing... clever. This was an amusing read and really informative, too. There is definitely something to be said for explaining lower-level concepts (e.g. TCP handshakes) using the common tongue. IMO, not a bad way to begin learning. Someone could perform the same experiment now and use Wireshark to see the raw packets, then draw conclusions to what is happening. Anyone know why the Python program is so slow? I'm looking at the code and my first guess would be this part[1] but I can't explain why, overall, it would be so slow that a remote host would close the connection. [1] https://github.com/jvns/teeceepee/blob/7e93970e16fbb0c3d4bee54bb9fb019b702fae5e/teeceepee/tcp.py#L144 https://github.com/jvns/teeceepee/blob/7e93970e16fbb0c3d4bee...
- delluminatus 12y ago
  I would be interested in seeing if using asyncio would resolve the network issues.
cookiemonster11 12y ago
This is not a TCP stack.
- prht 12y ago
  Indeed
  - cookiemonster11 12y ago
    It confuses me that people make such a big deal of their little 20 lines of code toy projects.
    - michaelhoffman 12y ago
      It's not a little 20 line of code toy project. It's an engaging and accessible writeup of some basic parts of TCP that happens to include some easy-to-understand code. It's pointless to people who understand how TCP works in depth, but the majority of programmers don't.
      - peterwwillis 12y ago
        It's an engaging and accessible writeup, true. Unfortunately, there are also several glaringly incorrect/misleading points in the article. The fact that it's getting upvoted is just..... strange.
    - ivraatiems 12y ago
      I don't think it's presented as a "big deal." I read it as a fun little experiment.
    - prht 12y ago
      More confusing part for me is that I knew there is TCP/IP stack for ages. TCP stack is all new to me.
    - Someone1234 12y ago
      It confuses me that instead of finding these projects uninteresting (as you claim) you took the time and effort to write a reply in a thread just to belittle and dismiss the article. I for one found it interesting/fun.
kesor 12y ago
And there is no mention of ScaPy? http://www.secdev.org/projects/scapy/ http://www.secdev.org/projects/scapy/
- ayrx 12y ago
  " I was much more comfortable with Python than C and I’d recently discovered the scapy networking library which made sending packets really easy." Did you even read the link?
- cookiemonster11 12y ago
  There is, in the second sentence of the article.
- deleted 12y ago
  [deleted]
kfnic 12y ago
Shouldn't the TCP handshake look like this: ---- SYN ----> <-- SYN/ACK -- ---- ACK ----> rather than having the client send two SYNs to the server?
- cookiemonster11 12y ago
  It should.
- jvns 12y ago
  absolutely! Fixed :)
beering 12y ago
I wonder if it would have a better success rate on a site other than Google's, since I've heard that Google's done extensive tuning of their TCP stack to send page data faster. Somebody's oversubscribed $3/month shared PHP hosting might not ramp up the speed as quickly.
- lazyant 12y ago
  yes, they also have a bigger initial TCP window than the default
- ivraatiems 12y ago
  I think it would still be significantly too fast. That oversubscribed PHP hosting is still sitting on a dedicated server that does nothing but serve TCP/IP all day. My hunch is that the code is very heavily optimized for most production servers, because they're all using the TCP/IP stack in the kernel. It won't be as fast as Google's, but it won't be as slow as python either.
- wmf 12y ago
  TCP is TCP. If you advertise a certain window size you better be prepared to receive that data.
kernelwaste 12y ago
This is not an implementation of a TCP stack.
c_plus_minus 12y ago
Ha! Nice read, perfect after lunch material with my coffee :) Good job
wslh 12y ago
And in Squeak (Smalltalk) ? take a look at: http://www.squeaksource.com/@Hl1Cdo4NwCmLqQl0/Im9cEg0J http://www.squeaksource.com/@Hl1Cdo4NwCmLqQl0/Im9cEg0J
- tonyg 12y ago
  I'm guessing you were linking to http://www.squeaksource.com/Net.html http://www.squeaksource.com/Net.html - Seaside uses non-persistent URLs by default, leading to problems like this...
tptacek 12y ago
The idea that Python is so slow that it's confusing TCP sounds wrong to me. I think it's more likely that your packet capture scheme is slow. It looks like you're using scapy, which I assume is in turn using libpcap... which may be buffering (in high-performance network monitoring, the packet capture interface goes out of its way to buffer). Which is something you can turn off. About 13 years ago, I wrote my own programming language expressly for the purpose of implementing network stacks, and had a complete TCP in it; I didn't have this problem. But I do have this problem all the time when I write direct network code and forget about buffering. Similarly: "Python is so slow that Google reset the network connection" seems a bit unlikely too. Google, and TCP in general, deals with slower, less reliable senders than you. :) What's the time between your SYN and their SYN+ACK?
- jvns 12y ago
  my favorite thing about writing blog posts is comments like this. Thank you! I didn't consider that the packet capture interface might do buffering. That might explain a lot of the problems I was having :)
  - tptacek 12y ago
    This is a great project. Keep playing with it! You might find that the serverside of TCP is more useful to have in Python than the clientside (having a userland IP/TCP serverside allows you to create fake hosts out of thin air).
    - blutoot 12y ago
      Your "userland TCP/IP" comment got me excited and I started digging around. I wonder if there's some benefit in trying to port something like TCP Daytona [1,2] in a high-level language like Python. I'm curious if that will somehow advance the adoption of SDNs. [1] https://github.com/jamesbw/tcp-daytona https://github.com/jamesbw/tcp-daytona [2] http://nms.lcs.mit.edu/~kandula/data/daytona.pdf http://nms.lcs.mit.edu/~kandula/data/daytona.pdf
      - 12y ago
bhakk 12y ago
shit?
kalleboo 12y ago
It seems odd to me that Google would time out that quickly. You could never reach Google from a GPRS connection if that was the case. I'd investigate the ACKs you're sending. Are you missing packets or sending them in the wrong order? In Uni we had a networking course where we got to build a network web server software stack from the bottom up, starting with Ethernet/MAC, TCP/IP, and then on the top HTTP, all being flashed onto a small network device (IIRC it was originally a CD-ROM server). It was an extremely enlightening exercise. I recommend you go deeper instead of just using a premade Python library for TCP!
sly010 12y ago
If an 8MHz microcontroller is fast enough to implement TCP, then Python should be fast enough too. Here is my two cents on the expirement: 1. You don't really have to ack every packet, you have to order them, drop duplicates and ack the last one. 2. Google ignores the TCP flow control algorithm and sends the first few hundred packets very quickly without waiting for acks. They do this to beat the latency of the flow control algorithm. That's why you end up with so many packets on your side. You could just try anything but google, and you would probably see that you have a less insane packet storm.
- Dylan16807 12y ago
  Even 8MHz is massive overkill. It's almost fast enough to bit-bang ethernet.
  - jacquesm 12y ago
    I'd like to see you bit-bang a 10MHz signal with an 8 MHz clock. You'd need something substantially faster (Nyquist and such).
    - TickleSteve 12y ago
      TCP != ethernet. A TCP stack can run over serial and othe link layers (PPP, SLIP, etc). PPP is absolutely doable on very small microcontrollers.
      - Sami_Lehtinen 12y ago
        Problem is that using PPP or SLIP doesn't relieve you from having to implement at least some kind of crude TCP stack. But if you use TCP offloading, then you can communicate using TCP as cheaply as doing any traditional serial communication. I've seen those crude TCP stack implementations and those are barely LAN capable. Using those over Internet wouldn't be a great idea. Often these TCP stacks are used just as serial - tcp converters. Using RWIN of 16 (max) bytes due UART buffering limits etc. Not sending ACK before whole 16 byte is cleared and so on. So with really low end devices, it's better to off load TCP to custom hardware. Many embedded devices use that approach. In such cases you'll only call to IP address and port, and when it connects, you'll have your TCP/IP connection in pure serial. TCP-modems isn't any different from other good old Hayes modems. It's just dial-up over Internet and TCP. Btw. If the system is able to run CPython, Linux, or something similar, it's already a pretty powerful system.
blutoot 12y ago
This wins the Internet today for me... "my kernel: lol wtf I never asked for this! RST! my Python program: ... :("
prht 12y ago
I don't know what TCP stack is, but in case if you are interested in implementing TCP/IP stack http://git.savannah.gnu.org/cgit/lwip.git/tree/src http://git.savannah.gnu.org/cgit/lwip.git/tree/src
malone 12y ago
I like your solution to prevent the kernel from interfering with your packets. An alternative method I've used in the past is to add an iptables rule to silently drop the incoming packets. libpcap sees the packets before they are dropped, so you'll still be able to react to them, but the kernel will ignore them (and therefore won't attempt to RST them).
lukego 12y ago
Cool :). I also learned a lot about networking by writing a TCP/IP stack in Common Lisp. http://lukego.livejournal.com/4993.html http://lukego.livejournal.com/4993.html if you are interested.
jnbiche 12y ago
This is a fun write-up. If you enjoy this kind of playing around with networking in a dynamic language, and don't want to have to worry about ARP spoofing to do these kinds of experiments, you may want to take a look a Snabb Switch. It provides userland networking in Lua, connecting to the NIC directly (only a handful of popular NICs currently supported) [0]. I've not used it yet, but I've read over the documentation and am itching for an opportunity to do so. 0. https://github.com/SnabbCo/snabbswitch https://github.com/SnabbCo/snabbswitch
- voltagex_ 12y ago
  I can't find the list of supported NICs in the documentation, but I found https://github.com/SnabbCo/snabbswitch/blob/master/src/lib/hardware/pci.lua#L54 https://github.com/SnabbCo/snabbswitch/blob/master/src/lib/h... which suggests that only a very small subset of Intel NICs are supported. One of those may be emulated by VirtualBox, though.
kashif 12y ago
I have actually implemented a TCP stack in python, unfortunately I can't share it publicly just yet. The authors problems are because she is not using RAW_IP_SOCKETS. Making TCP packets using pythons struct module is a breeze. I can post specific examples in code if anyone is interested. Finally you can write a proper TCP stack in python, there is no reason not to. Your user-space interpreted stack won't be as fast as a compiled kernel-space one - but it won't be feature short. PS: I guess, Google is probably sending him a SSL/TLS handshake which he isn't handling. Edit: Corrected author's gender as mentioned by kind poster.
- blutoot 12y ago
  I believe the author is a "she". It says at the banner of the blog.
  - blutoot 12y ago
    I think this downvoting thing is out of control. What was even remotely offensive about this correction?
    - dragonwriter 12y ago
      "offensive" isn't the only reason for downvoting. "Not a substantive addition to the conversation" is one (of many) other bases. If you are going to complain about downvoting [1] why do so based on the unwarranted assumption that the downmod must be for offensiveness? [1] And you shouldn't, see the Guidelines [2] under "In Comments" [2] https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
      - vacri 12y ago
        In my opinion, if HN doesn't want puzzled people to ask why they've been downmodded, then instead of putting a line in the Guidelines saying "don't do this", they should separate the downmod into "I disagree" and "flag this as a bad comment", with the latter item the one that reduces the text weight. It's clearly broken - people have been doing this for as long as I've been here, and it's not going to change (sadly, 'it' means both the behaviour, and HN's braindead downmod mechanism).
peterwwillis 12y ago
You don't need to spoof a different MAC or IP to implement your own stack on a raw socket, Python is not too slow to handle negotiating a connection, and your interpretation of how tcp/ip works is flawed. I highly recommend you read a good book about tcp/ip and learn how the kernel works with network applications of different types. In terms of using Scapy for your packet crafting, here are some guides with examples that may help you work around your issues. (Hint: use the Scapy-defined sending and receiving routines and don't implement your own, or stop using Scapy and implement your own raw packet sockets) http://securitynik.blogspot.com/2014/05/building-your-own-tcp-3-way-handshake.html http://securitynik.blogspot.com/2014/05/building-your-own-tc... https://github.com/yomimono/client-fuzzball/blob/master/fuzz_tcp.py https://github.com/yomimono/client-fuzzball/blob/master/fuzz... https://www.sans.org/reading-room/whitepapers/detection/ip-fragment-reassembly-scapy-33969 https://www.sans.org/reading-room/whitepapers/detection/ip-f... http://www.lopisec.com/2011/12/learning-scapy-syn-stealth-port-scanner.html http://www.lopisec.com/2011/12/learning-scapy-syn-stealth-po...
- jtakkala 12y ago
  I think you'll find that either publishing an ARP entry or filtering incoming packets in the kernel is required for handling a TCP stream over raw sockets. As the outbound TCP SYN is manually crafted and sent over a raw socket, without any corresponding state table entry on the sender's kernel, incoming TCP responses will be rejected by the kernel with a RST. I suggested to Julia that she manually publish an ARP entry for another IP which she could send and receive on. The kernel not having an interface with that IP assigned to it would ignore responses while also passing them to the raw socket. An alternative would be to use an iptables rule to drop incoming packets for the relevant flow - although that may be more difficult to manage depending on what you're doing.
lnkmails 12y ago
As someone who worked on writing protocol specs as code for simulation purposes, I can see how much fun this is. A pure python network simulator (ns2 is C++ and painfully hard to debug) would actually be nice and encourage a lot of theoreticians to get into real programming. I've spent a reasonable amount of time in the industry building distributed systems and I can say with confidence understanding low level protocols improves your thinking.
nodivbyzero 12y ago
Check this out: http://gafferongames.com/networking-for-game-programmers/sending-and-receiving-packets/ http://gafferongames.com/networking-for-game-programmers/sen... http://gafferongames.com/networking-for-game-programmers/reliability-and-flow-control/ http://gafferongames.com/networking-for-game-programmers/rel... C++, but very detailed articles.
beagle3 12y ago
While not immediately relevant, if you find this discussion interesting, have a look at sshuttle: sshuttle[0] is a pure-python one-way TCP VPN solution that is very well behaved and quite efficient. The source is highly readable as well. +1 to everything Avery Pennarun has released (including wvdial, bup) [0] https://github.com/apenwarr/sshuttle https://github.com/apenwarr/sshuttle
zzzeek 12y ago
My first thought was "it'll be slow as shit", then I clicked the article to confirm. Win!
js2 12y ago
All three volumes of TCP/IP Illustrated may be found on the Internet in pdf form, but they are well worth buying. Tangent: One of my favorite interview questions is to ask how traceroute works. The question works best when the candidate doesn't actually know. Then we can start to discuss bits and pieces of how TCP/IP works, until they can puzzle out the answer.
- fivre 12y ago
  Do you know if there's a good IPv6 equivalent? I have IPv6 Core Protocols Implementation but like the writing style TCP/IP Illustrated much more.
  - wmf 12y ago
    There is a 2nd edition.
fragmede 12y ago
Unfortunately, most people are going to read the headline, read the conclusion (that Python is too slow for TCP), and not realize that it's wrong. Hopefully someone will blog a response post that gets popular on HN proving just how wrong it is.
srean 12y ago
Too late to join the story, but I am really curious if datacenter nodes intended for heavy mapreduce use implement this layer in user space. The bottleneck for such processes is typically network I/O and I can imagine that taking control of the network in the user space might offer some modest to significant wins. For Hadoop in particular network packets needs to traverse quite a few layers before it is accessible to the application. Has anyone done this sort of a thing for mapreduce. Pointers to any writeup would be awesome. In fact TCP itself might be an overkill for mapreduce. The reduce functions used are typically associative and commutative. So as long as the keys and values are contained entirely within a datapacket, proper sequencing is not even needed. Any sequence would suffice.
- meowface 12y ago
  I believe Google has experimented with this sort of thing, as have a few other companies that sell a supposedly more efficient/faster userspace TCP stack as an enterprise product.
- wmf 12y ago
  There has been some work about using zero-copy RDMA and RoCE (which bypass the kernel as a side effect) for analytics. Hadoop in particular is so slow that the kernel is unlikely to be a bottleneck, but more optimized runtimes like Spark might benefit.
  - srean 12y ago
    I dont understand why is Hadoop so freaking slow. I am no fan of Java (to put it mildly) but Java does fairly well to keep in the 70~80% of a C++ code at the cost of 4 to 5 times more memory. My experience is that Hadoop is 4 to 6 times slower. Is it because of bad choice of internal algorithms, bad choice of internal data structures ? Bad I/O design ? Given the popularity it enjoys, and given its age, its a little frightening how much worse Hadoop is in comparison to some proprietary implementations. My hunch is that Hadoop's slowdown is in the shuffle phase, which is where faster network data transfer can help. I like abstractions that spark exposes but it still needs a lot of engineering to catchup. I have anecdotes where Spark is slower than Hadoop by quite a bit. All my experience is with Hadoop 1.x. Is Hadoop 2.x much better ?
norswap 12y ago
If you're interested in the matter, here's more background info on userland TCP stacks: http://perso.uclouvain.be/olivier.bonaventure/blog/html/2013/12/01/sandstorm.html http://perso.uclouvain.be/olivier.bonaventure/blog/html/2013...
smutticus 12y ago
I'm the author of hexcap(http://www.hexcap.org http://www.hexcap.org), an ncurses libpcap file editor and packet generator. I've also written many Scapy applications like this one(https://github.com/smutt/mcastClients https://github.com/smutt/mcastClients). I rewrote the DHCPv4 client in Scapy since the stock one is broken. Also as part of hexcap have made numerous fixes to dpkt. Needless to say, I've done a lot with Python and packets. If you're interested in writing a TCP/IP stack in Python I would recommend you use Python raw sockets, or possibly dnet[1] or pcapy[2]. The Scapy level of abstraction is too high for your needs. I agree with other posters who mention buffering in libpcap. Read the man page for pcap_dispatch to get a better idea of how buffering works in libpcap. Also try capturing packets with tcpdump with and without the '-l' switch. You'll see a big difference if your pkts/sec is low. Don't do arp spoofing. If you're writing a TCP/IP stack then you need to also code up iparp. If you don't want to do that, then use raw sockets and query the kernel's arp cache. On second thought you really need to use raw sockets if you want this to work. Using anything pcap based will still leave the kernel's TCP/IP stack involved, which is not what you want. [1] http://libdnet.sourceforge.net/pydoc/private/dnet-module.html http://libdnet.sourceforge.net/pydoc/private/dnet-module.htm... [2] http://corelabs.coresecurity.com/index.php?module=Wiki&action=view&type=tool&name=Pcapy http://corelabs.coresecurity.com/index.php?module=Wiki&actio...
Sami_Lehtinen 12y ago
There's no reason to ACK every packet. It's very common to deal with things selectively. SACK: http://packetlife.net/blog/2010/jun/17/tcp-selective-acknowledgments-sack/ http://packetlife.net/blog/2010/jun/17/tcp-selective-acknowl...
benjamincburns 12y ago
Please keep writing, Julia. The next time somebody asks me why I spend so much of my life absorbed by a screen, I'll point them at your blog and say "because discovery is exciting!"
tjz77 12y ago
What if every python virtual machine had a full TCP/IP stack? IPv6 has enough address space. Object storage takes care of disk access. Generally it might be way less efficent? What would the OS look like? It seems like a lot of OS services would disappear? You'd have a cloud of processes. Each process vm would be like a cell in a body. Maybe each process vm would load an auth module. Or not.

408 points by jvns 12y ago

It's not Python that's slow, but scapy, which is dog slow. In fact, it is so slow that it should come with big WARNINGs that it's only really meant for interactive use. Do the dissection yourself or use something built for that purpose. It's really surprising to me that lots of ppl are using scapy for things that require performance but then again if you look at the scapy website or the docs, it's not immediately apparent that their tool is not meant for this. Which I guess says a lot about the scapy developers rather than the scapy users. tl;dr Scapy is a joke, performance-wise.

philangist 12y ago

Fun read. Does anyone here know how to deal with the Python being slow at sending ACK packets problem? Or is it a built-in limitation that comes with dealing with high level languages?

crazypyro 12y ago
Implement it in C. Also its likely more a python problem than a "high level" language problem.
- fastball 12y ago
  I blame all networking issues on the GIL.
- jnazario 12y ago
  there's a userland stack written in C in honeyd. http://honeyd.org/ http://honeyd.org/ (slow to respond) and http://en.wikipedia.org/wiki/Honeyd http://en.wikipedia.org/wiki/Honeyd
  - crazypyro 12y ago
    Except that was written for an explicit purpose that wasn't speed. I fail to see how my suggestion of implementing low level networking in C and linking that to python code is considered a poor solution.
- mkonecny 12y ago
  This has nothing to do with Python or C. More likely he has a bug in his code.
  - crazypyro 12y ago
    Python being magnitudes slower than C for networking code has a lot to do with Python, actually. Other smarter people have already explained it better than I can up above.
lmm 12y ago
The answer to questions about performance is always "profile it". I mean, sure, you could probably get a crude speedup by just running it under pypy or some such, but the real answer is to profile it, see where the bottleneck is, and fix it. As the other post suggests, this is not a problem in all high-level languages; it's possible to write very high-performance code in Haskell or Ocaml, for example. But python's semantics are e.g. that every object has its own methods that can be overridden arbitrarily, which means that every single method call pretty much has to involve a hash table lookup. 99% of Python code never uses this extreme dynamism (usually you define methods on classes, and if you want an instance with a different implementation for a particular method then you make a subclass), but it's there, and a Python interpreter that didn't respect that would behave incorrectly.
- kyllo 12y ago
  So you're saying that late binding is the most significant reason why Python is slow? Is slowness just an inherent tradeoff in using a language that supports this powerful feature?
  - lmm 12y ago
    I'm not an expert but that's my understanding. Remember that it applies to some scopes as well as to object properties. In benchmarks the big divide I've seen is between languages that allow this and languages that don't. You can do sophisticated things where you compile objects assuming they won't be overridden and then back out the compile if something touches an object (the JVM does similar things where it will compile a never-overridden method as non-virtual and then detect when the class hierarchy changes), but that requires a lot of complexity that goes against the goals of CPython. Nowadays I mostly work in Scala, and anywhere where I would have used such a technique I find there's a way to do it "statically". So I'd be interested if there are good examples of what makes this a "powerful" technique, and to see if I can't replicate them "statically" with enough typeclasses etc.
  - marcosdumay 12y ago
    Not only late biding, but all the Python's powerfull reflection (not only hashes, but it's string hashes all the way down) makes it hard to write a fast compiler. Not impossible, as lmm said, just hard.
mbell 12y ago
I'd start with the question of 'why so much ACKing?'. In TCP you don't normally ACK every packet. You really only need one ACK per receive window, not one ACK per packet which is what the op's code appears to be trying to do. It also doesn't look like op's code is setting the window size anywhere so in effect she's saying to google's server 'ok got that one, send me a bunch more' and trying to do that for every packet at fast as possible is creating a runaway congestion problem. The best solution doesn't have anything to do with python, it's to implement window size (flow control) and only ack when you need to. ps - It's been several years since I've worked on a TCP stack so please correct anything I'm remembered incorrectly

mholt 12y ago

ARP spoofing... clever. This was an amusing read and really informative, too. There is definitely something to be said for explaining lower-level concepts (e.g. TCP handshakes) using the common tongue. IMO, not a bad way to begin learning. Someone could perform the same experiment now and use Wireshark to see the raw packets, then draw conclusions to what is happening. Anyone know why the Python program is so slow? I'm looking at the code and my first guess would be this part[1] but I can't explain why, overall, it would be so slow that a remote host would close the connection. [1] https://github.com/jvns/teeceepee/blob/7e93970e16fbb0c3d4bee54bb9fb019b702fae5e/teeceepee/tcp.py#L144 https://github.com/jvns/teeceepee/blob/7e93970e16fbb0c3d4bee...

delluminatus 12y ago
I would be interested in seeing if using asyncio would resolve the network issues.

cookiemonster11 12y ago

This is not a TCP stack.

prht 12y ago
Indeed
- cookiemonster11 12y ago
  It confuses me that people make such a big deal of their little 20 lines of code toy projects.
  - michaelhoffman 12y ago
    It's not a little 20 line of code toy project. It's an engaging and accessible writeup of some basic parts of TCP that happens to include some easy-to-understand code. It's pointless to people who understand how TCP works in depth, but the majority of programmers don't.
    - peterwwillis 12y ago
      It's an engaging and accessible writeup, true. Unfortunately, there are also several glaringly incorrect/misleading points in the article. The fact that it's getting upvoted is just..... strange.
  - ivraatiems 12y ago
    I don't think it's presented as a "big deal." I read it as a fun little experiment.
  - prht 12y ago
    More confusing part for me is that I knew there is TCP/IP stack for ages. TCP stack is all new to me.
  - Someone1234 12y ago
    It confuses me that instead of finding these projects uninteresting (as you claim) you took the time and effort to write a reply in a thread just to belittle and dismiss the article. I for one found it interesting/fun.

kesor 12y ago

And there is no mention of ScaPy? http://www.secdev.org/projects/scapy/ http://www.secdev.org/projects/scapy/

ayrx 12y ago
" I was much more comfortable with Python than C and I’d recently discovered the scapy networking library which made sending packets really easy." Did you even read the link?
cookiemonster11 12y ago
There is, in the second sentence of the article.
deleted 12y ago
[deleted]

kfnic 12y ago

Shouldn't the TCP handshake look like this: ---- SYN ----> <-- SYN/ACK -- ---- ACK ----> rather than having the client send two SYNs to the server?

cookiemonster11 12y ago
It should.
jvns 12y ago
absolutely! Fixed :)

beering 12y ago

I wonder if it would have a better success rate on a site other than Google's, since I've heard that Google's done extensive tuning of their TCP stack to send page data faster. Somebody's oversubscribed $3/month shared PHP hosting might not ramp up the speed as quickly.

lazyant 12y ago
yes, they also have a bigger initial TCP window than the default
ivraatiems 12y ago
I think it would still be significantly too fast. That oversubscribed PHP hosting is still sitting on a dedicated server that does nothing but serve TCP/IP all day. My hunch is that the code is very heavily optimized for most production servers, because they're all using the TCP/IP stack in the kernel. It won't be as fast as Google's, but it won't be as slow as python either.
wmf 12y ago
TCP is TCP. If you advertise a certain window size you better be prepared to receive that data.

kernelwaste 12y ago

This is not an implementation of a TCP stack.

c_plus_minus 12y ago

Ha! Nice read, perfect after lunch material with my coffee :) Good job

wslh 12y ago

And in Squeak (Smalltalk) ? take a look at: http://www.squeaksource.com/@Hl1Cdo4NwCmLqQl0/Im9cEg0J http://www.squeaksource.com/@Hl1Cdo4NwCmLqQl0/Im9cEg0J

tonyg 12y ago
I'm guessing you were linking to http://www.squeaksource.com/Net.html http://www.squeaksource.com/Net.html - Seaside uses non-persistent URLs by default, leading to problems like this...

tptacek 12y ago

The idea that Python is so slow that it's confusing TCP sounds wrong to me. I think it's more likely that your packet capture scheme is slow. It looks like you're using scapy, which I assume is in turn using libpcap... which may be buffering (in high-performance network monitoring, the packet capture interface goes out of its way to buffer). Which is something you can turn off. About 13 years ago, I wrote my own programming language expressly for the purpose of implementing network stacks, and had a complete TCP in it; I didn't have this problem. But I do have this problem all the time when I write direct network code and forget about buffering. Similarly: "Python is so slow that Google reset the network connection" seems a bit unlikely too. Google, and TCP in general, deals with slower, less reliable senders than you. :) What's the time between your SYN and their SYN+ACK?

jvns 12y ago
my favorite thing about writing blog posts is comments like this. Thank you! I didn't consider that the packet capture interface might do buffering. That might explain a lot of the problems I was having :)
- tptacek 12y ago
  This is a great project. Keep playing with it! You might find that the serverside of TCP is more useful to have in Python than the clientside (having a userland IP/TCP serverside allows you to create fake hosts out of thin air).
  - blutoot 12y ago
    Your "userland TCP/IP" comment got me excited and I started digging around. I wonder if there's some benefit in trying to port something like TCP Daytona [1,2] in a high-level language like Python. I'm curious if that will somehow advance the adoption of SDNs. [1] https://github.com/jamesbw/tcp-daytona https://github.com/jamesbw/tcp-daytona [2] http://nms.lcs.mit.edu/~kandula/data/daytona.pdf http://nms.lcs.mit.edu/~kandula/data/daytona.pdf
    - 12y ago

bhakk 12y ago

shit?

kalleboo 12y ago

It seems odd to me that Google would time out that quickly. You could never reach Google from a GPRS connection if that was the case. I'd investigate the ACKs you're sending. Are you missing packets or sending them in the wrong order? In Uni we had a networking course where we got to build a network web server software stack from the bottom up, starting with Ethernet/MAC, TCP/IP, and then on the top HTTP, all being flashed onto a small network device (IIRC it was originally a CD-ROM server). It was an extremely enlightening exercise. I recommend you go deeper instead of just using a premade Python library for TCP!

sly010 12y ago

If an 8MHz microcontroller is fast enough to implement TCP, then Python should be fast enough too. Here is my two cents on the expirement: 1. You don't really have to ack every packet, you have to order them, drop duplicates and ack the last one. 2. Google ignores the TCP flow control algorithm and sends the first few hundred packets very quickly without waiting for acks. They do this to beat the latency of the flow control algorithm. That's why you end up with so many packets on your side. You could just try anything but google, and you would probably see that you have a less insane packet storm.

Dylan16807 12y ago
Even 8MHz is massive overkill. It's almost fast enough to bit-bang ethernet.
- jacquesm 12y ago
  I'd like to see you bit-bang a 10MHz signal with an 8 MHz clock. You'd need something substantially faster (Nyquist and such).
  - TickleSteve 12y ago
    TCP != ethernet. A TCP stack can run over serial and othe link layers (PPP, SLIP, etc). PPP is absolutely doable on very small microcontrollers.
    - Sami_Lehtinen 12y ago
      Problem is that using PPP or SLIP doesn't relieve you from having to implement at least some kind of crude TCP stack. But if you use TCP offloading, then you can communicate using TCP as cheaply as doing any traditional serial communication. I've seen those crude TCP stack implementations and those are barely LAN capable. Using those over Internet wouldn't be a great idea. Often these TCP stacks are used just as serial - tcp converters. Using RWIN of 16 (max) bytes due UART buffering limits etc. Not sending ACK before whole 16 byte is cleared and so on. So with really low end devices, it's better to off load TCP to custom hardware. Many embedded devices use that approach. In such cases you'll only call to IP address and port, and when it connects, you'll have your TCP/IP connection in pure serial. TCP-modems isn't any different from other good old Hayes modems. It's just dial-up over Internet and TCP. Btw. If the system is able to run CPython, Linux, or something similar, it's already a pretty powerful system.

blutoot 12y ago

This wins the Internet today for me... "my kernel: lol wtf I never asked for this! RST! my Python program: ... :("

prht 12y ago

I don't know what TCP stack is, but in case if you are interested in implementing TCP/IP stack http://git.savannah.gnu.org/cgit/lwip.git/tree/src http://git.savannah.gnu.org/cgit/lwip.git/tree/src

malone 12y ago

I like your solution to prevent the kernel from interfering with your packets. An alternative method I've used in the past is to add an iptables rule to silently drop the incoming packets. libpcap sees the packets before they are dropped, so you'll still be able to react to them, but the kernel will ignore them (and therefore won't attempt to RST them).

lukego 12y ago

Cool :). I also learned a lot about networking by writing a TCP/IP stack in Common Lisp. http://lukego.livejournal.com/4993.html http://lukego.livejournal.com/4993.html if you are interested.

jnbiche 12y ago

This is a fun write-up. If you enjoy this kind of playing around with networking in a dynamic language, and don't want to have to worry about ARP spoofing to do these kinds of experiments, you may want to take a look a Snabb Switch. It provides userland networking in Lua, connecting to the NIC directly (only a handful of popular NICs currently supported) [0]. I've not used it yet, but I've read over the documentation and am itching for an opportunity to do so. 0. https://github.com/SnabbCo/snabbswitch https://github.com/SnabbCo/snabbswitch

voltagex_ 12y ago
I can't find the list of supported NICs in the documentation, but I found https://github.com/SnabbCo/snabbswitch/blob/master/src/lib/hardware/pci.lua#L54 https://github.com/SnabbCo/snabbswitch/blob/master/src/lib/h... which suggests that only a very small subset of Intel NICs are supported. One of those may be emulated by VirtualBox, though.

kashif 12y ago

I have actually implemented a TCP stack in python, unfortunately I can't share it publicly just yet. The authors problems are because she is not using RAW_IP_SOCKETS. Making TCP packets using pythons struct module is a breeze. I can post specific examples in code if anyone is interested. Finally you can write a proper TCP stack in python, there is no reason not to. Your user-space interpreted stack won't be as fast as a compiled kernel-space one - but it won't be feature short. PS: I guess, Google is probably sending him a SSL/TLS handshake which he isn't handling. Edit: Corrected author's gender as mentioned by kind poster.

blutoot 12y ago
I believe the author is a "she". It says at the banner of the blog.
- blutoot 12y ago
  I think this downvoting thing is out of control. What was even remotely offensive about this correction?
  - dragonwriter 12y ago
    "offensive" isn't the only reason for downvoting. "Not a substantive addition to the conversation" is one (of many) other bases. If you are going to complain about downvoting [1] why do so based on the unwarranted assumption that the downmod must be for offensiveness? [1] And you shouldn't, see the Guidelines [2] under "In Comments" [2] https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
    - vacri 12y ago
      In my opinion, if HN doesn't want puzzled people to ask why they've been downmodded, then instead of putting a line in the Guidelines saying "don't do this", they should separate the downmod into "I disagree" and "flag this as a bad comment", with the latter item the one that reduces the text weight. It's clearly broken - people have been doing this for as long as I've been here, and it's not going to change (sadly, 'it' means both the behaviour, and HN's braindead downmod mechanism).

peterwwillis 12y ago

You don't need to spoof a different MAC or IP to implement your own stack on a raw socket, Python is not too slow to handle negotiating a connection, and your interpretation of how tcp/ip works is flawed. I highly recommend you read a good book about tcp/ip and learn how the kernel works with network applications of different types. In terms of using Scapy for your packet crafting, here are some guides with examples that may help you work around your issues. (Hint: use the Scapy-defined sending and receiving routines and don't implement your own, or stop using Scapy and implement your own raw packet sockets) http://securitynik.blogspot.com/2014/05/building-your-own-tcp-3-way-handshake.html http://securitynik.blogspot.com/2014/05/building-your-own-tc... https://github.com/yomimono/client-fuzzball/blob/master/fuzz_tcp.py https://github.com/yomimono/client-fuzzball/blob/master/fuzz... https://www.sans.org/reading-room/whitepapers/detection/ip-fragment-reassembly-scapy-33969 https://www.sans.org/reading-room/whitepapers/detection/ip-f... http://www.lopisec.com/2011/12/learning-scapy-syn-stealth-port-scanner.html http://www.lopisec.com/2011/12/learning-scapy-syn-stealth-po...

jtakkala 12y ago
I think you'll find that either publishing an ARP entry or filtering incoming packets in the kernel is required for handling a TCP stream over raw sockets. As the outbound TCP SYN is manually crafted and sent over a raw socket, without any corresponding state table entry on the sender's kernel, incoming TCP responses will be rejected by the kernel with a RST. I suggested to Julia that she manually publish an ARP entry for another IP which she could send and receive on. The kernel not having an interface with that IP assigned to it would ignore responses while also passing them to the raw socket. An alternative would be to use an iptables rule to drop incoming packets for the relevant flow - although that may be more difficult to manage depending on what you're doing.

lnkmails 12y ago

As someone who worked on writing protocol specs as code for simulation purposes, I can see how much fun this is. A pure python network simulator (ns2 is C++ and painfully hard to debug) would actually be nice and encourage a lot of theoreticians to get into real programming. I've spent a reasonable amount of time in the industry building distributed systems and I can say with confidence understanding low level protocols improves your thinking.

nodivbyzero 12y ago

Check this out: http://gafferongames.com/networking-for-game-programmers/sending-and-receiving-packets/ http://gafferongames.com/networking-for-game-programmers/sen... http://gafferongames.com/networking-for-game-programmers/reliability-and-flow-control/ http://gafferongames.com/networking-for-game-programmers/rel... C++, but very detailed articles.

beagle3 12y ago

While not immediately relevant, if you find this discussion interesting, have a look at sshuttle: sshuttle[0] is a pure-python one-way TCP VPN solution that is very well behaved and quite efficient. The source is highly readable as well. +1 to everything Avery Pennarun has released (including wvdial, bup) [0] https://github.com/apenwarr/sshuttle https://github.com/apenwarr/sshuttle

zzzeek 12y ago

My first thought was "it'll be slow as shit", then I clicked the article to confirm. Win!

js2 12y ago

All three volumes of TCP/IP Illustrated may be found on the Internet in pdf form, but they are well worth buying. Tangent: One of my favorite interview questions is to ask how traceroute works. The question works best when the candidate doesn't actually know. Then we can start to discuss bits and pieces of how TCP/IP works, until they can puzzle out the answer.

fivre 12y ago
Do you know if there's a good IPv6 equivalent? I have IPv6 Core Protocols Implementation but like the writing style TCP/IP Illustrated much more.
- wmf 12y ago
  There is a 2nd edition.

fragmede 12y ago

Unfortunately, most people are going to read the headline, read the conclusion (that Python is too slow for TCP), and not realize that it's wrong. Hopefully someone will blog a response post that gets popular on HN proving just how wrong it is.

srean 12y ago

Too late to join the story, but I am really curious if datacenter nodes intended for heavy mapreduce use implement this layer in user space. The bottleneck for such processes is typically network I/O and I can imagine that taking control of the network in the user space might offer some modest to significant wins. For Hadoop in particular network packets needs to traverse quite a few layers before it is accessible to the application. Has anyone done this sort of a thing for mapreduce. Pointers to any writeup would be awesome. In fact TCP itself might be an overkill for mapreduce. The reduce functions used are typically associative and commutative. So as long as the keys and values are contained entirely within a datapacket, proper sequencing is not even needed. Any sequence would suffice.

meowface 12y ago
I believe Google has experimented with this sort of thing, as have a few other companies that sell a supposedly more efficient/faster userspace TCP stack as an enterprise product.
wmf 12y ago
There has been some work about using zero-copy RDMA and RoCE (which bypass the kernel as a side effect) for analytics. Hadoop in particular is so slow that the kernel is unlikely to be a bottleneck, but more optimized runtimes like Spark might benefit.
- srean 12y ago
  I dont understand why is Hadoop so freaking slow. I am no fan of Java (to put it mildly) but Java does fairly well to keep in the 70~80% of a C++ code at the cost of 4 to 5 times more memory. My experience is that Hadoop is 4 to 6 times slower. Is it because of bad choice of internal algorithms, bad choice of internal data structures ? Bad I/O design ? Given the popularity it enjoys, and given its age, its a little frightening how much worse Hadoop is in comparison to some proprietary implementations. My hunch is that Hadoop's slowdown is in the shuffle phase, which is where faster network data transfer can help. I like abstractions that spark exposes but it still needs a lot of engineering to catchup. I have anecdotes where Spark is slower than Hadoop by quite a bit. All my experience is with Hadoop 1.x. Is Hadoop 2.x much better ?

norswap 12y ago

If you're interested in the matter, here's more background info on userland TCP stacks: http://perso.uclouvain.be/olivier.bonaventure/blog/html/2013/12/01/sandstorm.html http://perso.uclouvain.be/olivier.bonaventure/blog/html/2013...

smutticus 12y ago

I'm the author of hexcap(http://www.hexcap.org http://www.hexcap.org), an ncurses libpcap file editor and packet generator. I've also written many Scapy applications like this one(https://github.com/smutt/mcastClients https://github.com/smutt/mcastClients). I rewrote the DHCPv4 client in Scapy since the stock one is broken. Also as part of hexcap have made numerous fixes to dpkt. Needless to say, I've done a lot with Python and packets. If you're interested in writing a TCP/IP stack in Python I would recommend you use Python raw sockets, or possibly dnet[1] or pcapy[2]. The Scapy level of abstraction is too high for your needs. I agree with other posters who mention buffering in libpcap. Read the man page for pcap_dispatch to get a better idea of how buffering works in libpcap. Also try capturing packets with tcpdump with and without the '-l' switch. You'll see a big difference if your pkts/sec is low. Don't do arp spoofing. If you're writing a TCP/IP stack then you need to also code up iparp. If you don't want to do that, then use raw sockets and query the kernel's arp cache. On second thought you really need to use raw sockets if you want this to work. Using anything pcap based will still leave the kernel's TCP/IP stack involved, which is not what you want. [1] http://libdnet.sourceforge.net/pydoc/private/dnet-module.html http://libdnet.sourceforge.net/pydoc/private/dnet-module.htm... [2] http://corelabs.coresecurity.com/index.php?module=Wiki&action=view&type=tool&name=Pcapy http://corelabs.coresecurity.com/index.php?module=Wiki&actio...

Sami_Lehtinen 12y ago

There's no reason to ACK every packet. It's very common to deal with things selectively. SACK: http://packetlife.net/blog/2010/jun/17/tcp-selective-acknowledgments-sack/ http://packetlife.net/blog/2010/jun/17/tcp-selective-acknowl...

benjamincburns 12y ago

Please keep writing, Julia. The next time somebody asks me why I spend so much of my life absorbed by a screen, I'll point them at your blog and say "because discovery is exciting!"

tjz77 12y ago

What if every python virtual machine had a full TCP/IP stack? IPv6 has enough address space. Object storage takes care of disk access. Generally it might be way less efficent? What would the OS look like? It seems like a lot of OS services would disappear? You'd have a cloud of processes. Each process vm would be like a cell in a body. Maybe each process vm would load an auth module. Or not.