11 ms·
How Unreliable is UDP?
- IvyMike 12y agoThis is interesting, but in my experience, you can't use this data to usefully extrapolate anything. Slightly different hardware, topology, OSes, drivers, system load, network "weather conditions", etc, can radically change the results. My biggest UDP "surprise" was on a system using UDP but treating it as perfect, because the system designers knew that the hardware bus they were using was "100% reliable". And they were right--the hardware wasn't dropping anything. Too bad that the OSes on either end would discard UDP packets when they faced buffer pressure.
- colanderman 12y ago> Too bad that the OSes on either end would discard UDP packets when they faced buffer pressure. Which unfortunately is all they can do, in the limit. Otherwise, if applications (or the OS itself) can't keep up with the incoming packet/data rate, buffers would grow without bound. Not good for a production system.
- noselasd 12y agoAlso keep in mind this note: http://technet.microsoft.com/en-us/library/cc940021.aspx http://technet.microsoft.com/en-us/library/cc940021.aspx Basically, if you send() 2 or more UDP datagrams in quick succession, and the OS has to resolve the destination with ARP, all but the 1. packet is dropped until you get an ARP reply. (This behavior isn't entirely unique to windows btw.)
- georgemcbay 12y agoThis is really testing "how unreliable are the network routers, cables and network adapters (and drivers!) that I'm pushing traffic through". How unreliable is UDP as a protocol? Unreliable. This is really a binary state, not a percentage-measurable value. If you need reliability (in ordering or delivery) you need to layer on top of it and unless your network usage has very specific constraints (eg. low latency is far more important than strict ordering, as with most fast action games) you should almost certainly just use TCP rather than end up badly reimplementing it over UDP.
- Mithaldu 12y ago> low latency is far more important than strict ordering, as with most fast action games On the other hand, in that situation TCP is the worst choice you can make, as, especially on bad lines, like say Whistler in the canadian mountains, it's easy to get TCP into a state where it builds up a massive latency because it insists on trying to push every single packet through.
- JoeAltmaier 12y agoAgreed. TCP stands for "Transmission control protocol" which started out as file transfer. These days it is exquisitely unsuitable for most things it get used for, even fetching web pages. The delays, retries and congestion controls are set arbitrarily and rarely adjusted. In this modern world of wireless roaming and streaming media, TCP has little or nothing to offer, except that its there.
- TheLoneWolfling 12y agoI've long wished for a reliable protocol that was negotiated on a per-link basis. I mean: we have the processing power now. (So, effectively, packets aren't removed from the router's buffer until it knows the next link has the packet in buffer. Lots of implementation details to be gone over, though.) It seems a mite... silly... to resend a packet from the other side of the world just because your ISP couldn't shove the packet down your internet connection fast enough.
- lmm 12y agoWhen packets are being sent too fast the sender needs to be slowed down. Otherwise the buffers would just fill up. And in traditional TCP, the only thing that tells the sender this is dropped packets. The smart solution you're looking for is TCP ECNs - a way for the routers to say "I'm buffering it for now, but you'd better slow down". If you're running linux it's a kernel setting you can enable (they're disabled by default as some routers mishandle them).
- Zikes 12y ago"I would tell you a joke about UDP, but you probably wouldn't get it."
- temuze 12y agoWell, based on these results, you probably will.
- georgemcbay 12y agowouldn't it but sense make.
- monstermonster 12y agoA UDP packet walks into a bar. A UDP packet walks into a bar.
- BCM43 12y agoA walks UDP packet bar a into.
- danatkinson 12y agoI think this is definitely a more accurate representation of the joke. Although maybe drop one of the words. :)
- TheLoneWolfling 12y agoA walks UDP packet bar into bar. ?
- testhater 12y ago...and does your mom
- nanijoe 12y agoMight make more sense as "A TCP Packet walks into a bar" , since TCP supports retransmissions
- dxhdr 12y agoNice post. iperf is a pretty good tool for playing around with this stuff. In my limited testing with EC2 instances, packet size and send rate both have very large influences on packet loss. Usually loss will be stable up until some threshhold and then it will go nuts with 40-70% dropped, probably from some buffers somewhere filling up. I'd be curious to see more tests with those as variables.
- gojomo 12y agoFor a fuller picture: • Timestamps are necessary: UDP loss is bursty, when any component anywhere along the path (including the endpoints) is momentarily too burdened (buffers, CPU, wires) to forward every packet. • Try packets larger than the 'path MTU' between the two endpoints: any packet larger than that is fragmented somewhere en route, and then loss of any one fragment causes the full UDP packet to be lost. • Try alongside other traffic on the endpoints, and note that TCP streams can't find their achievable rates (or split the available bandwidth amongst themselves) without hitting the packet-dropping pressure that also affects UDP. So perhaps try with 1, 2, or more TCP streams trying to max wire-speed, between the same machines at the same time. Note also you can create arbitrarily-high loss rates by simply choosing to send more than the path and endpoints can handle. (Let one side send X thousand in a tight loop; on the other side only check for 1 per second for X thousand seconds. Many will be lost, depending on how much you've exceeded the available bandwidth/buffering between the send and receive.)
- xamolxix 12y ago> • try alongside other traffic on the endpoints, and note that TCP streams can't find their achievable rates (or split the available bandwidth amongst themselves) without hitting the packet-dropping pressure that also affects UDP. So perhaps try with 1, 2, or more TCP streams doing a max-speed copy at the same time. I agree. Having (or not) other traffic on the network will impact UDP throughput dramatically. As one who does VoIP QoS will soon find out.
- julie1 12y agoAvoid timestamps: computer are not perfect, their clocks tends to drifts in non predictable ways (even with NTP). I suspect seen quite real bugs due to clock drifting (and linux non monotonic timer (which impacted python) for a while (but problem is solved)) resulting in packet drops because networks stack don't like to see packets arriving in the future when I was using pyzmq. http://lists.zeromq.org/pipermail/zeromq-dev/2013-October/022996.html http://lists.zeromq.org/pipermail/zeromq-dev/2013-October/02... My intuition is we are close to relativistic problems. (Can't prove it, I have to code a kikoo lol form for tomorrow).
- mey 12y agoI've seen UDP fail drastically (dropping north of 50% of packet loss) on controlled networks. Once you hit a certain congestion on a network or a network buffer fills up you suddenly things go a bit sideways. I'm not sure if switches and OS's consider UDP lower priority then TCP or simply that TCP has mechanisms to recover from packet storms, but TCP continues to "function" where UDP starts hitting the floor en masse. Conversely I've pondered using UDP packets as a "canary in the coal mine" for networks to monitor it's health.
- colanderman 12y agoThis is how congestion works: queues are not infinitely deep, so the only way to deal with congestion is to (at some point) start dropping packets. TCP (that is, most TCP flow control algorithms) specifically uses packet loss as an indicator of network congestion / how much bandwidth is available, and will back off and retry dropped packets at a lower rate; hence why TCP continues to function (albeit at degraded performance). UDP has no such mechanism, hence you see firsthand the dropped packets. A well-written application should back off in such a scenario so as not to flood the network, but of course many won't. (Of course TCP may very well be prioritized over UDP! It's just not necessary to explain your observations.)
- mey 12y agoAlso priority may be a function of the network configuration (QoS etc) not biased against UDP by default.
- tptacek 12y agoIt's often the opposite. In order to prevent the Internet from melting, TCP does an elaborate dance with all the TCP speakers to back off and avoid congesting the network. UDP does no such dance, and so can monopolize traffic at the expense of TCP. Obviously this presumes that the UDP protocol in question has some mechanism for handling lost packets. But, for instance, if you're doing lossy video or forward error correction, end users do not deal directly with lost packets.
- verbatim 12y ago
- keithwinstein 12y agoUDP is just about as reliable or unreliable as IP. It's a shim on top of IP to let unprivileged users send IP datagrams that can be multiplexed back to the right user. (Hence the name: "user" datagram protocol.) Lots of people talk about a "TCP or UDP" design decision, but that's usually not the relevant question for an application developer. Most UDP applications wouldn't just blat their data directly into UDP datagrams, just like almost nobody blats their data directly into IP datagrams. Typical applications generally want some sort of transport protocol on top of the datagrams (whether UDP-in-IP or just plain IP) that cares about reliability and often about fairness. That could be TCP-over-IP, but some (like MixApp, which wanted TCP but not the kernel's default implementation) use TCP-over-UDP-over-IP, BitTorrent uses LEDBAT-over-UDP, Mosh uses SSP-over-UDP, Chrome uses QUIC-over-UDP and falls back to SPDY-over-TCP and HTTP-over-TCP, etc. The idea that using UDP somewhere in the stack means "designing and implementing a reliable transport protocol from scratch" is silly. It doesn't mean that any more than using IP does. The question of SOCK_STREAM vs. SOCK_DGRAM is more typically, "Does the application want to use the one transport protocol that the kernel implements, namely a particular flavor of TCP, or does it want to use some other transport protocol implemented in a userspace library?"
- huhtenberg 12y ago> UDP is just about as reliable or unreliable as IP That's in theory. In practice ISPs routinely classify, prioritize and shape application traffic (based on either TCP/UDP ports or results of deep packet inspection), so running SSL over IP will yield different IP loss profile than when running SSL over UDP.
- antirez 12y agoFun how this is more or less the contrary of what I think, so I'll add my point of view as a counter argument. IMHO if you need a transport layer, you are 99.9% of cases well served by TCP and should not consider UDP at all if not as a last resort to implement your transport protocol. Instead UDP is the way to go when you don't need a transport layer at all, and you can do with trivial ack / resend strategy (DNS is the obvious example), or just best-effort transfers where packet loss is not an issue at all (think a thermometer sending readings every second). In those cases the TCP three way handshake would be an overkill, and the benefits of TCP are minimal because of the use case.
- Animats 12y agoThe writer is testing over links between data centers. Those tend to have good bandwidth and no "deep packet" examination and munging. Try testing over consumer-grade links, and see what DOCSIS and Comcast are doing to your packets.
- balls187 12y agoInstead of calling UDP an "unreliable protocol", we called it "best effort."
- nanijoe 12y agoThe question may have been better phrased as "How unreliable is the internet" . TCP and UDP packets will be dropped at just about the same rate across any network, the only difference is that TCP keeps track, and re-sends any packets that did not arrive. Each protocol has its own place (and uses)...
- JoeAltmaier 12y agoAlmost. The Internet does that. But home routers are notoriously buggy. UDP often gets bad treatment there. E.g. your UDP stream may have large gaps when one of the kids upstairs is uploading to facebook - their TCP traffic gets priority over UDP which is often just dropped. This is because you can generate traffic much faster than the router can dump it to your ISP, and the little buffers fill up. Also there was an ATT home router that dropped every other UDP packet! Really! Its a matter of, if you don't test it, it doesn't work. And home appliances get tested on delivering web pages etc, not UDP streaming.
- deleted 12y ago[deleted]
- film42 12y agoI thought UDP is to TCP as C++ is to Java with regards to memory management. Meaning, TCP will do whatever it can to ensure packet integrity, while UDP pretty much leaves packet integrity up to you. Is that correct? If so, then regardless of UDP actually appearing somewhat reliable, the spec makes no guarantees that it will be consistently reliable.
- JoeAltmaier 12y agoThere's a triplet that loosely describes protocols: { Reliable, Stream, Connection} Each can be true or false. So TCP is { True, True/False, True } and UDP is { False, False, False }. (TCP can be stream or datagram) There are (were) other protocols possible but these are about all that's supported/tested these days. Also there's fragmentation/reassembly issues with UDP. If you send a 30K UDP datagram packet, it gets sent as many Ethernet-sized chunks. The receiver is supposed to put them back together. If one is lost, the entire thing is lost. And to boot, Linux didn't do UDP reassembly at all until recently (last year?)
- colanderman 12y ago(TCP can be stream or datagram) Not at all! There's no way to ask for delivery of a datagram of unknown size (which is pretty much the definition of a datagram-oriented protocol), as in UDP, RDP, SCTP, and other datagram-oriented protocols. (No, not even the PSH bit guarantees this!) This has implications when implementing application-layer protocols. In stream-oriented protocols like HTTP, which intersperse headers with payload, it is impossible to read the header from the OS's socket without also (possibly) reading part of the payload (unless you're reading a single octet at a time). Your application is thus forced to implement buffering on top of what the OS provides (if the OS does not itself provide a general pushback mechanism). With an (ordered) datagram-oriented protocol, this is not a problem, as you can ask the OS to return a single datagram corresponding exactly to the header, and process it before retrieving any of the payload. And to boot, Linux didn't do UDP reassembly at all until recently (last year?) Wow, that surprises me. Good to know.
- fl0wenol 12y agoLinux has always supported UDP reassembly. Quoting a manpage: "By default, Linux UDP does path MTU (Maximum Transmission Unit) discovery. This means the kernel will keep track of the MTU to a specific target IP address and return EMSGSIZE when a UDP packet write exceeds it. When this happens, the application should decrease the packet size. Path MTU discovery can be also turned off using the IP_MTU_DISCOVER socket option or the /proc/sys/net/ipv4/ip_no_pmtu_disc file; see ip(7) for details. When turned off, UDP will fragment outgoing UDP packets that exceed the interface MTU. However, disabling it is not recommended for performance and reliability reasons." And they ain't kidding, because there are plenty of layer 3 devices out there who play fast and loose with fragments of UDP packets; it used to be you could assume they'd drop anything that doesn't have a header, so by default UDP on linux doesn't encourage it. It could be that distributions have been bundling sysctl.conf files with ip_no_pmtu_disc set to true, being that most modern layer 3 devices no longer mistreat UDP so badly; this may be what you're experiencing in the last year.
- alex_duf 12y agoThis is relevant for Server to server communication : with a fiber link or a copper one. I'm not sure it's that reliable for a end user on a wireless network. Still, it's good to know.
- jeffreyrogers 12y agoThe reliability of UDP is mainly going to depend on the end nodes you're trying to connect. The backbone of the internet is very reliable and if you're connecting over ethernet or through some other wired connection you'll see very few dropped packets and thus UDP will be very reliable. If you're connecting with wireless, expect to see some more dropped packets, and if you're connecting with bluetooth expect to see even more (because of its low power bluetooth is more likely to drop packets than either a wifi or wired connection).
- richardwhiuk 12y agoWireless (802.11) has built in re-transmit on a per link basis, so in practice the drop rate isn't as big as you'd expect - the main problem is packet jitter, not loss.
- jeffreyrogers 12y agoAh interesting, I wasn't aware of that. I think the Bluetooth part is still valid however, as is my point about the loss rate being determined largely by how the end hosts are connected to the network core.
- IgorPartola 12y agoSo UDP is great. I love the simplicity of it, and the fact that it adds very little overhead while being sufficiently low level means you can do all the fun networking things yourself. What would be great is a "third half" of this equation: a popular protocol a la SCTP. Imagine not having to re-invent message size headers for every damn protocol: the size of your packet is embedded in the datagram just like UDP, but the transmission is reliable just like TCP. HTTP would suddenly get so much better!
- TD-Linux 12y agoWebRTC actually uses SCTP, but the problem is that SCTP gets filtered out a lot by firewalls and NAT that don't know how to handle it. The solution is to stuff SCTP inside UDP, which works OK because UDP is pretty minimal itself.
- IgorPartola 12y agoDid not know that WebRTC used, it, but did know about the NAT and firewall problem. That's the part I find unfortunate: a reliable general purpose datagram-based protocol is a shoe-in for so many application protocols, yet vendors tend to do bad things like this for no really good reason.
- e12e 12y agoSadly, not just SCTP. Anything not-tcp, not-udp, not-icmp (are there others in common use?) tends to get nuked by firewalls. Great fun if you try to run IPSEC with GRE for example. Really sad to have a protocol number in the IP header, only to have firewalls dropping anything but a few whitelisted ones... :-/ On another note... does anyone know if RDP[1] has seen any use lately? I'm guessing it's pretty redundant vis-a-vis TCP in practice? [1] http://tools.ietf.org/html/rfc908 http://tools.ietf.org/html/rfc908
- colanderman 12y agoDon't forget that SCTP is also multihomed, providing the extra redundancy (though not extra performance) of MPTCP as well.
- 12y ago
- kokey 12y agoI've been quite happy with UDP lately. I set up a prototype log analysis system and just to get testing quickly I just sent the CSV log data as is over UDP between servers. It works so well that I still haven't bothered to swap it out with something else.
- runeks 12y agoI'll the share the comment here that I posted on the article page. OP, you should have done latency tests as well. Whether packets arriving out-of-order is unacceptable to an application or not depends on the time distribution of packet delays. If, for example, 99.9% of packets arrive in-order within, say, 10 ms, then that is perfectly acceptable to use for video streaming, for example. Even if that means a packet is 1000 packets late, this just requires a bigger buffer on the receiving end. As long as it's below a certain duration, out-of-order packets aren't necessarily a problem even for live content. The grouping of 5 packets together and thus checking whether a packet arrive out-of-order more than six places seems arbitrary. If, for example, a packet arrives out-of-order 50 packets late, but it's actually only 1 ms late, then it's not a problem. If it comes along 1 second too late, then it is a problem.
- latch 12y agoYes, you're right. That was obvious once I got the data. I still thought what I had was worth writing about :) As you say, that grouping is arbitrary and, it was more likely to include packets that shouldn't have been included (multiple seconds later) than those that did. Though, that's more todo with the fact that I sent so few packets (5-10) in each burst.
- drawkbox 12y agoFor the best of both before SCTP exists, if ever, RUDP (Reliable UDP) [1] where needed is what most game developers/engines use on the networking layer such as RakNet [2] (recently bought and open sourced by Oculus), enet [3], and the many based on those. RUDP or similar systems allow network messaging to ack when needed and unreliable by default which is fine for positional updates with some missing data using interpolation and extrapolation to simulate missing data. Global game states and determining the winner/ending the level might need a reliable call. With UDP and only some reliable calls you drastically improve real-time performance with less queueing. TCP is ok for turn-based or near-real-time but high action real-time games almost all use UDP with a RUDP twist. Using a mix can also be harmful [4] so the best option is RUDP where needed, default to UDP. UDP is great because it is a broadcast, almost like TV/radio in that you can show the data you receive and smooth the rest. Although, TV/radio needs every data frame to prevent static/lag but imagine the broadcaster having to error check all connections, it would quickly saturate. UDP allows saturation later. Games can predict movement and smooth out missing data for most game actions, you only need a few points for a bezier to be smooth or you might have variables for speed/direction/forces that you can predict with. Pretty good reliability and lack of ordering are not really an issue, if out of order (timestamp older than last) discard and use next or predict until the next valid message (too much of this leads to lag but normal UDP operations it is enough and actually smoother). [1] https://tools.ietf.org/html/draft-ietf-sigtran-reliable-udp-00 https://tools.ietf.org/html/draft-ietf-sigtran-reliable-udp-... [2] https://github.com/OculusVR/RakNet https://github.com/OculusVR/RakNet [3] http://enet.bespin.org/ http://enet.bespin.org/ [4] http://www.isoc.org/INET97/proceedings/F3/F3_1.HTM http://www.isoc.org/INET97/proceedings/F3/F3_1.HTM
- comex 12y agoTwo other reasons to use UDP for low-latency use cases, even if you cannot handle actual missed packets and thus are fully reliable: - If your bandwidth use is small, you can just spam multiple copies of each packet to decrease the chance that a laggy retransmission will be needed. If you're sending packets at a constant high rate, you can instead include copies of the last N messages in each packet, rather than just the new data. - You can send packets directly between two NATted hosts using STUN, rather than having to rely on UPnP or manual port forwarding. Pretty obvious, but I only see one other mention of this fact in the thread, and it vastly increases the likelihood of being able to make a direct connection between two random consumer devices.
- sigzero 12y agoDoesn't the "U" in UDP stand for unreliable? Joking!
- willejs 12y agoThe title should read: How Unreliable is the network? The answer is: Very.
- bradleyland 12y agoIf you want to know how reliable UDP is, ask a VoIP engineer. The audio component of VoIP calls is called RTP (also referred to as media). Media is passed using UDP, because there's no point in retransmitting voice packets. The time sensitivity of voice communication makes it a frivolous act. By the time the retransmitted packets arrive, the audio stream has already been reassembled and played back to the end user with defects. I used to run a pretty sizable VoIP network over a DS0-based Carrier Ethernet network. When you have tight control over the network, UDP is extremely reliable. We would go long periods of time with 0% packet loss (as reported by AdTran VQM). When we did see loss, it was normally because of a problem we were already aware of. That's the beauty of DS0. When a DS0-based circuit drops frames, alarm bells sound.
- ArkyBeagle 12y agoOne thing to consider - how quickly do you need to know whether the node on the other end has gone off to see the wizard? With TCP, it might be a while - you may not get notification that he's gone until he comes back. So you end up with keepalives, and... might as well have used UDP if your media can stand it.
- zwieback 12y agoUpon customer request we implemented a UDP-based application protocol on a wireless network that had lots of low coverage areas. We thought they were crazy but it turned out that the overhead of keeping a TCP connection in an area of poor coverage can be overwhelming. The app retried a few times (and of course 802.11 retries on it's own), when responses didn't arrive but in the end the user was the real retry mechanism. To our surprise (not the customer's) everyone was happy in the end. Packet ordering wasn't an issue in our app, though, so I'm not sure what my point is other than sometimes UDP is reliable enough.
- Rapzid 12y agoUDP, the protocol, is entirely unreliable. The article is testing the reliability of the internet to deliver datagrams.
- jhallenworld 12y agoI'm wonder which vendor's routers are re-ordering packets. I know this was a big no-no in the core routers I worked on because of the bad effect it has on TCP. It was one of the things that customers checked in their product evaluations.
- chetanahuja 12y agoVery interesting analysis. By the virtue of the crazy business we've decided to be in (creating a mobile optimized, reliable protocol over UDP), we've had some experience with this... over mobile networks. I can confirm that sending packets at a slow and steady rate, with proper packet pacing (this is all being done by our user level protocol stack using UDP as the underlying datagram sending transport), we commonly see 0% packet loss. Of course, some of the actual packet loss at the media level is often being masked by the media protocol itself. That's why you'd often see cases where a packet meanders it's way to the destination after 4-5x the normally observed ping latency. This sort of stuff plays havoc with TCP congestion control algorithms (because by then, TCP send side has already decided that the packet is lost and it can only be due to congestion... so it backs the heck off). A lot of our win comes because of doing these things more in tune with how mobile networks actually behave.
- dspillett 12y agoThe "unreliable" in UDP doesn't mean "expect this to fail", it just means "things are not guaranteed to arrive, and if they do they might not arrive in the right order". So "make sure you can handle (where "handle" could simply be "safely ignore") failed/unordered transmission" rather than "expect failed/unordered transmission". If you can't cope with or ignore missed/unordered reception, wrap your comms in a stream based protocol (TCP, one of the other existing ones, or some contraption of your own) that manages reordering and retransmission as needed. Maybe there is a better word to cover that then "unreliable" which sounds quite definite, but I can't think of one off the top of my head.
- ay 12y agoArguably "expect this to fail X% of the time" is a more robust approach - because in-order delivery or zero-loss situations are then simply best cases that "just work". But you can not make work what you can't test. So anyone embarking on making their own transport protocol should test how their app works with various amounts of packet loss (1%, 5% and 20% are three reasonable marks to try which may happen in real world - especially in the wireless links case), as well as jitter, and variable delay. To do such testing, it's handy to use a Linux module called "netem" which allows to simulate the delay, jitter and loss: http://www.linuxfoundation.org/collaborate/workgroups/networking/netem https://calomel.org/network_loss_emulation.html
- dspillett 12y agoYou also need vary the pattern of packet loss as well as the overall frequency: You could lose 20% of your packets over a short period in roughly even distribution (perhaps due to "constant" background noise), or you could lose a sequence of packets amounting to that much (due to a sudden burst of interference on a particular hop of a route).
- decisiveness 12y agoThe whole point of TCP's design is to make delivery reliable, and in order on stressed/lossy networks. These test reveal exactly what you'd expect, as they are run between servers under prime networking conditions.
- froztbyte 12y ago"How unreliable is UDP? Let's test with these 5 network paths that I have no control over, and have no idea what their design constraints are...that should be totally authoritive, right?" Pretty bad test conditions on this, tbh.
- martincmartin 12y agoHave two machines (A & B) send UDP packets to a third machine (C). Give them all the same networking hardware, e.g. 1 Gb/sec NICs. Have A & B send at full speed, so a total of 2 Gb headed to C every second, which can only receive 1 Gb. You'll get a lot of dropped packets.
- Tloewald 12y agoIt's interesting that a lot of the arguments against UDP are because networks deprioritize the packets or that routers don't handle them. If you know a message is going to be lost if you don't handle it, perhaps it should receive better or at least equal priority. Since TCP is chattier, anything that encourages people to use a lighter weight protocol would then benefit everyone, since there would be more advantages to using the less resource-intensive option. It's an interesting conundrum.
- esbranson 12y agoFinally, a informational discussion on HN..