6 ms·
Ex Aspera Dev here. I did the encryption and early parallel work. There is a lot of good science behind fasp. An advantage it has over IETF protocols is that b
by ncmncm 7y ago
Ex Aspera Dev here. I did the encryption and early parallel work.
There is a lot of good science behind fasp. An advantage it has over IETF protocols is that both ends trust one another. Another advantage, until recently, was out-of-order delivery.
The protocol totally ignores drops, for flow control. Instead, it measures change in transit time. The receiver knows, the sender needs to know, but the useful lifetime of the measurement is less than the transit time. This should make an engineer think "control theory!", and did. So, the receiver reports a stream of transit time samples back to the sender, which feeds them into a predictor, which controls transmission rate. Simple, in principle, but the wide Internet is full of surprises.
If you think this wouldn't be able to go a thousand times faster than TCP, you have never tried moving a file to China or India over TCP. :-) (Customers used to report 5% drop rates.) Drops and high RTT are devastating to traditional TCP throughput on high-packet-rate routes; read about "slow-start" sometime, and do the math. Problem is that for untrusted peers, drops are the only trustworthy signal of congestion. Recent improvements where routers tag packets to say "I was really, really tempted to drop this!" help some.
Torrents get the out-of-order delivery and the lower sensitivity to drops, but its blocks are too big.. Others commented that opening lots of connections gets around some TCP bottlenecks, but that helps much only when the drop rate isn't too high (i.e. not to India or China).
- qorrect 7y agoAre you familiar with UDT (http://udt.sourceforge.net/ http://udt.sourceforge.net/) ? How does it compare to Aspera ? Thanks!
- ncmncm 7y agoI was not familiar with it. It looks like you could build a good-enough file transfer system on top of UDT, but probably not with the rate control protocols they provide. One problem with UDP protocols over the open internet is that you cannot trust UDP packet contents, which could come from anywhere.
- whatshisface 7y agoStep one: send a file that's cryptographically signed. Step two: the client solves the problem of "which of these duplicate and conflicting packets do I need to assemble together to make a message that is appropriately signed?"
- ncmncm 7y agoI solved it by cryptographically signing each packet. The signature was checked before decrypting the packet. Some customers (e.g. DOD) sent in clear, unsigned. They trusted their networks.
- light_hue_1 7y ago> Some customers (e.g. DOD) sent in clear, unsigned. They trusted their networks. Wow. I had no idea. This may be one of the dumbest things I've ever read. Encryption is free.
- ncmncm 7y agoDidn't use to be. VIA/centaur was the first to add AES instructions, later picked up by Intel and AMD. Its not free, but not expensive. A brilliant woman from, where, Slovenia? figured a way to use SSE instructions to encrypt just as fast as the built-in instructions, I think 2 cycles per byte. I guess DJB's ciphers run much the same way, nowadays.
- exikyut 7y agoIf you ever track down a citation for that I'd love to know more.
- ncmncm 7y agoMight be Emilia Kasper, now apparently at Google. Probably it was 2 cycles per bit, not byte. There are pretty good references starting from https://crypto.stanford.edu/vpaes/ https://crypto.stanford.edu/vpaes/ .
- big_chungus 7y ago> The protocol totally ignores drops, for flow control. Instead, it measures change in transit time. The receiver knows, the sender needs to know, but the useful lifetime of the measurement is less than the transit time. This should make an engineer think "control theory!" (and did). So, the receiver reports a stream of transit time samples back to the sender, which feeds them into a predictor, which controls transmission rate. Simple, in principle, but the wide Internet is full of surprises. Would you be willing to talk more about this, for someone not as intimately-familiar with the details?
- ncmncm 7y agoI've reported most of what I know. I was involved in the data layer. Flow control was done by the PhDs. But, imagine the route as a series of packet queues. Packet is sent, arrives at the first router, spends time in the queue, gets sent, put on the next queue, waits, goes, until it's delivered. If the total time taken, compared to the last packet's, is more, that means the queues are growing. If the total time is less, the queues are shrinking. You adjust your rate to stabilize them. Other flows are being added and removed all the time, so the optimal rate varies in steps over the course of the session. This involves control theory, Kalman filters, PID, the whole business.
- exikyut 7y agoAs a general question, I'm curious how a tactile learner might be able to get a decent grasp of pretty much everything you just described, in a practical setting, without breaking the bank. :) How good is GNSS at simulating latency, buffer saturation, etc on a simulated network and how much complexity would I need to contemplate factoring in to build an accurate model of the average organically evolved, ad-hoc, multi-WAN-spanning sprawling corporate network? I can't think of a more concrete, instructive starting point for someone interested in figuring out an alternative/reimplementation.
- ncmncm 7y agoIt really is a matter of learning the hard way the phenomena that show up on real networks in the wild. Aspera spent a decade incorporating responses to all these lessons into their code. You can get 80% there quickly, 90% there in two years, but the rest is just hard. A versatile simulation environment feeding the algorithm with synthetic time events according to a configured schedule is probably the best way to ramp up quickly, but I don't know anything about GNSS. One tricky phenomenon is bimodal and even N-modal delay samples from traffic split across different routes. Around 2014, Aspera replaced most of the code wrapped around the algorithms to become much more nimble about applying them in more circumstances.
- bogomipz 7y ago>"Another advantage, until recently, was out-of-order delivery." Can you elaborate on this? Which recent development are you referring to? Thanks.
- ncmncm 7y agoDoesn't TCP have an ability to re-send dropped packets, now, without sending everything after them? I thought they called it selective re-send, or selective ack, or something. Maybe the receiver could send an ACK with a bitmap of which packets it got, and the sender would fill in the holes?
- zlynx 7y agoI believe it's Selective ACK aka SACK. It allows TCP to ACK ranges of packets so the sender can see that it only needs to resend an earlier block. I've seen rather good speeds over some bad links from my bittorrent server using large window sizes (up to 9), BBR flow control, ECN fully enabled, and SACK.
- bogomipz 7y agoYes it has had selective acknowledgments(SACK) for quite a long time(RFC 2018.) I thought the OP was referring to something perhaps more recent since I believe SACK might predate Aspera.
- ncmncm 7y agoMost of our customers didn't have SACK-capable network stacks. It used to take a very long time for IETF specs to find their way into deployed systems.
- grizzles 7y agoThis is something bittorrent could get better at. It would be cool to update the bittorent spec to use eg. a resnet to predict endpoint and path saturation from moment to moment. It's kind of amazing really that the near future will probably be about removing massive amounts of code & design from all sorts of incredibly carefully engineered systems like the linux kernel in favor of a pile of linear algebra that can just figure things out. resistance is futile.
- MrEldritch 7y agoIf you think the best answer to predicting things is to just throw a neural net at it, you should probably learn more control theory. :) (Less flippantly - neural nets are mainly for the case where you can't effectively do manual feature engineering because your inputs are too varied and complex, you don't particularly care about the mathematical properties or guarantees of the solution, and where you don't even really know what the solution should look like [so are forced to randomly initialize a huge nonlinear system and optimize it until useful behaviors pop out]. Something like predicting path saturation would be far better suited to traditional approaches.)
- grizzles 7y agoI was thinking in terms of relative human effort. Adding a neural network into an open source project like bittorrent is going to be very fast to integrate and will probably offer performance that meets the satisficing criteria of being instantly better than the currently offered level of performance. Edit: By the way, I accepted your initial opinion about traditional approaches being better at face value but that was like ~45 minutes ago. After thinking about it a bit more I realized that there is alot of interesting network data that could be fed into such a NN, moving it (ding!) back into the nonlinear/complex system column. Suddenly my approach is looking competitive to also being the optimal one (excluding hybrids).
- tonyarkles 7y ago> I realized that there is alot of interesting network data that could be fed into such a NN Care to elaborate?
- bsder 7y agoWeren't a lot of these problems that whole reason for SCTP (Stream Control Transport Protocol) (https://en.wikipedia.org/wiki/SCTP https://en.wikipedia.org/wiki/SCTP)?
- zxcmx 7y agoI'm curious how much the observed speed of Aspera on the Internet might rely on "nicer" protocols (TCP) backing off as Aspera causes congestion^H^H maximises link utilisation. Do you think the protocol might cause problems if it were more widely used?
- ncmncm 7y agoExcellent question. Interestingly (at least, I find it interesting) fasp can happily use up all the bandwidth the various TCP streams aren't using without affecting TCP rates at all. It also backs off and shares bandwidth with other instances of itself or other well-behaved protocols. With some coordination between them, multiple senders can share the available bandwidth in any chosen proportion. So, you can set up where A always gets 70% when it is sending at all, and B, C, and D share whatever is left when they have anything to send, but all without ever slowing whatever TCP traffic is running. An administrator can (literally) drag up and down the rates of ongoing transfers according to organizational priorities. That sort of featurism is part of why Aspera can command crazy prices. It sounds like people are getting good results with fdt. It leads me to wonder if fdt has any of this good-network-citizen capability. A lot of people using fdt and making the net worse for everybody else would be an unfortunate outcome. Aspera was pretty careful about it. (I keep trying to write ftd, but that is the florist service.)
- exikyut 7y agoDo you have any anecdotal experience with how this sort of approach affects very short, bursty stop-and-start transmissions (notwithstanding the fact that you obviously can't go faster than link latency)?
- eps 7y agoRate control sounds nearly identical to TCP Vegas then, no? But out of order delivery and (I'm guessing) out of order retransmits appear to be unique to fasp. What did you mean about these being unique "until recently"? [edit - nevermind, saw below you were referring to sack] PS. I remember looking at fasp ~10 years ago and it looked like a fantastic tech. Someone's taking things that many people only talked about and putting them into a product that really worked well. Cutting edge stuff. A job like - a dream for many :)
- fanf2 7y agoAnd more recently Google's BBR is a newer delay-controlled congestion control algorithm - http://www.potaroo.net/ispcol/2017-05/bbr.html http://www.potaroo.net/ispcol/2017-05/bbr.html So it'll be interesting to see if BBR+QUIC can be made to do what FASP does.
- ncmncm 7y agoNo, quite different from Vegas, if the graphs are to be believed. What I would see on real trans-oceanic links was a sub-second ramp up to a flat, stable rate, and stepwise rate changes during the session as other flows started and stopped, with no visible oscillation. (I am looking at https://blog.apnic.net/2017/05/09/bbr-new-kid-tcp-block/ https://blog.apnic.net/2017/05/09/bbr-new-kid-tcp-block/ .) Fasp does not rely on RTT ACK timing, which is very noisy, but only outgoing delays. The routers are applying their own algorithms to simulate sane queue timing, but they are doing all kinds of crazy shit under the hood to get better link performance. The queues, IOW, are a fiction provided to the endpoints to simplify their job and generate understandable, thus tunable full-route dynamics. But the ACKs get completely different treatment, because traditionally nothing cared about ACK timing. So, "onset of queuing", e.g., is pretty meaningless in practice, and fasp makes no attempt to detect it.