13 ms·
A Desperate Plea for a Free Software Alternative to Aspera (2018)
- TNick 7y agoIf you think about giving zmq a go, take a look at my library, here [1], [2]. Maybe we can talk about compiling a test to see how it compares? [1] https://pypi.org/project/p2p0mq/ https://pypi.org/project/p2p0mq/ [2] https://p2p0mq.readthedocs.io/en/latest/ https://p2p0mq.readthedocs.io/en/latest/
- Supermancho 7y agoZMQ is too unreliable at transferring data which is a fraction of the stated use-case. Streaming realtime data, ZMQ had problems dropping data unpredictably under load testing, so ZMQ never made it out of testing for us. The protocol is too naive.
- TNick 7y agoIn the past I see that Tsunami, UDT, or GridFTP were suggested. Any results?
- fanf2 7y agoYes, I got a similar list in response to my link on Twitter https://twitter.com/fanf/status/1210858696558944256 https://twitter.com/fanf/status/1210858696558944256 But it looks like GridFTP and Globus are basically dead https://opensciencegrid.org/technology/policy/gridftp-gsi-migration/ https://opensciencegrid.org/technology/policy/gridftp-gsi-mi... Tsunami sounds good but maybe needs some updating? I haven’t looked closely...
- moonbug 7y agoGlobus isn't dead at all globus.org, it just turned into something that solved an actual problem.
- reidacdc 7y agoGlobus is definitely not dead[1], it has an active community with strong uptake in the Materials Science world (where I work), among others. The Globus folks have a repo for many popular Linux distros, it's reasonably straightforward to get started. (I'm a user of it, not a developer or necessarily an advocate.) It's not free for "managed" (Enterprise) applications, though, which is what the OP seems to be looking for, and I'm not sure it's the best choice for high-speed. Something that I think is free is the CERN FTS [2], which can use a GridFTP back-end, so possibly you can roll your own high-speed big-data infrastructure that way. [1] https://www.globus.org https://www.globus.org [2] https://fts.web.cern.ch/ https://fts.web.cern.ch/
- CaliforniaKarl 7y agoI concur that Globus is definitely not dead! We're transferring 10s to 100s of TB per month with it. Globus is continuing to maintain GridFTP for this purpose, as part of Globus Connect Server and Globus Connect Personal, and it will happily saturate a multi-Gbps link. And personally, I'd like to see more people getting a Globus subscription. It's cost-effective when compared to tools like Aspera, and helps fund the development of Globus software and features.
- jonke 7y agoI worked with a couple of the commercial alternatives to aspera and aspera as well and I used UDT, Tsunami, GridFTP, syncthing[1] as a poor mans alternative to Aspera. For real transfer with big (> 500TB) Aspera will deliver what it says over WAN (over the Atlantic). If you have better connections and not so much data syncthing will probably work if you can have someone manually take care of all exceptions.[2] If you know your data and you can build quite a lot of stuff yourself you can get almost the same speed as Aspera with UDT. There is also a go implementation of UDTs used by kcptun[3] but I haven't tried that. 1 https://github.com/syncthing/syncthing https://github.com/syncthing/syncthing 2 I don't intend to disrespect syncthing here, but when your dealing with TB scale 24/7 things are never really up all the time, the network is unreliable, the sender host filesystem is corrupt and the receiver host filesystem is full or broken or ... 3 https://github.com/xtaci/kcptun https://github.com/xtaci/kcptun
- rjvs 7y agoHas anyone had good experiences with KCP? Syncthing removed their implementation [1] because it didn't live up to expectations. 1 https://github.com/syncthing/syncthing/issues/4737 https://github.com/syncthing/syncthing/issues/4737
- lazylizard 7y agoI think this lists some alternatives...https://www.chpc.utah.edu/documentation/data_services.php https://www.chpc.utah.edu/documentation/data_services.php I'm curious. How fast are these compared to netcat/socat/mbuffer?
- tyingq 7y agoIt mentions rclone with the parallel option of "--transfers=N". I wonder how well that does.
- dgemm 7y agoI have seen this software before. I suppose the speed boost results from using multiple parallel connections, meaning it could make better use of aggregate links and multipath networks? Is there anything else to it?
- kylek 7y agoUsed a few of their products at a previous company. That's the gist of the protocol. It's UDP-based and the software has intelligence to use bandwidth as efficiently as possible between links. We had a few servers sending to many endpoints concurrently, and I'd regularly see the systems cap out their 10gbit connections. License costs are pretty brutal and it does need decent amount CPU to make the most of it.
- big_chungus 7y agoAre you able to recommend a resource for how to use bandwidth most efficiently like this? I'm interested in learning more, and would appreciate it. Also, might using raw sockets not offer better performance and lower CPU usage?
- kwhat4 7y agoThere is quite a bit of documentation on this particular snake oil: https://patents.justia.com/patent/20090063698 https://patents.justia.com/patent/20090063698 https://en.wikipedia.org/wiki/Fast_and_Secure_Protocol https://en.wikipedia.org/wiki/Fast_and_Secure_Protocol
- dboreham 7y agoA proprietary protocol that somehow facilities throughout orders of magnitude than non proprietary alternatives. I call BS.
- sansnomme 7y agoIf they are like Riot and have their own fiber, it is not unlikely. There are tons of boring optimizations open source software also won't bother with without significant sponsorship (some could be specific to a particular file format). If you look at most network (and also video) protocols, they are accumulations of decades of improvements rather than neural network style overnight breakthroughs.
- big_chungus 7y agoDo you know of any resources where I can learn more about these boring optimizations?
- SaxonRobber 7y agoMIT open courseware has a course on Performance Engineering which details some of the types of low level optimizations that can be done. It requires a lot of measuring and problem specific solutions.
- brians 7y agoI’ve helped with that course in the past. It’s awesome. It’s completely unrelated to WAN networking. I found the IETF groups working on these problems more helpful.
- big_chungus 7y agoThanks. Any specific recommendations for someone fairly new to low-level networking? IETF docs can be a little dense :)
- 7y ago
- phonon 7y agoHave you tried https://github.com/openrq-team/OpenRQ https://github.com/openrq-team/OpenRQ or https://github.com/harmony-one/go-raptorq https://github.com/harmony-one/go-raptorq or https://github.com/cberner/raptorq https://github.com/cberner/raptorq (the last seems to have excellent decoding/encoding performance)? (Raptor codes (a type of FEC) can essentially transfer over UDP at the underlying line rate, even if packet drop is high. In other words, if it is a 1 gbit/s link, with 5% packet loss, you will be able to send over 900 mbits/s without needing any TCP features). This is also helps high latency scenarios, as you don't need test/back off how much data to send...you just send as much Raptor encoded data as you want down the pipe, and as long as you send enough recovery packets with it, you will be able to reconstruct the original flawlessly). (Also see https://par.nsf.gov/servlets/purl/10066600 https://par.nsf.gov/servlets/purl/10066600 http://www1.icsi.berkeley.edu/~pooja/HowUseRaptorQ.pdf http://www1.icsi.berkeley.edu/~pooja/HowUseRaptorQ.pdf https://a1f9fb7d-b120-4c73-98e3-5cdb4ec8a2ab.filesusr.com/ugd/4c5b11_93bc499bed254bf29057bfb5f8c44867.pdf https://a1f9fb7d-b120-4c73-98e3-5cdb4ec8a2ab.filesusr.com/ug... https://youtu.be/bYPbat-FFTo https://youtu.be/bYPbat-FFTo https://github.com/mk-fg/python-libraptorq https://github.com/mk-fg/python-libraptorq)
- Scaevolus 7y agoYou don't need 5% packet loss on a high latency 1Gbps link to trash performance. Even .1% packet loss is enough to cut throughput massively on long fat networks (LFNs). Erasure coding helps, but it demands much more CPU than simply having a very wide window for retransmits. For bulk data transfer of files, retransmitting any part of a 1GB window is trivial.
- traverseda 7y agoYou mean with raptorQ 5% loss trashes performance by more than 5%?
- phonon 7y agoNo, the overhead is fairly minimal. A 5% packet loss (which is very bad) on a gigabit link where the sender is thousands of miles away might have an effective throughput of 930 mbits per second with RaptorQ. With regular TCP you would get only a few megabits/s, as it would back down until the loss rate was nearly zero.
- deleted 7y ago[deleted]
- ovidiul 7y agoThis looks interesting as well as it's developed at CERN and uses plain tcp protocol https://github.com/fast-data-transfer/fdt https://github.com/fast-data-transfer/fdt
- YarickR2 7y agoSecond this. When we needed to deal with somewhat similar issues (quickly transfer multi-gigabyte archives over massively unreliable link) , we ended up with FDT ; it can be easily scripted , and transfers data between two hosts (without using a network of torrent peers) at that hosts' link speed. Used it to transfer builds between US and Russia, and Russia and China .
- branzo 7y ago+1 Used fdt to transfer a 6TB archive out of AWS very smoothly at full speed.
- beojan 7y agoOr [rucio](https://rucio.cern.ch/ https://rucio.cern.ch/).
- grizzles 7y agoWhy not use torrents over wireguard? From an engineering perspective I'd personally be hard pressed to come up with a more optimal solution for private big data transfer. If Aspera is as good as the parent suggests it's probably because they are operating a bittorrent-like network of geo-distributed peers. It would be very cheap to emulate that using cloud computing providers, especially if you are discarding the data after transfer.
- grizzles 7y agoYou can compress your data beforehand using zip or another tool. Eg. Tools like https://en.wikipedia.org/wiki/Compression_of_Genomic_Sequencing_Data https://en.wikipedia.org/wiki/Compression_of_Genomic_Sequenc...
- ncmncm 7y agoZip is now effectively obsolete. Good ones lately are Zstd and Lz4, depending on your needs. Lz4 doesn't compress as much, but it is very, very fast. Zstd compresses better than Zip, and unpacks many times faster. It can re-use a custom dictionary saved off from a previous compression, so you can get quick startup time for later fragments.
- grizzles 7y agoZip is just a file format, it supports multiple compression algorithms. You can put any compression algorithm in a zip implementation. I mentioned it because it's the oldest and most well known. The parent would probably be best off using one of the specialty genomics compression tools, that's why I linked to them.
- ncmncm 7y agoOK, I was really comparing to zstd and lz4 to gzip, which was the original algorithm in zip, and the one most people still use. Clearly, if you can plug better algorithms into Zip, you should, providing the Zip archive format is still good enough.
- sgt101 7y agoThere are other wan accelerators out there - like Riverbed. But, I think Aspera uses some specialized compression for genomics data, and genomics compression is not easy. My suggestion would be to talk to the telco that provides your wan. If you are using a direct internet connection my suggestion would be : don't.
- ncmncm 7y agoRiverbed always stank. Aspera never did in-flight compression when I was there. In-flight compression got even less interesting around 2010 when the long fiber links got faster than customers' (shared) storage systems. Users have good reasons to compress at rest, before starting a transfer.
- ncmncm 7y agoI want to make clear that I have no information on Riverbed's current products.
- wtracy 7y agoThe author is discussing intercontinental data transfers. Not many organizations have a WAN that can cross oceans without hitting the internet.
- quasarj 7y agoMy group is about to start using Aspera as well.. I too wish there was a good alternative I could suggest to my bosses. And I'm still skeptical of their claims... but I haven't gotten to test it over an actual bad link yet (over a good link it does nothing, which a simple test will prove).
- KaiserPro 7y agoIt only really makes sense when you have a >1gig link with a latency of 100ms or more. Or anything with a >0.1% packet loss.
- CaliforniaKarl 7y agoI don't know what your needs are, but Globus might be a good alternative. https://www.globus.org https://www.globus.org It's being run out of UChicago, and I know they have at least one large company using them. Feel free to reach out to the Globus team, or shoot me an email!
- saagarjha 7y ago> Aspera is owned by IBM - does their client spy on what else is running on my system and report it back to their headquarters as business intelligence? If asked, would they share this information with the government? Without the source code, it’s impossible to tell! I mean, with some reverse engineering, mentioned later in the post, it’s really not.
- tonyg 7y agoI wonder why something like https://github.com/htcat/htcat https://github.com/htcat/htcat doesn't fit the bill.
- cornmeal1 7y agoA Desperate Plea To Come Work For Me For Free. (Also if you could write a paper on it and put me as the co-author, that'd be great.)
- ewwhite 7y agoI've used UDT via UDR for rsync optimization and transfers over high-latency links: https://github.com/LabAdvComp/UDR https://github.com/LabAdvComp/UDR
- patelh 7y agoNice! Thanks for the info.
- mattbillenstein 7y agoI benchmarked Aspera against alternatives for getting data from on-prem storage into the cloud (google cloud storage) a year or two ago on a 10Gbps link. Aspera was faster on a single stream, but if you have a lot of files to move around (you usually do) you can just multiplex a bunch of tcp streams to get the same throughput. So like in the example on their page, if you have 20 wget's hitting their ftp and there are no other bottlenecks, the throughput will be similar to using Aspera - and you have just a greater variety of free tooling for tcp based those protocols...
- ncmncm 7y agoSending on lots of TCP streams can get you to 80% line utilization if the drop rate is good and low. That's often good enough. On some lines, with the "wrong" number of TCP connections, the rate will oscillate, and you will be lucky to get 50%. Tuning that is a black art some people enjoy.
- mattbillenstein 7y agoYeah, I was probably working in an environment with pretty low packet loss. Looking at the problem holistically - if you're going to spend money on Aspera to fix situations with high packet loss, maybe you're better off spending money on a higher quality connection and using tcp.
- ncmncm 7y agoYes, that works when latency is low. On trans-oceanic links you often have to take what you can get, drops and all, and physics has something to say about latency. There is no reason fasp couldn't be reverse-engineered and deployed for free. But it involves real engineering. Once you did, you would have something valuable, and might be inclined to milk it. Aspera has lots of lock-in now for ancillary features of the product -- monitoring, encryption in flight, encryption at rest, bandwidth use scheduling, yada yada; but in the early days, competitors (Signiant, Riverbed) had all that, but not the flow control sauce, so failed on difficult routes. "Enterprises" like not to care about the difficulty of routes, and it's amazing how many consider the license cheap.
- john_moscow 7y agoHonestly speaking, a good way to solve this is not trivial. You would need to: 1. Measure the existing performance on specific data sets. 2. Understand how many of the bottlenecks come from the network latency vs. I/O limits vs. CPU bottlenecks (if using compression). 3. See if any domain-specific compression is needed. 4. Document the typical use cases. Understand why the current solution sucks (e.g. requires redundant user actions). Write down user interaction scenarios. Design the UI to be as efficient as possible for those scenarios. This is a non-trivial amount of work that would require a lot of back-and-forth interaction and on-the-go requirement changes and I don't think it's entirely honest to ask someone to do this work for free in the name of cancer research (after all, you are not donating most of your paycheck to charities, are you?). If the existing solution by IBM sucks, how about making a Request For Proposal [0] and seeing if smaller software vendors could offer something better given that you are actually willing to pay for the work? P.S. A student/hobbyist can probably whip out some sort of a parallel TCP-like thing with a large window for free, but you would get the same performance by just cranking up the TCP window size via sysctl and using multiple HTTP threads (htcat was suggested earlier in the comments). [0] https://en.wikipedia.org/wiki/Request_for_proposal https://en.wikipedia.org/wiki/Request_for_proposal
- KaiserPro 7y agoEx Aspera user here. I worked in VFX, so I've been using aspera since before it was owned by IBM. Depending on what line speed you have, but for a gig link we made a simple protocol using parallel TCP streams. Basically, it chunked up the large file into configurable sized chunks, and assigned a chunk to each stream. Another stream passed the metadata. This has all the advantage of TCP, with less of the drawbacks of a custom UDP protocol. For transferring files from london to SF we were getting 800mbit/s over a 1 gig link.
- justinsaccount 7y agoThe 'proof' shown that aspera is 200x faster shows an ETA of 2:15 compared to 00:09. First of all, this is 15x faster, not 200. I tried downloading the file using axel -n 32, and it took 1:28, which is only 9x slower. Curious though, when it started out with all threads running it was transferring data at 29594.2KB/s and hit 50% after only a few seconds, but by the end with only a few threads left running it was only doing 5625.8KB/s. It looks like the performance varies a lot, possibly due to multiple interfaces or wan links being used.. or issues with their server. Using axel -n 64 over http I was able to fetch it in 50s, which is only 5x slower, and most of that time was spent from 98 to 100% as the last few connections finally finished. A smarter client that more aggressively re-fetched chunks being downloaded over slow connections would likely match the same performance.
- StavrosK 7y agoYeah, and one window shows speeds in MB/s and the other in Mbps (a 10x difference in units right there).
- justinsaccount 7y agoah, I bet they missed that which is how they ended up with 200x instead of under 20x
- flas9sd 7y agonot to take away from the posts plea. Apart from bit vs bytes, it should be noted the remotes are also on different machines with potentially varying network throughput
- richardji 7y agoI wonder how will aria2 stack up if you use it instead of wget? https://aria2.github.io/ https://aria2.github.io/
- catalogia 7y ago> We’d have to fly all over the world with suitcases full of harddrives to be able to collect our data! The postal system or commercial courier services seem like better options than buying plane tickets..
- ncmncm 7y agoEx Aspera Dev here. I did the encryption and early parallel work. There is a lot of good science behind fasp. An advantage it has over IETF protocols is that both ends trust one another. Another advantage, until recently, was out-of-order delivery. The protocol totally ignores drops, for flow control. Instead, it measures change in transit time. The receiver knows, the sender needs to know, but the useful lifetime of the measurement is less than the transit time. This should make an engineer think "control theory!", and did. So, the receiver reports a stream of transit time samples back to the sender, which feeds them into a predictor, which controls transmission rate. Simple, in principle, but the wide Internet is full of surprises. If you think this wouldn't be able to go a thousand times faster than TCP, you have never tried moving a file to China or India over TCP. :-) (Customers used to report 5% drop rates.) Drops and high RTT are devastating to traditional TCP throughput on high-packet-rate routes; read about "slow-start" sometime, and do the math. Problem is that for untrusted peers, drops are the only trustworthy signal of congestion. Recent improvements where routers tag packets to say "I was really, really tempted to drop this!" help some. Torrents get the out-of-order delivery and the lower sensitivity to drops, but its blocks are too big.. Others commented that opening lots of connections gets around some TCP bottlenecks, but that helps much only when the drop rate isn't too high (i.e. not to India or China).
- qorrect 7y agoAre you familiar with UDT (http://udt.sourceforge.net/ http://udt.sourceforge.net/) ? How does it compare to Aspera ? Thanks!
- ncmncm 7y agoI was not familiar with it. It looks like you could build a good-enough file transfer system on top of UDT, but probably not with the rate control protocols they provide. One problem with UDP protocols over the open internet is that you cannot trust UDP packet contents, which could come from anywhere.
- whatshisface 7y agoStep one: send a file that's cryptographically signed. Step two: the client solves the problem of "which of these duplicate and conflicting packets do I need to assemble together to make a message that is appropriately signed?"
- aamargulies 7y agoTry looking at bbcp perhaps? http://pcbunn.cithep.caltech.edu/bbcp/using_bbcp.htm http://pcbunn.cithep.caltech.edu/bbcp/using_bbcp.htm
- dnautics 7y agobbcp doesn't work so well for some reason. Where I used to work we had a senior dev that claimed that he could just repurpose bbcp for this. Then he gave up and wrote something custom in c++. Then he rewrote it in go, but used (among other things) non-multithreadsafe primitives, then when asked to make it encrypted, he used epoll with tls (which is apparently not a thing in go? I don't know). I told him he should just use DTLS or hell even a one-time-pad encrypted UDP stream with backpressure management, but he didn't listen to me. Then he rewrote it in C++ again and then went back to the go version. When I left the company, it still wasn't working. Also, this senior dev never wrote unit tests. On the other hand, maybe bbcp will work, and the senior dev just didn't know what he was doing. There was another senior dev who, seeing what was coming down the pike, left his job, and on his way out he was like, "yeah you can do it with bbcp, just you gotta tweak your tcp congestion rules on all the hops (which we could do, but is probably not an option for OP)"
- CaliforniaKarl 7y agoThe other senior dev was pretty close. We support bbcp for transfers to/from some of our environments, but we don't do the tuning. Instead, we set things up to be as accessible as possible (jumbo frames enabled, minimum # of hops from the DTN to the border), and leave it to the user to set all the bbcp options.
- branzo 7y agoDid I miss something or bbcp seems a bit abandoned to itself?
- na85 7y ago>On the other hand, maybe bbcp will work, and the senior dev just didn't know what he was doing I mean... based on your description it sure sounds like he didn't know what he was doing.
- moonbug 7y agogridftp's the tool to use, and Globus makes it easy.
- killjoywashere 7y agoHow about mpscp from Sandia? https://www.osti.gov/servlets/purl/1146079 https://www.osti.gov/servlets/purl/1146079
- randall 7y agoDat isn't designed for speed, but I think academic data is it's primary goal, so modifying it for speed seems within its charter. https://www.datprotocol.com/ https://www.datprotocol.com/
- edwintorok 7y agoWould QUIC or TCP with BBR congestion control help?
- compressedgas 7y agoTCP with PCC, even? I don't know how BBR compares with PCC. Previously https://news.ycombinator.com/item?id=8381480 https://news.ycombinator.com/item?id=8381480 (2014)
- unixhero 7y agoCan they not leverage torrents for this purpose?
- deleted 7y ago[deleted]
- krisoft 7y agoLooks like the plea has been up since a while. Does anyone know if they had any progress on the problem? Is the problem still standing?
- amelius 7y agoQuestion: if everybody used Aspera (instead of conventional transfer methods), would that negatively impact internet speeds overall?
- yoz-y 7y agoWay back when I had a problem on my network that was capping the FTP/SFTP speeds to about a fifth of my download speed. Torrents worked fine. To fix my problem I have cobbled up an extremely hacky solution which consisted of opening several listening sockets, ssh-ing to the remote machine and net-catting the file by chunks in several processes in parallel. It actually worked. The code is here https://github.com/jlegeny/blazing-speed https://github.com/jlegeny/blazing-speed more for archiving purposes than to be used by anybody for anything.
- 01CGAT 7y agoFTR, they say: "As you can see, the ascp client performs over 200x faster than FTP, and can download the whole file in 9 seconds. It’s pretty magical." But the example shows MB/s vs Mb/s. It's not over 200x times faster, but about 17 times faster.
- 01CGAT 7y agoMaybe OpenVPN + UDPspeeder could do something: https://github.com/wangyu-/UDPspeeder https://github.com/wangyu-/UDPspeeder
- gnufx 7y agoThere's a meta answer about the general approach to such problems: Ask what other research areas must have tackled it, and then find out what they do. The obvious answer in this case is HEP, which would lead to some of the tools mentioned. (There's also https://www.psc.edu/index.php/hpn-ssh https://www.psc.edu/index.php/hpn-ssh but I don't know how it compares.) It's unfortunate how many researchers don't do such research...
- deleted 7y ago[deleted]
- danbmil99 7y agoQuestion at the meta-level: what software/platforms are producing, analyzing, modifying & consuming this data? Is it possible some legacy systems that produce and consume these massive files on-site would more sensibly run in the cloud, directly & selectively accessing the data chunks they need over fast backbone connections? Also, is there any room for an rsync type approach, sending compressed deltas rather than naively sending huge files that may be redundant? Not to say disintermediating a BigCO expensive patented vendor-locked-in MLPOS (Market Leading Piece Of Shit) doesn't sound exciting -- it does. I'm just curious and a bit skeptical that it is always necessary to mass-copy all this data over and over.
- rcthompson 7y agoThe systems that produce massive DNA sequence files are machines reading the sequences of actual physical DNA molecules. The cloud can't sequence DNA, as far as I know.
- ncmncm 7y agoMost DNA sequences are only slightly different from sequences you already have. There ought to be mature differs by now that would enable you to send just differences. If not, there is a huge opportunity for somebody.
- vmilner 7y agoThere are such differs. https://en.wikipedia.org/wiki/Compression_of_Genomic_Sequencing_Data https://en.wikipedia.org/wiki/Compression_of_Genomic_Sequenc...
- ncmncm 7y agoThank you. I see that the state of the art remains very primitive, 18kx results notwithstanding.
- rcthompson 7y agoSure, you can compress the sequences by diffing against the published genome sequence, and many tools exist to do just that. But that alone maxes out at 50% compression, because the other half of the file is the quality scores that quantify the confidence level of each base call, for which there is no reference. Of course there are patterns in the quality scores, and you can compress these as well using the usual techniques for general compression, but it's going to limit how much you can compress without information loss. In addition, the sequencer assigns an unique ID to each sequence read. It's often ok to discard this information, but occasionally you need it in order to trace a specific problematic sequence through the entire analysis pipeline. So if you want lossless compression, you have to compress these as well. They should be rather compressible, since they are typically a prefix global to the file followed by a number that increments by 1 for each sequence. But for a general compressor you can't just automatically assume that, because according to the (lack of a) file format spec, each ID could be anything as long as they're unique within the file. So if you want guaranteed lossless compression of any FASTQ file, your compressor needs to be sufficiently general. Beyond that, compression of sequences by encoding only the differences from a specific reference sequence somewhat ties you down to using a specific reference, since that reference sequence is required for decompression. This is inconvenient because the reference sequence for a species is updated over time, and if you want to use the new reference, you'll need to decompress with the old reference and then re-compress with the new one. Do you want to do that for all your data, every time the reference is updated? Reference-based compression also means the compressed files are no longer self-contained, which may not be acceptable for certain use cases. None of these issues are fundamentally impossible to address, but the point is that it's not nearly as simple as it sounds, and anything fancy you do to try and compress a sequence file will generally impose some additional requirements on what the receiver needs to do in order to read the file. And many receivers are not very technically inclined researchers whose plates are already full of other things they need to be doing besides figuring out how to decompress your new unfamiliar sequence compression format,
- electrum 7y agoFacebook open sourced WDT, which seems to have the same goals: https://github.com/facebook/wdt https://github.com/facebook/wdt
- binarnosp 7y agoWhat about UDT [1]? Several open source libraries exist already. [1]: https://en.wikipedia.org/wiki/UDP-based_Data_Transfer_Protocol https://en.wikipedia.org/wiki/UDP-based_Data_Transfer_Protoc...
- deleted 7y ago[deleted]
- zxcvbn4038 7y agoI feel the author’s pain - Aspera is a clunky 90s styled Ruby on Rails app that is about the most cloud unfriendly piece of software I’ve encountered, I’d love an alternative. I honestly don’t care about about their proprietary UDP protocol, it’s nothing special, just another way to copy bits onto a wire. Dime a dozen. The true value of Aspera is they provide an integrated browser plug-in that lets the technically challenged reliably upload large files. If the transfer is interrupted or either side changes addresses it deals with it gracefully. It’s also a bridge to AWS S3. I spent some time looking for a replacement and while there are numerous download managers that facilitate people downloading large files, I couldn’t find any upload managers with the same level of integration and polish. About the closet thing I could find is Cyberduck, but it’s not integrated with the web browser, not as easy for technically challenged people to use, and there is no support (community support but seems really hit and miss). However it does make good use of Amazon’s multiple upload api and will happily fill whatever wire it’s connected to. Torrent software has largely the same pros and cons Cyberduck does.
- xxgrinlobs 7y agoTry NKN (New Kind of Network) for this task https://medium.com/nknetwork/nkn-file-transfer-high-throughput-file-transfer-using-nkn-client-21d261d87658 https://medium.com/nknetwork/nkn-file-transfer-high-throughp...
- TheNetEffect 7y agoAnother commercial alternative is M-Strean - non proprietary TCP based parallel streaming. Limited to work with CIFS / SMB / NFS and S3 compatible / OpenStack Swift compatible / Ceph / Azure storage. Commercial model is server based with no additional bandwidth charges . Data stored in a non proprietary fashion so bi-modal access to data possible. Not sure if it is working with QUIC or not as it is mentioned in blog posts. Seems heavily focused on the media and entertainment industry. https://storagemadeeasy.com/M-Stream/ https://storagemadeeasy.com/M-Stream/
- danbmil99 7y agoBtw attempted to contact you at the email you put on your page. Interested in talking about this project.
- telendram 7y agoBeyond more profound changes to the underlying transport protocol, bandwidth acceleration can be provide through a combination with fast compression. A good setup here would be to pipe data to `tar`, then `zstd`, then `ssh`/`scp`, and do the reverse on the receiver side. From an end-user standpoint, all this could be abstracted away if one could get `wget` (named in the article) to support zstd compression natively, which is possible since zstd is also a web compression standard. Such compression support is, by the way, available in `wget2`.