4 ms·

I'd also be interested to hear this. I'm running a project that's 10gb in size and uploading the data to AWS S3 was absurdly slow. Is there any way to speed u

by fnbr 10y ago

I'd also be interested to hear this. I'm running a project that's 10gb in size and uploading the data to AWS S3 was absurdly slow. Is there any way to speed up the upload that you found? 10 GB was painful though, I can't imagine uploading terabytes.

xapata 10y ago
The fastest way to upload is to ship hard drives in an airplane.
- ams6110 10y ago
  OK -- Tannenbaum's "station wagon full of tapes" updated for the 21st century.
  - dekhn 10y ago
    Tannenbaum always forgot to include the time writing and reading tapes. Typical 10TB hard drives (which most people use for data interchange instead of tapes) only have ~100MB/sec bandwidth (~ same as 1Gbit NIC).
phillc73 10y ago
I don't work in this specific field, but did previously, during the first decade of this century, in broadcast video distribution. At the time, UDP based tools such as Aspera[1], Signiant[2] and FileCatalyst[3] were all the rage for punting large amount of data over the public Internet. [1] http://asperasoft.com/ http://asperasoft.com/ [2] http://www.signiant.com/ http://www.signiant.com/ [3] http://filecatalyst.com/ http://filecatalyst.com/
- jerven 10y ago
  Aspera, is the current winner in Bioinformatics. The European Bioinformatics Institute and US NCBI are both big users of it. Mainly for INSDC (Genbank/ENA/DDBJ) and SRA (Short Read Achive) uploads. For UniProt a smaller dataset we just use it to clone servers and data from Switzerland to the UK and US at 1GB/s over wide area internet. Very fast, and quite affordable.
  - dekhn 10y ago
    I used aspera for a while, but plain old HTTP over commodity networks works fine if you balance your transfers over many TCP connections.
collyw 10y ago
Jim Kent wrote a small program parafetch - basically an ftp client that parallelized uploads. It worked reasonably well for speeding things up maybe 10x. You can get it somewhere on the UCSC web site in his software repository, though it involves compiling the C code.
dekhn 10y ago
For GCS, the gsutil program can saturate 1GB NIC using "gsutil -m cp -R"

by fnbr 10y ago

xapata 10y ago

The fastest way to upload is to ship hard drives in an airplane.

ams6110 10y ago
OK -- Tannenbaum's "station wagon full of tapes" updated for the 21st century.
- dekhn 10y ago
  Tannenbaum always forgot to include the time writing and reading tapes. Typical 10TB hard drives (which most people use for data interchange instead of tapes) only have ~100MB/sec bandwidth (~ same as 1Gbit NIC).

phillc73 10y ago

I don't work in this specific field, but did previously, during the first decade of this century, in broadcast video distribution. At the time, UDP based tools such as Aspera[1], Signiant[2] and FileCatalyst[3] were all the rage for punting large amount of data over the public Internet. [1] http://asperasoft.com/ http://asperasoft.com/ [2] http://www.signiant.com/ http://www.signiant.com/ [3] http://filecatalyst.com/ http://filecatalyst.com/

jerven 10y ago
Aspera, is the current winner in Bioinformatics. The European Bioinformatics Institute and US NCBI are both big users of it. Mainly for INSDC (Genbank/ENA/DDBJ) and SRA (Short Read Achive) uploads. For UniProt a smaller dataset we just use it to clone servers and data from Switzerland to the UK and US at 1GB/s over wide area internet. Very fast, and quite affordable.
- dekhn 10y ago
  I used aspera for a while, but plain old HTTP over commodity networks works fine if you balance your transfers over many TCP connections.

collyw 10y ago

Jim Kent wrote a small program parafetch - basically an ftp client that parallelized uploads. It worked reasonably well for speeding things up maybe 10x. You can get it somewhere on the UCSC web site in his software repository, though it involves compiling the C code.

dekhn 10y ago

For GCS, the gsutil program can saturate 1GB NIC using "gsutil -m cp -R"