4 ms·

Do you chunk the upload so if I lose internet for a second, it can just upload the chunks that failed? This is an issue I have with one of my employees. They o

Do you chunk the upload so if I lose internet for a second, it can just upload the chunks that failed? This is an issue I have with one of my employees. They only get consumer Comcast, and the connection is crap. I ended up writing a bash script that will chunk and scp the files to a central server, and if it fails a chunk, it leaves it in the directory for her to manually re-run. That said, I pay $20/mo for the file server (which i would no matter what) and it has 3TB of outbound bandwidth and unlimited inbound, and is less than 5% of your cost.

microcolonel 8y ago
> I ended up writing a bash script that will chunk and scp the files to a central server, and if it fails a chunk, it leaves it in the directory for her to manually re-run. Use Syncthing, it traverses NATs, it is trivial to set up, it is seemingly very secure by its nature, and crucially for you, it transfers in blocks. You can administer it through a web interface with credentials (or with none, if it's just listening on loopback).
- jermaustin1 8y ago
  I'll have to check that out. I was just desperate one afternoon after dropbox was failing to sync because of connectivity issues, and I knew bash just well enough to split, scp, rm, repeat.
  - rocmcd 8y ago
    You may also want to look into using rsync as an alternative to scp. rsync is much smarter about how it syncs data, so it should be better for your use case.
  - davrosthedalek 8y ago
    Look at rsync. rsync -avP file host:/path/on/host Repeat if it fails. It will not retransfer already transferred data.
- davehorne 8y ago
  A lot of why our users enjoy MASV is because it's dead simple to use for non-technical users. Not that syncthing is very hard but as a video pro dealing with clients the easier it is for them to just click a download button the more likely it will happen. Also because it's pay-as-you-go it's easy to charge back the transfer cost as a part of the project bill.
  - microcolonel 8y ago
    Most certainly. Another major benefit is that you folks provide storage, which inherently simplifies administering something like this. One benefit of Syncthing/BT Sync that I have not seen emulated by other vendors is peer-to-peer transfers, which can be considerably faster if both peers are on the same network or an adjacent one (including one on the same provider, or connected to the same exchange, or if the central service has no nodes on the same continent as the peer). The performance is just unmatched for such transfers, in my experience. A peer-based protocol could be a considerable help even if you don't intend on allowing direct peer-to-peer transfers, because (if implemented with sufficient care) it can allow you to choose better nodes to maximize throughput and integrity. I think there is a very legitimate need for centralized administration, I have all of my Syncthing node configurations synchronized over Syncthing itself (which is a bit of a risk, but not overly so), but I would consider paying for something (if I could have similar confidence and flexibility with the client software) more integrated, with some safeguards against accidentally disabling configuration sync (which would require me to directly configure the node) to a specific node, and some integrated ability to have nodes self-report and self-allocate.
    - davehorne 8y ago
      Absolutely agree. We really built our network out to be independent of our file transfer tool. It creates routes across any cloud provider and can integrate into any cloud storage provider. It also uses machine learning to determine which routes are optimal based on previous performance and time of day. I see a future where you could use our network for many different products such as an accelerated VPN, streaming live content, or as you pointed out as a pass through network for a P2P connection if both endpoints are available but it would route across our network to avoid congestion. Potentially even being able to make a copy of the transfer in the cloud for archive. Even P2P relies on routing across the public internet which is prone to congestion and is set up to do least-cost routing for the ISP's. Usually it's less of an issue though because the protocols help ensure it pushes through that congestion. Lot's of cool possibilities for this in the future and nowhere near enough time!
Retric 8y ago
Is your 20$/mo server 1Gbit? At a fairly common 100Mbit these file transfers would be 1/10 the speed and also need to be downloaded. Even slower if you need to send to multiple people. Running bit torrent let's peers share the data you upload and get the same net effect as this. But, that's not as far as I know a browser plugin. PS: I don't think it's a big deal, but executed well it might be a valuable niche.
- jermaustin1 8y ago
  It averages 400Mbit upload and 350Mbit download, not that it matters to me personally, as my employee can rarely upload faster than 4Mbit. That said, their 1Gbit connection will probably average around the same speed if they have more than a few users uploading 50GB files during normal business hours.
  - davehorne 8y ago
    Just to clarify we will fill a 1 Gbps pipe but our servers are provisioned to do much more than 1 Gbps. If many users are uploading at the same time we scale up beyond 1 Gbps that's not our network bandwidth maximum.
    - jermaustin1 8y ago
      That's good to know. 1Gbit is our "max" but that works just fine for the company's "network share" If we ever were to start hitting our bandwidth caps, I can pay to increase those, but the speed is pretty much fixed without changing providers. If speed were to ever be an issue, I would probably look at a different service like yours. It would be a massive hit at your current pricing, though.
davehorne 8y ago
Although we do chunk the upload and send the chunks in parallel to the server we do not at this point have the recover functionality implemented. We are able to recover from a disruption in internet connectivity with retry logic in javascript but if your computer crashed completely we would not recover. That is on the roadmap and expected to be released in the next couple of months though.
davehorne 8y ago
In terms of cost there are certainly cheaper ways to accomplish transfers. MASV is designed to be very fast and we have 9 servers across the world to enable transfers anywhere. If your doing local transfers your own file server is likely good enough but if your sending hundreds of GB's cross country or across the world you will likely run into major performance issues. It really comes down to how much your time is worth and if file transfer are a consistent requirement for your business. This is certainly intended to be a B2b tool less for consumer use.
justinsaccount 8y ago
huh? Why didn't you just use rsync?
- martin-adams 8y ago
  I presume you know that rsync is a better solution to the problem. Not all of us do. Would you mind explaining why rsync is a better solution?
  - vortico 8y ago
    For starters, it's a proper replacement for "a bash script that will chunk and scp the files to a central server, and if it fails a chunk, it leaves it in the directory for her to manually re-run". It's installed on almost all Linux and Mac systems, so if your customer is using a script already, they might as well use rsync.
  - seszett 8y ago
    > Would you mind explaining why rsync is a better solution? rsync basically chunks the files and sends the missing (or differing) chunks over scp, leaves the files in the destination directory if something fails, it is more or less an automated tool that already does what the GP is talking about.
    - mbesto 8y ago
      > it is more or less an automated tool that already does what the GP is talking about. But requires someone to be comfortable using a CLI.
      - j1elo 8y ago
        But that 'someone' is someone who is already running a Bash script and re-running it each time an error occurs. So I don't see the problem here about using a CLI.
        8y ago
halbritt 8y ago
I use a javascript library that works in the browser and supports S3 file copy chunking and resuming. Host that in a simple S3 bucket, create a target bucket with acceleration enabled and bob's your uncle. It's very efficient. I've never seen it fail to consume all of the available upstream bandwidth from the client. If this is of interest, I can go dig up the library I used.

microcolonel 8y ago

> I ended up writing a bash script that will chunk and scp the files to a central server, and if it fails a chunk, it leaves it in the directory for her to manually re-run. Use Syncthing, it traverses NATs, it is trivial to set up, it is seemingly very secure by its nature, and crucially for you, it transfers in blocks. You can administer it through a web interface with credentials (or with none, if it's just listening on loopback).

jermaustin1 8y ago
I'll have to check that out. I was just desperate one afternoon after dropbox was failing to sync because of connectivity issues, and I knew bash just well enough to split, scp, rm, repeat.
- rocmcd 8y ago
  You may also want to look into using rsync as an alternative to scp. rsync is much smarter about how it syncs data, so it should be better for your use case.
- davrosthedalek 8y ago
  Look at rsync. rsync -avP file host:/path/on/host Repeat if it fails. It will not retransfer already transferred data.
davehorne 8y ago
A lot of why our users enjoy MASV is because it's dead simple to use for non-technical users. Not that syncthing is very hard but as a video pro dealing with clients the easier it is for them to just click a download button the more likely it will happen. Also because it's pay-as-you-go it's easy to charge back the transfer cost as a part of the project bill.
- microcolonel 8y ago
  Most certainly. Another major benefit is that you folks provide storage, which inherently simplifies administering something like this. One benefit of Syncthing/BT Sync that I have not seen emulated by other vendors is peer-to-peer transfers, which can be considerably faster if both peers are on the same network or an adjacent one (including one on the same provider, or connected to the same exchange, or if the central service has no nodes on the same continent as the peer). The performance is just unmatched for such transfers, in my experience. A peer-based protocol could be a considerable help even if you don't intend on allowing direct peer-to-peer transfers, because (if implemented with sufficient care) it can allow you to choose better nodes to maximize throughput and integrity. I think there is a very legitimate need for centralized administration, I have all of my Syncthing node configurations synchronized over Syncthing itself (which is a bit of a risk, but not overly so), but I would consider paying for something (if I could have similar confidence and flexibility with the client software) more integrated, with some safeguards against accidentally disabling configuration sync (which would require me to directly configure the node) to a specific node, and some integrated ability to have nodes self-report and self-allocate.
  - davehorne 8y ago
    Absolutely agree. We really built our network out to be independent of our file transfer tool. It creates routes across any cloud provider and can integrate into any cloud storage provider. It also uses machine learning to determine which routes are optimal based on previous performance and time of day. I see a future where you could use our network for many different products such as an accelerated VPN, streaming live content, or as you pointed out as a pass through network for a P2P connection if both endpoints are available but it would route across our network to avoid congestion. Potentially even being able to make a copy of the transfer in the cloud for archive. Even P2P relies on routing across the public internet which is prone to congestion and is set up to do least-cost routing for the ISP's. Usually it's less of an issue though because the protocols help ensure it pushes through that congestion. Lot's of cool possibilities for this in the future and nowhere near enough time!

Retric 8y ago

Is your 20$/mo server 1Gbit? At a fairly common 100Mbit these file transfers would be 1/10 the speed and also need to be downloaded. Even slower if you need to send to multiple people. Running bit torrent let's peers share the data you upload and get the same net effect as this. But, that's not as far as I know a browser plugin. PS: I don't think it's a big deal, but executed well it might be a valuable niche.

jermaustin1 8y ago
It averages 400Mbit upload and 350Mbit download, not that it matters to me personally, as my employee can rarely upload faster than 4Mbit. That said, their 1Gbit connection will probably average around the same speed if they have more than a few users uploading 50GB files during normal business hours.
- davehorne 8y ago
  Just to clarify we will fill a 1 Gbps pipe but our servers are provisioned to do much more than 1 Gbps. If many users are uploading at the same time we scale up beyond 1 Gbps that's not our network bandwidth maximum.
  - jermaustin1 8y ago
    That's good to know. 1Gbit is our "max" but that works just fine for the company's "network share" If we ever were to start hitting our bandwidth caps, I can pay to increase those, but the speed is pretty much fixed without changing providers. If speed were to ever be an issue, I would probably look at a different service like yours. It would be a massive hit at your current pricing, though.

davehorne 8y ago

Although we do chunk the upload and send the chunks in parallel to the server we do not at this point have the recover functionality implemented. We are able to recover from a disruption in internet connectivity with retry logic in javascript but if your computer crashed completely we would not recover. That is on the roadmap and expected to be released in the next couple of months though.

davehorne 8y ago

In terms of cost there are certainly cheaper ways to accomplish transfers. MASV is designed to be very fast and we have 9 servers across the world to enable transfers anywhere. If your doing local transfers your own file server is likely good enough but if your sending hundreds of GB's cross country or across the world you will likely run into major performance issues. It really comes down to how much your time is worth and if file transfer are a consistent requirement for your business. This is certainly intended to be a B2b tool less for consumer use.

justinsaccount 8y ago

huh? Why didn't you just use rsync?

martin-adams 8y ago
I presume you know that rsync is a better solution to the problem. Not all of us do. Would you mind explaining why rsync is a better solution?
- vortico 8y ago
  For starters, it's a proper replacement for "a bash script that will chunk and scp the files to a central server, and if it fails a chunk, it leaves it in the directory for her to manually re-run". It's installed on almost all Linux and Mac systems, so if your customer is using a script already, they might as well use rsync.
- seszett 8y ago
  > Would you mind explaining why rsync is a better solution? rsync basically chunks the files and sends the missing (or differing) chunks over scp, leaves the files in the destination directory if something fails, it is more or less an automated tool that already does what the GP is talking about.
  - mbesto 8y ago
    > it is more or less an automated tool that already does what the GP is talking about. But requires someone to be comfortable using a CLI.
    - j1elo 8y ago
      But that 'someone' is someone who is already running a Bash script and re-running it each time an error occurs. So I don't see the problem here about using a CLI.
      - 8y ago

halbritt 8y ago

I use a javascript library that works in the browser and supports S3 file copy chunking and resuming. Host that in a simple S3 bucket, create a target bucket with acceleration enabled and bob's your uncle. It's very efficient. I've never seen it fail to consume all of the available upstream bandwidth from the client. If this is of interest, I can go dig up the library I used.