5 ms·

Working from home has really put a spotlight on the terrible asymmetric upload speeds of most cable internet. I can get 1 gb down but only 50 mb upload. Certai

Working from home has really put a spotlight on the terrible asymmetric upload speeds of most cable internet. I can get 1 gb down but only 50 mb upload. Certain tasks (like uploading a docker image) I cant do at all from my personal computer. The layman has no idea the difference, and even most legislators don't understand the issue ("isn't 1 gb fast enough?")

__MatrixMan__ 2y ago
I got lucky and fiber became available in my neighborhood around the same time I noticed how painful pushing images over cable was. Hopefully you'll get that option soon too. For the unlucky, maybe we can take advantage of the fact that most image pushes have a predecessor which they are 99% similar to. With some care about the image contents (nar instead of tar, gzip --rsyncable, etc) we ought to be able to save a lot of bandwidth by using rsync on top of the previous version instead of transferring each image independently.
LoganDark 2y ago
> I can get 1 gb down but only 50 mb upload. Certain tasks (like uploading a docker image) I cant do at all from my personal computer. As someone who used to work with LLMs, I feel this pain. It would take days for me to upload models. Other community members rent GPU servers to do the training on just so that their data will already be in the cloud, but that's not really a sustainable solution for me since I like tinkering at home. I have around the same speeds, btw. 1Gb down and barely 40Mb up. Factor of 25!
- latency-guy2 2y ago
  I feel your pain, I haven't been in ML world directly for a few years now but I've done the same exercise multiple times. The worst part is that block compression actually does not help if it doesn't do a significantly good job of compression AND decompression. My use case had to immediately deploy the models across a few nodes in a live environment at customer sites. Cloud wasn't an option for us and fiber was also unavailable many times. The fastest transport protocol was someone's car and a workday of wages.
  - LoganDark 2y ago
    > The fastest transport protocol was someone's car and a workday of wages. This is actually the entire premise of AWS Snowball: send someone a bunch of storage space, have them copy their data to that storage, then just ship the storage back with the data on it. It can be several orders of magnitude faster and easier than an internet transfer. Sneakernet really works. https://en.wikipedia.org/wiki/Sneakernet https://en.wikipedia.org/wiki/Sneakernet
    - fragmede 2y ago
      it would be totally cyberpunk to have a data cafe where you bring your hard drive to upload to the cloud and you'd pay by the terabyte/s. have all day? cheap. need to do it in 30 mins? pay up.
packetlost 2y ago
This. I've been fighting AT&T for awhile because they told the FCC (via their broadband maps [0]) that they supply fiber to my condo, so I bought it expecting to get fiber. Well when I finally go to set up internet service, they only offer 50/5 DSL service. Fortunately I can get cable that has usable down speeds but the up is substantially less than 50 with garbage routing. I'm not very happy. [0]: https://broadbandmap.fcc.gov/ https://broadbandmap.fcc.gov/
- dsissitka 2y ago
  I'm not sure if it's true but I've heard they take that very seriously. Have you filed a complaint with the FCC? Both times I had to do it things got sorted very quickly. https://consumercomplaints.fcc.gov/hc/en-us/articles/115002206106-Internet-Form-Descriptions-of-Complaint-Issues https://consumercomplaints.fcc.gov/hc/en-us/articles/1150022...
  - packetlost 2y ago
    I actually did file a complaint, but the FCC rejected it for reasons that I were not communicated.
- silisili 2y ago
  Good luck. At one time I was experiencing high ping times and near non existent speed from ATT Fiber to Online.fr's network. I did 80% of the diagnostics for them and provided the details and of course a nudge as to what I felt the issue could be. It's extremely frustrating to be a networking person having to deal with home internet CS. To my surprise, it actually did get to their networking team who replied saying the peer was fine and try again. The problem with that was that it came 8 months later, long after I'd left the area and didn't even have service with them anymore.
sneak 2y ago
Why are you building images at home for upload? In my experience, it’s much easier to upload code or commits and build/push artifacts in/from the datacenter, whether manually or via CI. It can be as simple as exporting DOCKER_HOST=“ssh://root@host”. Docker handles uploading the relevant parts of your cwd to the server. I have a wickedly fast workstation but spot instances that are way way faster (and on 10gbps symmetric) are pennies. Added bonus: I can use them from a slow computer with no degradation.
imp0cat 2y ago
YMMV, but this can be usually mitigated by connecting to a machine in a datacentre at work and doing all the stuff there (like building and uploading docker images).

__MatrixMan__ 2y ago

I got lucky and fiber became available in my neighborhood around the same time I noticed how painful pushing images over cable was. Hopefully you'll get that option soon too. For the unlucky, maybe we can take advantage of the fact that most image pushes have a predecessor which they are 99% similar to. With some care about the image contents (nar instead of tar, gzip --rsyncable, etc) we ought to be able to save a lot of bandwidth by using rsync on top of the previous version instead of transferring each image independently.

LoganDark 2y ago

> I can get 1 gb down but only 50 mb upload. Certain tasks (like uploading a docker image) I cant do at all from my personal computer. As someone who used to work with LLMs, I feel this pain. It would take days for me to upload models. Other community members rent GPU servers to do the training on just so that their data will already be in the cloud, but that's not really a sustainable solution for me since I like tinkering at home. I have around the same speeds, btw. 1Gb down and barely 40Mb up. Factor of 25!

latency-guy2 2y ago
I feel your pain, I haven't been in ML world directly for a few years now but I've done the same exercise multiple times. The worst part is that block compression actually does not help if it doesn't do a significantly good job of compression AND decompression. My use case had to immediately deploy the models across a few nodes in a live environment at customer sites. Cloud wasn't an option for us and fiber was also unavailable many times. The fastest transport protocol was someone's car and a workday of wages.
- LoganDark 2y ago
  > The fastest transport protocol was someone's car and a workday of wages. This is actually the entire premise of AWS Snowball: send someone a bunch of storage space, have them copy their data to that storage, then just ship the storage back with the data on it. It can be several orders of magnitude faster and easier than an internet transfer. Sneakernet really works. https://en.wikipedia.org/wiki/Sneakernet https://en.wikipedia.org/wiki/Sneakernet
  - fragmede 2y ago
    it would be totally cyberpunk to have a data cafe where you bring your hard drive to upload to the cloud and you'd pay by the terabyte/s. have all day? cheap. need to do it in 30 mins? pay up.

packetlost 2y ago

This. I've been fighting AT&T for awhile because they told the FCC (via their broadband maps [0]) that they supply fiber to my condo, so I bought it expecting to get fiber. Well when I finally go to set up internet service, they only offer 50/5 DSL service. Fortunately I can get cable that has usable down speeds but the up is substantially less than 50 with garbage routing. I'm not very happy. [0]: https://broadbandmap.fcc.gov/ https://broadbandmap.fcc.gov/

dsissitka 2y ago
I'm not sure if it's true but I've heard they take that very seriously. Have you filed a complaint with the FCC? Both times I had to do it things got sorted very quickly. https://consumercomplaints.fcc.gov/hc/en-us/articles/115002206106-Internet-Form-Descriptions-of-Complaint-Issues https://consumercomplaints.fcc.gov/hc/en-us/articles/1150022...
- packetlost 2y ago
  I actually did file a complaint, but the FCC rejected it for reasons that I were not communicated.
silisili 2y ago
Good luck. At one time I was experiencing high ping times and near non existent speed from ATT Fiber to Online.fr's network. I did 80% of the diagnostics for them and provided the details and of course a nudge as to what I felt the issue could be. It's extremely frustrating to be a networking person having to deal with home internet CS. To my surprise, it actually did get to their networking team who replied saying the peer was fine and try again. The problem with that was that it came 8 months later, long after I'd left the area and didn't even have service with them anymore.

sneak 2y ago

Why are you building images at home for upload? In my experience, it’s much easier to upload code or commits and build/push artifacts in/from the datacenter, whether manually or via CI. It can be as simple as exporting DOCKER_HOST=“ssh://root@host”. Docker handles uploading the relevant parts of your cwd to the server. I have a wickedly fast workstation but spot instances that are way way faster (and on 10gbps symmetric) are pennies. Added bonus: I can use them from a slow computer with no degradation.

imp0cat 2y ago

YMMV, but this can be usually mitigated by connecting to a machine in a datacentre at work and doing all the stuff there (like building and uploading docker images).