3 ms·

Ask HN: How to update a large network of computers

1 points by huherto 18y ago

I am working for a company that has several thousand stores located in a large geographic area. Every time data changes have to be delivered to the stores (around 500MB each) there are about a dozen servers transferring the info to each individual store. As you probably imagine it takes a long time to do it and it requires a lot of bandwidth. It will be faster if the stores could be updated from other stores, specially if they are close from each other. Do you know of any software that can help us do it. May be some type of mirroring or maybe git or some other version control system work for it. Any suggestions?

noodle 18y ago
http://en.wikipedia.org/wiki/BitTorrent http://en.wikipedia.org/wiki/BitTorrent (i would link to the protocol but the link doesn't show up correctly here) there are probably better options, but this one is already implemented and likely wouldn't be that hard to make work for your situation. might require you writing something yourself to suit your needs, or you might be able to find a turnkey solution.
- huherto 18y ago
  Ah, thanks I will look into it
st3fan 18y ago
Assuming these stores are behind DSL ... DSL upload speeds are usually terribly slow. Are you sure that it would actually be faster than downloading from some central server?
- huherto 18y ago
  I think so, even if the DSL upload is slower, we would not have the bottle neck in the central servers.
st3fan 18y ago
Some other thoughts: You mention data changes. So what about sending just the changes? What is the real problem? That it takes a lot of time? Or that it requires bandwidth? If downloads are slow, are your central servers overloaded? What about putting the data on a CDN or Amazon S3?
- huherto 18y ago
  The time.
  - st3fan 18y ago
    Sounds like you simply don't have enough bandwidth then. Fortunately that is an easily solvable problem in 2008 :-)
bayareaguy 18y ago
A friend of mine told me this was a Google interview question. Anyways, I'd recommend taking a look at this: The Frisbee server uses a custom multicast protocol to distribute highly-compressed disk images to consenting clients. The Frisbee client is a multi-threaded marvel that receives, decompresses, and smacks data down on your hard drive, really, really fast. Frisbee can be used to install complete disk images or single partition images. - http://www.cs.utah.edu/flux/papers/frisbee-usenix03 http://www.cs.utah.edu/flux/papers/frisbee-usenix03 - http://www.emulab.net/software.php3 http://www.emulab.net/software.php3
bayareaguy 18y ago
Here's another useful tool I just remembered. http://saf.bio.caltech.edu/nettee.html http://saf.bio.caltech.edu/nettee.html

1 points by huherto 18y ago

noodle 18y ago

http://en.wikipedia.org/wiki/BitTorrent http://en.wikipedia.org/wiki/BitTorrent (i would link to the protocol but the link doesn't show up correctly here) there are probably better options, but this one is already implemented and likely wouldn't be that hard to make work for your situation. might require you writing something yourself to suit your needs, or you might be able to find a turnkey solution.

huherto 18y ago
Ah, thanks I will look into it

st3fan 18y ago

Assuming these stores are behind DSL ... DSL upload speeds are usually terribly slow. Are you sure that it would actually be faster than downloading from some central server?

huherto 18y ago
I think so, even if the DSL upload is slower, we would not have the bottle neck in the central servers.

st3fan 18y ago

Some other thoughts: You mention data changes. So what about sending just the changes? What is the real problem? That it takes a lot of time? Or that it requires bandwidth? If downloads are slow, are your central servers overloaded? What about putting the data on a CDN or Amazon S3?

huherto 18y ago
The time.
- st3fan 18y ago
  Sounds like you simply don't have enough bandwidth then. Fortunately that is an easily solvable problem in 2008 :-)

bayareaguy 18y ago

A friend of mine told me this was a Google interview question. Anyways, I'd recommend taking a look at this: The Frisbee server uses a custom multicast protocol to distribute highly-compressed disk images to consenting clients. The Frisbee client is a multi-threaded marvel that receives, decompresses, and smacks data down on your hard drive, really, really fast. Frisbee can be used to install complete disk images or single partition images. - http://www.cs.utah.edu/flux/papers/frisbee-usenix03 http://www.cs.utah.edu/flux/papers/frisbee-usenix03 - http://www.emulab.net/software.php3 http://www.emulab.net/software.php3

bayareaguy 18y ago

Here's another useful tool I just remembered. http://saf.bio.caltech.edu/nettee.html http://saf.bio.caltech.edu/nettee.html