4 ms·

I don't really understand. So the goal of this is for the program to tell the person with files to load them up on a hard disk and then give them to the person

by thefool 16y ago

I don't really understand. So the goal of this is for the program to tell the person with files to load them up on a hard disk and then give them to the person requesting the files, who then picks them up from the dropbox, loads them on her computer, and then puts the flash drive back into the dropbox? If so, then I guess all this really does is gives you a nice way of understanding who has which files, right? If this is how it works, then I don't really understand what the external caches do for anyone.

drewcrawford 16y ago
The system you've described doesn't really scale to, say, 1000 people with 10TB of files. Even with a lot of flash drives in that cache. The sneakernet is engineered basically to let you keep making trips to that cache outside your dorm room and somehow magical files appear from a block or two away. Also, files from you magically appear a block or two away when other people ask for them. The sneakernet takes care of all the routing to make lots of files show up in lots of places geographically close to you without too much overhead. I've found lots of times I will request a file that the system in its cleverness has already cached locally on my HD. So "getting it off of the sneakernet" is actually instantaneous.
- pyre 16y ago
  > I've found lots of times I will request a file that the system in its cleverness has already cached locally on my HD. So "getting it off of the sneakernet" is actually instantaneous. Huh? I thought that the point was that request files and then get them. How do you already have a file that you didn't request? Is this distributing files across people (i.e. FreeNet) so you have a bunch of files that are only on your machine for the purpose of being an intermediary to others? Or is this running some sort of predictive algorithm that tries to figure out what you will request before you do?
  - JoachimSchipper 16y ago
    I think these files were on the disk for the purpose of distributing them to other who had requested them.
  - drewcrawford 16y ago
    In a very generic answer to your question, suppose you're moving a 1GB file from A to B with an 8GB flash drive. It doesn't cost anyone anything to move an extra 7GB of data that somebody might need later. You can 8x your throughput for free. But, at the same time, you don't want a flash drive to be in a constant state of being near-full so it can't fit the new StarCraft beta. So there's some logic that takes files that seem to be spreading around anyway and sort of spreads them around ahead of when they're requested. This both increases redundancy and increases network throughput. It's not exactly machine learning, but it does, in practice, provide decent results.
- thefool 16y ago
  By "magic" do you mean "through the internet" or is something cleverer going on? Also, by cache are you generally referring to some place where files are stored? Or does it mean something else? I understand the system I just described doesn't really scale well, I'm just not really understanding how it does work from the documentation in the wiki.
  - drewcrawford 16y ago
    cache == a flash drive or group of flash drives (although it could also be a USB HD, etc.) Not through the internet. Through moving from flash drive to flash drive. If you follow the "testing" section along with the network maps for both the "basic configuration guide" and the "large configuration guide", that should clear up a lot of the magic.
    - thefool 16y ago
      Oh ok, now it makes sense. You might want to say like storage device instead of cache, that would have made it less confusing for me to understand from reading the wiki. It also seems like the system is just begging for caches to go missing (though I understand that as all the files are encrypted, it should be hard in theory to actually get data if you arent authorized access to sneakernet). Are you hashing the files all the time to make sure that files aren't getting corrupted in transit or when taken from someones external cache?

by thefool 16y ago

drewcrawford 16y ago

The system you've described doesn't really scale to, say, 1000 people with 10TB of files. Even with a lot of flash drives in that cache. The sneakernet is engineered basically to let you keep making trips to that cache outside your dorm room and somehow magical files appear from a block or two away. Also, files from you magically appear a block or two away when other people ask for them. The sneakernet takes care of all the routing to make lots of files show up in lots of places geographically close to you without too much overhead. I've found lots of times I will request a file that the system in its cleverness has already cached locally on my HD. So "getting it off of the sneakernet" is actually instantaneous.

pyre 16y ago
> I've found lots of times I will request a file that the system in its cleverness has already cached locally on my HD. So "getting it off of the sneakernet" is actually instantaneous. Huh? I thought that the point was that request files and then get them. How do you already have a file that you didn't request? Is this distributing files across people (i.e. FreeNet) so you have a bunch of files that are only on your machine for the purpose of being an intermediary to others? Or is this running some sort of predictive algorithm that tries to figure out what you will request before you do?
- JoachimSchipper 16y ago
  I think these files were on the disk for the purpose of distributing them to other who had requested them.
- drewcrawford 16y ago
  In a very generic answer to your question, suppose you're moving a 1GB file from A to B with an 8GB flash drive. It doesn't cost anyone anything to move an extra 7GB of data that somebody might need later. You can 8x your throughput for free. But, at the same time, you don't want a flash drive to be in a constant state of being near-full so it can't fit the new StarCraft beta. So there's some logic that takes files that seem to be spreading around anyway and sort of spreads them around ahead of when they're requested. This both increases redundancy and increases network throughput. It's not exactly machine learning, but it does, in practice, provide decent results.
thefool 16y ago
By "magic" do you mean "through the internet" or is something cleverer going on? Also, by cache are you generally referring to some place where files are stored? Or does it mean something else? I understand the system I just described doesn't really scale well, I'm just not really understanding how it does work from the documentation in the wiki.
- drewcrawford 16y ago
  cache == a flash drive or group of flash drives (although it could also be a USB HD, etc.) Not through the internet. Through moving from flash drive to flash drive. If you follow the "testing" section along with the network maps for both the "basic configuration guide" and the "large configuration guide", that should clear up a lot of the magic.
  - thefool 16y ago
    Oh ok, now it makes sense. You might want to say like storage device instead of cache, that would have made it less confusing for me to understand from reading the wiki. It also seems like the system is just begging for caches to go missing (though I understand that as all the files are encrypted, it should be hard in theory to actually get data if you arent authorized access to sneakernet). Are you hashing the files all the time to make sure that files aren't getting corrupted in transit or when taken from someones external cache?