4 ms·

Have we come to rely on "cloud services" so much that we've forgotten how to do this ourselves? When Dropbox came out, many people LOL'd (especially here on th

Have we come to rely on "cloud services" so much that we've forgotten how to do this ourselves? When Dropbox came out, many people LOL'd (especially here on this site where most people could build Dropbox in a weekend) because it isn't that complicated. It was just pretty and targeting the less-technical. But now here we are, on the same site where people used to be able to build this in a weekend, asking how to find an alternative because (presumably) nobody can build this in a weekend.

imclaren 3y ago
I think that you can’t build it in a weekend. Well I couldn’t. As soon as you try to rapidly and reliably sync, say, 20k+ files over 3+ clients with intermittent internet connections, it gets surprisingly complicated.
- withinboredom 3y ago
  That’s a rather specific edge case and I can’t think of any sync that handles it relatively well except for BitTorrent. At that point you are limited by so many things, it’s not worth optimizing for because all the error cases you have to handle (inode exhaustion, network limits, etc). 99.99% of the time only a single file changes at a time, which is the use case here.
  - imclaren 3y ago
    Hmm the macOS photos folders have this many files so I’m not sure it’s an edge case to sync this many files between at least 2 clients. And the issues you mention are real even when syncing relatively static large folders. But you’re right that there are many such error cases to handle if you want a real iCloud / Dropbox alternative.
  - al_borland 3y ago
    Is it really an edge case these days? A user’s very first sync is likely to be the largest one, to seed all the data into the could. That could easily be thousands of files. Installing on a desktop, laptop, and mobile device would bring 3 clients into the mix right away. So the user’s first use is likely to be one of the more complex operations, and the one they absolutely needs to work, as a failure here will lead to the user looking elsewhere for a solution.
    - withinboredom 3y ago
      > A user’s very first sync is likely to be the largest one, to seed all the data into the could. You are almost guaranteed that the first sync results in zero conflicts. It's just a mass-upload. Even if you bring other clients in, there is still a very low chance and "last-one-wins" is a pretty decent policy for who gets the original filename and then other gets a " (copy)" suffix or something. Doesn't really matter as long as it's documented and intuitive.
      - imclaren 3y ago
        FWIW having built a sync service I can tell you that this is an issue in practice, and you have to test for these differences in any event.
        withinboredom 3y ago
        My point is that for a weekend PoC that you want to use for yourself… it’s rather a non-issue. Further, you need to create a lot of primitives before you can even tackle the problem correctly, so initially, it is an unsolvable problem until you have those primitives.
LVB 3y ago
I suspect that relatively few people, even on this site, could build a background file sync application for iOS very easily.
- withinboredom 3y ago
  I would agree with you there… but in 2007ish, there was no smart phone. There was basically only Windows, weird Macs, and some people on Linux for some reason. JQuery was still the way we did things during those times. So, yeah, I guess back then, things were relatively straightforward to build something like this. Today, there are many OS’s, file systems, ipv6, shitty wifi, and a whole host of problems that didn’t really exist back then. Heck, back then, most people didn’t even have laptops so you didn’t even need to worry about your programs going to sleep in the middle of processing.
wtallis 3y ago
A big part of the problem is that Dropbox is a fundamentally different product than what it used to be. Today's Dropbox is not even attempting to be a tool that simply synchronizes directory contents across all of your machines. Instead, it is trying to be a network filesystem with a local cache, with various other features layered on top. It's a lot more complex, built with leakier abstractions. The original Dropbox could be a viable product to disrupt today's Dropbox for many of the same reasons Dropbox originally achieved success.

imclaren 3y ago

I think that you can’t build it in a weekend. Well I couldn’t. As soon as you try to rapidly and reliably sync, say, 20k+ files over 3+ clients with intermittent internet connections, it gets surprisingly complicated.

withinboredom 3y ago
That’s a rather specific edge case and I can’t think of any sync that handles it relatively well except for BitTorrent. At that point you are limited by so many things, it’s not worth optimizing for because all the error cases you have to handle (inode exhaustion, network limits, etc). 99.99% of the time only a single file changes at a time, which is the use case here.
- imclaren 3y ago
  Hmm the macOS photos folders have this many files so I’m not sure it’s an edge case to sync this many files between at least 2 clients. And the issues you mention are real even when syncing relatively static large folders. But you’re right that there are many such error cases to handle if you want a real iCloud / Dropbox alternative.
- al_borland 3y ago
  Is it really an edge case these days? A user’s very first sync is likely to be the largest one, to seed all the data into the could. That could easily be thousands of files. Installing on a desktop, laptop, and mobile device would bring 3 clients into the mix right away. So the user’s first use is likely to be one of the more complex operations, and the one they absolutely needs to work, as a failure here will lead to the user looking elsewhere for a solution.
  - withinboredom 3y ago
    > A user’s very first sync is likely to be the largest one, to seed all the data into the could. You are almost guaranteed that the first sync results in zero conflicts. It's just a mass-upload. Even if you bring other clients in, there is still a very low chance and "last-one-wins" is a pretty decent policy for who gets the original filename and then other gets a " (copy)" suffix or something. Doesn't really matter as long as it's documented and intuitive.
    - imclaren 3y ago
      FWIW having built a sync service I can tell you that this is an issue in practice, and you have to test for these differences in any event.
      - withinboredom 3y ago
        My point is that for a weekend PoC that you want to use for yourself… it’s rather a non-issue. Further, you need to create a lot of primitives before you can even tackle the problem correctly, so initially, it is an unsolvable problem until you have those primitives.

LVB 3y ago

I suspect that relatively few people, even on this site, could build a background file sync application for iOS very easily.

withinboredom 3y ago
I would agree with you there… but in 2007ish, there was no smart phone. There was basically only Windows, weird Macs, and some people on Linux for some reason. JQuery was still the way we did things during those times. So, yeah, I guess back then, things were relatively straightforward to build something like this. Today, there are many OS’s, file systems, ipv6, shitty wifi, and a whole host of problems that didn’t really exist back then. Heck, back then, most people didn’t even have laptops so you didn’t even need to worry about your programs going to sleep in the middle of processing.

wtallis 3y ago

A big part of the problem is that Dropbox is a fundamentally different product than what it used to be. Today's Dropbox is not even attempting to be a tool that simply synchronizes directory contents across all of your machines. Instead, it is trying to be a network filesystem with a local cache, with various other features layered on top. It's a lot more complex, built with leakier abstractions. The original Dropbox could be a viable product to disrupt today's Dropbox for many of the same reasons Dropbox originally achieved success.