3 ms·

I can speculate too, but the reason I asked OP is because he would know and have actual numbers. Regarding your questions: 1) Loss of data only matters in som

by blagie 3y ago

I can speculate too, but the reason I asked OP is because he would know and have actual numbers. Regarding your questions: 1) Loss of data only matters in some contexts. For cloud backups, it really doesn't matter much since it's already a backup. It only matters if the cloud provider AND my local system go down at the same time. 2) Still, at scale, avoiding loss of data is CHEAP. Something analogous to RAID6, with data striped across data centers, means the overhead can be quite low. 3) Yes, I do pay people salaries. Buying disks -- even stupidly and just having two copies of everything -- works out a lot cheaper than cloud storage. Ask any video shop if they use (1) cloud storage or (2) a storage closet full of hard drives. The numbers don't work out for me. By my estimate, I could provide the same service which rsync does for *much* less money and dominate the cloud storage market. I'm obviously wrong, or someone would have done so already, but I don't know how I'm wrong.

acdha 3y ago
> Loss of data only matters in some contexts It matters in all contexts where you’re charging people to store their stuff. The iron law of backups is that if you haven’t tested them, you don’t have them and that’s the case here because people will usually only use yours after they’ve found a problem locally. Consider also the problem of synchronization: if your remote backup fails, someone with the average American anemic upload might need weeks to re-upload everything, especially if they don’t want to interfere with everything else on their network.
- blagie 3y ago
  Personal feedback: You have a tendency to make obvious, unsubstantiated, often slightly incorrect statements with a very high level of confidence and assuming you're talking to an idiot. I'm not learning anything from you. Let's both move on. This is my last post in this thread. If you do want to learn something yourself, I'd suggest, as a homework exercise, taking the time to architect this out, and running some numbers on things like: - How often local devices fail (HDD AFR) - How often remote devices fail - Overhead of monitoring / catching each type of failure - Time for synchronization If you can, include algorithms along the lines of those in RAID5. I would also back-of-the-envelope cost estimates. The goal of backups is to decrease odds of data loss to some manageable, quantifiable risk (e.g. from ≈1% AFR to e.g. ≈0.01% AFR), and what tolerable levels of risk are is an ROI and expected value calculation: odds of failure * cost of failure should be less than the cost of the backup system. It's a good homework exercise, and purely for your benefit. E.g. that's the sort of question I might ask on an interview, and answers like yours would not lead to a hire decision. The people I least enjoy work with operate from best practices (often using strong language like "iron laws"), and can't operate from first principles. In this case, this is all quantifiable. You don't need to post your response. There's a reason I asked someone who might now, and you cutting in was a pure distraction. I'm done here.
  - acdha 3y ago
    I find it interesting that you are claiming to have known all of this all but your original proposal was “plugging a 20TB HDD into a RaspberryPI, tossing it on a high speed connection (and charging for transfer), and getting 6x annual returns”. If I was asking someone else for free training, I’d want my question to accurately reflect my starting knowledge and willingness to do my own homework.
    - blagie 3y ago
      Imagine this scenario: - You have an uncle with skin cancer. You bump into an oncologist at a party. You ask some technical questions. Annoying lady cuts in and starts talking about essential oils. Oncologist wanders off. - You're an armchair general (a nerd who has never been in the army, but reads a lot), and you bump into a government intelligence analyst who studies the situation in Ukraine. It's a goldmine! Someone who read an article on Fox News cuts in and starts lecturing you on Putin. It's not even quite wrong, but it's so shallow it's pointless. The analyst wanders off. That's basically how I read your intervention. I didn't ask you for free training or anything else. I asked rsync a question. You decided to butt into a conversation uninvited, without having anything to contribute, figured you knew more on the topic than a cloud storage provider, and proceeded to deliver a lecture at the level of what my child might learn in their grade school IT class. I'm not claiming to be brilliant, but I don't get why people like you do that. I think that's a place you could provide a more interesting insight than about anything technical. (And to answer your question: I wrote a question and not an essay; an extreme example is often just a concise way of phrasing something like this)
      - acdha 3y ago
        > You have an uncle with skin cancer. You bump into an oncologist at a party. You ask some technical questions. Annoying lady cuts in and starts talking about essential oils. You know, you could have written that without the casual sexism. I also find it interesting how you feel this intense need to simultaneously claim that you already knew about all of the same details I mentioned but also dismiss them as irrelevant: if they’re grade school level, what does that say about your original question not indicating awareness of them? If they’re “essential oils”-level quackery, which is hard to reconcile with them being in grade school curriculum, why are you so concerned about saying you knew everything already?

by blagie 3y ago

acdha 3y ago

> Loss of data only matters in some contexts It matters in all contexts where you’re charging people to store their stuff. The iron law of backups is that if you haven’t tested them, you don’t have them and that’s the case here because people will usually only use yours after they’ve found a problem locally. Consider also the problem of synchronization: if your remote backup fails, someone with the average American anemic upload might need weeks to re-upload everything, especially if they don’t want to interfere with everything else on their network.

blagie 3y ago
Personal feedback: You have a tendency to make obvious, unsubstantiated, often slightly incorrect statements with a very high level of confidence and assuming you're talking to an idiot. I'm not learning anything from you. Let's both move on. This is my last post in this thread. If you do want to learn something yourself, I'd suggest, as a homework exercise, taking the time to architect this out, and running some numbers on things like: - How often local devices fail (HDD AFR) - How often remote devices fail - Overhead of monitoring / catching each type of failure - Time for synchronization If you can, include algorithms along the lines of those in RAID5. I would also back-of-the-envelope cost estimates. The goal of backups is to decrease odds of data loss to some manageable, quantifiable risk (e.g. from ≈1% AFR to e.g. ≈0.01% AFR), and what tolerable levels of risk are is an ROI and expected value calculation: odds of failure * cost of failure should be less than the cost of the backup system. It's a good homework exercise, and purely for your benefit. E.g. that's the sort of question I might ask on an interview, and answers like yours would not lead to a hire decision. The people I least enjoy work with operate from best practices (often using strong language like "iron laws"), and can't operate from first principles. In this case, this is all quantifiable. You don't need to post your response. There's a reason I asked someone who might now, and you cutting in was a pure distraction. I'm done here.
- acdha 3y ago
  I find it interesting that you are claiming to have known all of this all but your original proposal was “plugging a 20TB HDD into a RaspberryPI, tossing it on a high speed connection (and charging for transfer), and getting 6x annual returns”. If I was asking someone else for free training, I’d want my question to accurately reflect my starting knowledge and willingness to do my own homework.
  - blagie 3y ago
    Imagine this scenario: - You have an uncle with skin cancer. You bump into an oncologist at a party. You ask some technical questions. Annoying lady cuts in and starts talking about essential oils. Oncologist wanders off. - You're an armchair general (a nerd who has never been in the army, but reads a lot), and you bump into a government intelligence analyst who studies the situation in Ukraine. It's a goldmine! Someone who read an article on Fox News cuts in and starts lecturing you on Putin. It's not even quite wrong, but it's so shallow it's pointless. The analyst wanders off. That's basically how I read your intervention. I didn't ask you for free training or anything else. I asked rsync a question. You decided to butt into a conversation uninvited, without having anything to contribute, figured you knew more on the topic than a cloud storage provider, and proceeded to deliver a lecture at the level of what my child might learn in their grade school IT class. I'm not claiming to be brilliant, but I don't get why people like you do that. I think that's a place you could provide a more interesting insight than about anything technical. (And to answer your question: I wrote a question and not an essay; an extreme example is often just a concise way of phrasing something like this)
    - acdha 3y ago
      > You have an uncle with skin cancer. You bump into an oncologist at a party. You ask some technical questions. Annoying lady cuts in and starts talking about essential oils. You know, you could have written that without the casual sexism. I also find it interesting how you feel this intense need to simultaneously claim that you already knew about all of the same details I mentioned but also dismiss them as irrelevant: if they’re grade school level, what does that say about your original question not indicating awareness of them? If they’re “essential oils”-level quackery, which is hard to reconcile with them being in grade school curriculum, why are you so concerned about saying you knew everything already?