16 ms·

This is all because someone thought it would be “easy” to add a stream of recurring revenue for cloud photo storage but didn’t take the time to design the servi

This is all because someone thought it would be “easy” to add a stream of recurring revenue for cloud photo storage but didn’t take the time to design the service from a technical perspective to be resilient enough to not crash and burn.

cbg0 6y ago
More likely than not some manager decided all those extra nines weren't really necessary.
- nikanj 6y ago
  The manager probably put five 9s onto the spec sheet, and provided absolutely no resources. You’ll never guess who’s going to get the heat for this, the manager or the poor person who didn’t follow the spec.
  - StavrosK 6y ago
    A reliability guarantee I like to call "nine fives".
    - hinkley 6y ago
      There was an article about “five 8’s” during the Trough of Disillusionment for the original hype cycle.
- deleted 6y ago
  [deleted]
- jiofih 6y ago
  I don’t know why you’re being downvoted since you can clearly see echoes of this manager in the comments above...
  - dheera 6y ago
    Too many managers, not enough engineers on HN. As an engineer I can definitely attest to managers saying "99.9% is enough"
StreamBright 6y ago
Or maybe an engineering mistake? I do not think you can claim anything without knowing the details. AWS lost customer data too, I would not try to assign blame without reading the post mortem.
- Sharlin 6y ago
  According to TFA (which apparently nobody reads) it most certainly was an engineering mistake. Files in long-term storage were accidentally subject to the same code that deletes files in short-term (30 day) storage.
  - akerro 6y ago
    It's sound similar to "junior dev had access to prod database" scenario which really isn't engineering mistake but process/administration.
    - imdsm 6y ago
      It seems more likely that there was some process that performed a "delete older than 30 days" on the long term storage. Likely untested code. That's what the article says anyway, and we all know that the public message is always the best spin on a scenario.
    - hinkley 6y ago
      I knew a guy who used a different color scheme for terminal sessions connected to production servers, so he didn’t do something in the wrong window. A guy. One. How many people have I met with shell access to production servers?
      - akerro 6y ago
        I have all hosts share the same .zshrc and config with yadm, so all terminals look the same and share keyboard shortcuts. My oh-my-zsh with powerlevel10k is configured in a similar way: when I'm connected by ssh to a host it shows hostname@domain or hostname@ip on RIGHT side of a screen. It's not much, but it's easily visible, sometimes even annoying, but at least I never deleted wrong docker images from dev :)
numpad0 6y ago
My two cents is going to be that the ad tech MacBook deleted the old S3 bucket for mockup but it wasn’t a mockup at that point
- stingraycharles 6y ago
  What do you mean? The article says it was a software bug that accidentally deleted both short-term and long-term storage, while it should have been only short-term.
- mosselman 6y ago
  Are you trying to discredit using MacBooks in tech or something?
nabla9 6y ago
This can be just calculated risk. Incredible damage for some users can be negligible damage to the company. In this case just 10TB of data was lost, so maybe thousands of users for the long term storage option. Losing customer data has potential for big reputation or brand damage, but surprisingly often the damage is relatively small. Putting in too much effort when potential damage to business is minimal may not be worth of the cost.
- ChuckNorris89 6y ago
  This. Outside of FAANG, Gov and finance, very few consumer facing companies have the budgets to engineer redundancies.
  - nix23 6y ago
    >budgets to engineer redundancies You dont need that, but a backup ;)
    - virgilp 6y ago
      They obviously do have _some_ backups (since were able to restore all videos and all photos - but not all photos at the original resolution). In the end it's all a tradeoff... ignoring cost, we all know what to do to get pretty safe storage. But how much are you really willing to pay, per month, for that safety?
      - nix23 6y ago
        >but not all photos at the original resolution Thats why you backup the original data and NOT the downscaled etc one. >But how much are you really willing to pay, per month, for that safety? You have to backup ALL your customers data, otherwise DONT go into file-hosting business.
        ChuckNorris89 6y ago
        With your argument, Google photos should also not be in business since they also don't back up your original resolution unless you pay.
fheld 6y ago
"Is image.canon free to use? Yes, it is free to use." from: image.canon/st/en/faq.html so as of now there is no recurring revenue added as of now
- prepend 6y ago
  Selling customer data is a recurring stream. There’s likely also something in there about selling access to photo sets for training purposes.
- jacquesm 6y ago
  And there won't be at this rate. Besides, it is now a net negative for their users. 'The cloud is not a backup' should be a mantra that everybody that uses the cloud for their work and personal data be familiar with.
  - traveler01 6y ago
    I use the Cloud to store all my work and it's my way of backup. I trust Google's capacity to save my files. But, this being said, you SHOULD have other ways of backing up your data, like another HDD, storing locally in your PC too. Being too careful is never enough.
    - tinco 6y ago
      Sure, Google can save your files. But will it? and will you never lose access to Google? The problem with these big cloud providers is there are employees and robots with a big "disable account button", but there's no one you can call to talk about reenabling your account. I don't know what the odds are that Google is going to hit that "disable account" button on your account, it's probably incredible small, but if it would ever happen they might have permanently deleted all your stuff before you get far enough in their customer support system to have them reinstate your account. If my livelihood depended on Google storing my files, I'd store a backup of my data somewhere that I can access in person if push comes to shove. As a matter of fact it does, and I do.
hoorayimhelping 6y ago
How do you know this? Because this explanation: >Canon has announced the results of its investigation into the loss of image data on the image.canon cloud platform. According to Canon, when the company switched over to a new version of the image.canon software on 30 July, the code to control the short-term storage operated on both the short-term storage and the long-term storage functions, causing the loss of some images stored for more than 30 days. seems pretty banal. Updating a service and forgetting to update the backups is not an uncommon thing, and quite a few software companies have published post mortems saying they did exactly that.

cbg0 6y ago

More likely than not some manager decided all those extra nines weren't really necessary.

nikanj 6y ago
The manager probably put five 9s onto the spec sheet, and provided absolutely no resources. You’ll never guess who’s going to get the heat for this, the manager or the poor person who didn’t follow the spec.
- StavrosK 6y ago
  A reliability guarantee I like to call "nine fives".
  - hinkley 6y ago
    There was an article about “five 8’s” during the Trough of Disillusionment for the original hype cycle.
deleted 6y ago
[deleted]
jiofih 6y ago
I don’t know why you’re being downvoted since you can clearly see echoes of this manager in the comments above...
- dheera 6y ago
  Too many managers, not enough engineers on HN. As an engineer I can definitely attest to managers saying "99.9% is enough"

StreamBright 6y ago

Or maybe an engineering mistake? I do not think you can claim anything without knowing the details. AWS lost customer data too, I would not try to assign blame without reading the post mortem.

Sharlin 6y ago
According to TFA (which apparently nobody reads) it most certainly was an engineering mistake. Files in long-term storage were accidentally subject to the same code that deletes files in short-term (30 day) storage.
- akerro 6y ago
  It's sound similar to "junior dev had access to prod database" scenario which really isn't engineering mistake but process/administration.
  - imdsm 6y ago
    It seems more likely that there was some process that performed a "delete older than 30 days" on the long term storage. Likely untested code. That's what the article says anyway, and we all know that the public message is always the best spin on a scenario.
  - hinkley 6y ago
    I knew a guy who used a different color scheme for terminal sessions connected to production servers, so he didn’t do something in the wrong window. A guy. One. How many people have I met with shell access to production servers?
    - akerro 6y ago
      I have all hosts share the same .zshrc and config with yadm, so all terminals look the same and share keyboard shortcuts. My oh-my-zsh with powerlevel10k is configured in a similar way: when I'm connected by ssh to a host it shows hostname@domain or hostname@ip on RIGHT side of a screen. It's not much, but it's easily visible, sometimes even annoying, but at least I never deleted wrong docker images from dev :)

numpad0 6y ago

My two cents is going to be that the ad tech MacBook deleted the old S3 bucket for mockup but it wasn’t a mockup at that point

stingraycharles 6y ago
What do you mean? The article says it was a software bug that accidentally deleted both short-term and long-term storage, while it should have been only short-term.
mosselman 6y ago
Are you trying to discredit using MacBooks in tech or something?

nabla9 6y ago

This can be just calculated risk. Incredible damage for some users can be negligible damage to the company. In this case just 10TB of data was lost, so maybe thousands of users for the long term storage option. Losing customer data has potential for big reputation or brand damage, but surprisingly often the damage is relatively small. Putting in too much effort when potential damage to business is minimal may not be worth of the cost.

ChuckNorris89 6y ago
This. Outside of FAANG, Gov and finance, very few consumer facing companies have the budgets to engineer redundancies.
- nix23 6y ago
  >budgets to engineer redundancies You dont need that, but a backup ;)
  - virgilp 6y ago
    They obviously do have _some_ backups (since were able to restore all videos and all photos - but not all photos at the original resolution). In the end it's all a tradeoff... ignoring cost, we all know what to do to get pretty safe storage. But how much are you really willing to pay, per month, for that safety?
    - nix23 6y ago
      >but not all photos at the original resolution Thats why you backup the original data and NOT the downscaled etc one. >But how much are you really willing to pay, per month, for that safety? You have to backup ALL your customers data, otherwise DONT go into file-hosting business.
      - ChuckNorris89 6y ago
        With your argument, Google photos should also not be in business since they also don't back up your original resolution unless you pay.

fheld 6y ago

"Is image.canon free to use? Yes, it is free to use." from: image.canon/st/en/faq.html so as of now there is no recurring revenue added as of now

prepend 6y ago
Selling customer data is a recurring stream. There’s likely also something in there about selling access to photo sets for training purposes.
jacquesm 6y ago
And there won't be at this rate. Besides, it is now a net negative for their users. 'The cloud is not a backup' should be a mantra that everybody that uses the cloud for their work and personal data be familiar with.
- traveler01 6y ago
  I use the Cloud to store all my work and it's my way of backup. I trust Google's capacity to save my files. But, this being said, you SHOULD have other ways of backing up your data, like another HDD, storing locally in your PC too. Being too careful is never enough.
  - tinco 6y ago
    Sure, Google can save your files. But will it? and will you never lose access to Google? The problem with these big cloud providers is there are employees and robots with a big "disable account button", but there's no one you can call to talk about reenabling your account. I don't know what the odds are that Google is going to hit that "disable account" button on your account, it's probably incredible small, but if it would ever happen they might have permanently deleted all your stuff before you get far enough in their customer support system to have them reinstate your account. If my livelihood depended on Google storing my files, I'd store a backup of my data somewhere that I can access in person if push comes to shove. As a matter of fact it does, and I do.

hoorayimhelping 6y ago

How do you know this? Because this explanation: >Canon has announced the results of its investigation into the loss of image data on the image.canon cloud platform. According to Canon, when the company switched over to a new version of the image.canon software on 30 July, the code to control the short-term storage operated on both the short-term storage and the long-term storage functions, causing the loss of some images stored for more than 30 days. seems pretty banal. Updating a service and forgetting to update the backups is not an uncommon thing, and quite a few software companies have published post mortems saying they did exactly that.