5 ms·

I keep hearing about this coming up, yet no one ask talked about why the data is irretrievable. What it implies is that the internal tool(s) that the support s

by geuis 16y ago

I keep hearing about this coming up, yet no one ask talked about why the data is irretrievable. What it implies is that the internal tool(s) that the support staff uses was designed to do deletion instead of status changes. I imagine they have a web interface that fires off some backend script which a) runs sql deletes on an account ID and all tables and b) fires off scripts that physically deletes all photos across all servers. Taking those implications in hand, it points to an incredibly poorly designed system for account management. Some years ago, there was an internal planning meeting where a decision was made to do this. Some people objected and were overruled. The ramifications of that project manager are still sticking it to people to this day. I have a hard time believing that Flickr would have purposely engineered their systems in the manner I've described. Not impossible, but unlikely. So I think its more likely that the data is still there, but that customer service has no method to resurrect it. Its probably a mix of internal policies and the lack of devoting a couple of engineers to fix the problem that is causing all of this.

potatolicious 16y ago
> "some backend script which a) runs sql deletes on an account ID and all tables and b) fires off scripts that physically deletes all photos across all servers." Neither should happen. You never actually delete rows from your tables. You mark them as deleted, sure, but the data needs to be able to come back. Space is cheap, there are simply very few cases where outright deletion from a DB table is warranted. I have a hard time believing Flickr would make a mistake like this. I agree with you - odds are if they actually let an engineer loose on this problem it'll be licked with a DB backfill in 20 minutes.
- abscondment 16y ago
  I bet that in cases of copyright infringement a logical delete isn't legally sufficient. Sure, you can leave the database pointers in place... but you're going to need to actually remove the infringing content. In that case, at least, a true delete tool would be needed. Like any Internet discussion about copyrights or other people's backend scripts, this is all purely speculative.
  - InclinedPlane 16y ago
    I don't buy it. Simply making infringing data unavailable has to be legally sufficient, except in cases involving removing secret or confidential data (which would likely require a court case) there's no harm in having the data merely inaccessible. Moreover, it simply must be the case that this is how it works in practice. Consider how deep this rabbit hole goes. If expunging any copies of copyrighted data were strictly necessary, then: What of data existing on backup media? Should those be re-hydrated, the target data expunged, and then recreated in abridged form? What of data existing in various content caches (such as memcached)? Should all of those systems be flushed of any possible contaminant? What of the actual data on physical media? Deleting a file on any modern form of media does not expunge the data, it merely unlinks the location of the data on disk from the file system directory. It might be necessary, depending on filesystem and drive type, to scan all of the unused sectors on an entire disk to find out if the target data or any part of it existed. As you see, quite quickly you get into absurdities. Nobody goes to that much trouble to delete merely copyright infringing data.
  - enjo 16y ago
    I can't speak for Flickr, but I can tell you that one of their larger competitors simply does a logical delete. That seems to take care of any legal obligation. I should note that this not true in the case of child porn. Once the feds give them the signal that is completely purged.
  - phaylon 16y ago
    Also, depending on the country there might be specific privacy laws that matter.
- T-hawk 16y ago
  > You never actually delete rows from your tables. You mark them as deleted, sure, but the data needs to be able to come back. Space is cheap, there are simply very few cases where outright deletion from a DB table is warranted. This comes with a cost, and it's not a small one. Every single query that ever touches this table, in every single piece of code owned by the company, written or maintained by every single developer (consultant or transient or offshore, none of which ever got training sessions on this requirement) needs to be enhanced to respect "where isDeleted <> true". It's well understandable if the expected cost to recover after the fact from an occasional issue is smaller than the cost of implementing and maintaining that in the first place.
  - derobert 16y ago
    No, that's what views are for. A sane approach would be to do all your table access via views, and the view definitions have the 'where is_deleted = 0' or whatever. Then your database handles it. Updates to the view work as well, in pretty much every database system, including MySQL even.
    - T-hawk 16y ago
      That just shifts the problem. Every piece of code and every developer now needs to know to hit that view rather than directly hit the table. Except the code that actually deals with deleting and undeleting which has to hit the table directly (or another view or sp.) And you still have the original problem in writing and managing the views themselves - every developer writing or maintaining one of those views needs to know to respect "where is_deleted = 0". Using views adds a layer of abstraction. That's often a good idea and a step in the right direction, but does come at the cost of increasing the overall complexity of the system. It's an incremental step to the positive, but no revolutionary solution.
- smackfu 16y ago
  You never delete, and then people bitch that they can't really leave the service. See: Facebook.
rtghnyhjm 16y ago
It's an engineering decision - Flikr has to host a large amount of data for very little cost, since most users aren't paying. Space isn't cheap by the Petabyte! The design for a system where you had multiple redundant copies of all the data and could roll back any changes to the data set is different - and would cost a lot more.
- sdizdar 16y ago
  Here is what I don't understand. They have multiple redundant copies in case of unlikely scenario of disaster, but they have no copies or protection in case of quite likely scenario of operator error. This seems backward to me. EDIT: I don't believe that Yahoo! Flickr does not have a way to recover these things. Or I'm completely wrong.
  - InclinedPlane 16y ago
    I think it's likely that Flickr has a way to recover the data but the front line support personnel don't, and they don't want to have some dev spend a few hours to fix a single customer's problem, even if it was Flickr's mistake.
  - hrktb 16y ago
    Reverting a full db should be straightforward (stop the db server and revert a dump). For a single account it's more of a pita. Retrieving all the datas associated with one account, put it in a format readable from the production environment and recommit all the changes while dealing with the potential errors and inconstencies seems quite a huge deal. Messing unrelated data in the process would also be a nightmare. Of course, not deleting anything from the start wouod be the way.
    - gaius 16y ago
      It's trivial in Oracle, just do a Flashback Query. You would just need to support that in whatever front end interface the customer support people use.
mmaunder 16y ago
"..but that customer service has no method to resurrect it". Having seen this kind of effect from a brain drain in the past, I think you're exactly right. Software needs love to survive and this is what it looks like when software and a company becomes unloved.
theDoug 16y ago
Indeed. I'd have sympathy for Flickr if this story didn't come up almost monthly in the last few years. Between this and the account deletions of those who spoke loudly over Flickr's other account deletions, more and more Flickr is becoming a hostile system. The lack of a retrieval system could be from a lack of Yahoo wishing to invest in upgrades to the platform in general. The last few years have seen few changes for users beyond geotagging and tagging persons, and competitors (free and otherwise) have stepped far ahead in terms of personalisation and galleries. I'm very curious to see if Flickr turns into another 'Delicious' situation in a couple years.
- InclinedPlane 16y ago
  I think that danger is very real. I used to be a Flickr Pro member but I've been taking fewer pictures the last few years (for several reasons) and can't justify the Pro account anymore. However, the side affect of that is that now some of my older pictures are less accessible and my old photo sets aren't browseable. At least to me this seems like a bad business plan for Flickr, especially amidst an abundance of rabid competition.
  - dtc 16y ago
    Could you point me to some of the rabid competition that might offer a similar pricing model?
    - InclinedPlane 16y ago
      Flickr's main competition isn't with services that have feature parity with them, it's more with the growing list of free image sharing sites. Imgur, twitpic, facebook, picassa web, even dropbox. Flickr's value add is rapidly becoming smaller and smaller, which could very easily undermine their business model.
      - xyzzyb 16y ago
        Exactly. I used to have a flickr account but since dropbox/facebook came along if I want to share anything it goes in dropbox (public) or facebook (public, but it's easy for my family to see the pics).
citadrianne 16y ago
It sounds like Flickr is throwing engineers at the problem now--they responded with a statement which is in the updated post: http://www.observer.com/2011/tech/flickr-accidentally-deletes-users-4000-photos-and-cant-get-them-back http://www.observer.com/2011/tech/flickr-accidentally-delete...
callmeed 16y ago
Are you saying that a freemium service should never delete data—or that a customer service rep shouldn't be able to do it? I can see why Flickr might engineer things this way. Most of their users don't pay (granted, this guy was a pro user). Deleting images when an account is deleted saves money.
random42 16y ago
How should the deletion system be designed?

by geuis 16y ago

potatolicious 16y ago

> "some backend script which a) runs sql deletes on an account ID and all tables and b) fires off scripts that physically deletes all photos across all servers." Neither should happen. You never actually delete rows from your tables. You mark them as deleted, sure, but the data needs to be able to come back. Space is cheap, there are simply very few cases where outright deletion from a DB table is warranted. I have a hard time believing Flickr would make a mistake like this. I agree with you - odds are if they actually let an engineer loose on this problem it'll be licked with a DB backfill in 20 minutes.

abscondment 16y ago
I bet that in cases of copyright infringement a logical delete isn't legally sufficient. Sure, you can leave the database pointers in place... but you're going to need to actually remove the infringing content. In that case, at least, a true delete tool would be needed. Like any Internet discussion about copyrights or other people's backend scripts, this is all purely speculative.
- InclinedPlane 16y ago
  I don't buy it. Simply making infringing data unavailable has to be legally sufficient, except in cases involving removing secret or confidential data (which would likely require a court case) there's no harm in having the data merely inaccessible. Moreover, it simply must be the case that this is how it works in practice. Consider how deep this rabbit hole goes. If expunging any copies of copyrighted data were strictly necessary, then: What of data existing on backup media? Should those be re-hydrated, the target data expunged, and then recreated in abridged form? What of data existing in various content caches (such as memcached)? Should all of those systems be flushed of any possible contaminant? What of the actual data on physical media? Deleting a file on any modern form of media does not expunge the data, it merely unlinks the location of the data on disk from the file system directory. It might be necessary, depending on filesystem and drive type, to scan all of the unused sectors on an entire disk to find out if the target data or any part of it existed. As you see, quite quickly you get into absurdities. Nobody goes to that much trouble to delete merely copyright infringing data.
- enjo 16y ago
  I can't speak for Flickr, but I can tell you that one of their larger competitors simply does a logical delete. That seems to take care of any legal obligation. I should note that this not true in the case of child porn. Once the feds give them the signal that is completely purged.
- phaylon 16y ago
  Also, depending on the country there might be specific privacy laws that matter.
T-hawk 16y ago
> You never actually delete rows from your tables. You mark them as deleted, sure, but the data needs to be able to come back. Space is cheap, there are simply very few cases where outright deletion from a DB table is warranted. This comes with a cost, and it's not a small one. Every single query that ever touches this table, in every single piece of code owned by the company, written or maintained by every single developer (consultant or transient or offshore, none of which ever got training sessions on this requirement) needs to be enhanced to respect "where isDeleted <> true". It's well understandable if the expected cost to recover after the fact from an occasional issue is smaller than the cost of implementing and maintaining that in the first place.
- derobert 16y ago
  No, that's what views are for. A sane approach would be to do all your table access via views, and the view definitions have the 'where is_deleted = 0' or whatever. Then your database handles it. Updates to the view work as well, in pretty much every database system, including MySQL even.
  - T-hawk 16y ago
    That just shifts the problem. Every piece of code and every developer now needs to know to hit that view rather than directly hit the table. Except the code that actually deals with deleting and undeleting which has to hit the table directly (or another view or sp.) And you still have the original problem in writing and managing the views themselves - every developer writing or maintaining one of those views needs to know to respect "where is_deleted = 0". Using views adds a layer of abstraction. That's often a good idea and a step in the right direction, but does come at the cost of increasing the overall complexity of the system. It's an incremental step to the positive, but no revolutionary solution.
smackfu 16y ago
You never delete, and then people bitch that they can't really leave the service. See: Facebook.

rtghnyhjm 16y ago

It's an engineering decision - Flikr has to host a large amount of data for very little cost, since most users aren't paying. Space isn't cheap by the Petabyte! The design for a system where you had multiple redundant copies of all the data and could roll back any changes to the data set is different - and would cost a lot more.

sdizdar 16y ago
Here is what I don't understand. They have multiple redundant copies in case of unlikely scenario of disaster, but they have no copies or protection in case of quite likely scenario of operator error. This seems backward to me. EDIT: I don't believe that Yahoo! Flickr does not have a way to recover these things. Or I'm completely wrong.
- InclinedPlane 16y ago
  I think it's likely that Flickr has a way to recover the data but the front line support personnel don't, and they don't want to have some dev spend a few hours to fix a single customer's problem, even if it was Flickr's mistake.
- hrktb 16y ago
  Reverting a full db should be straightforward (stop the db server and revert a dump). For a single account it's more of a pita. Retrieving all the datas associated with one account, put it in a format readable from the production environment and recommit all the changes while dealing with the potential errors and inconstencies seems quite a huge deal. Messing unrelated data in the process would also be a nightmare. Of course, not deleting anything from the start wouod be the way.
  - gaius 16y ago
    It's trivial in Oracle, just do a Flashback Query. You would just need to support that in whatever front end interface the customer support people use.

mmaunder 16y ago

"..but that customer service has no method to resurrect it". Having seen this kind of effect from a brain drain in the past, I think you're exactly right. Software needs love to survive and this is what it looks like when software and a company becomes unloved.

theDoug 16y ago

Indeed. I'd have sympathy for Flickr if this story didn't come up almost monthly in the last few years. Between this and the account deletions of those who spoke loudly over Flickr's other account deletions, more and more Flickr is becoming a hostile system. The lack of a retrieval system could be from a lack of Yahoo wishing to invest in upgrades to the platform in general. The last few years have seen few changes for users beyond geotagging and tagging persons, and competitors (free and otherwise) have stepped far ahead in terms of personalisation and galleries. I'm very curious to see if Flickr turns into another 'Delicious' situation in a couple years.

InclinedPlane 16y ago
I think that danger is very real. I used to be a Flickr Pro member but I've been taking fewer pictures the last few years (for several reasons) and can't justify the Pro account anymore. However, the side affect of that is that now some of my older pictures are less accessible and my old photo sets aren't browseable. At least to me this seems like a bad business plan for Flickr, especially amidst an abundance of rabid competition.
- dtc 16y ago
  Could you point me to some of the rabid competition that might offer a similar pricing model?
  - InclinedPlane 16y ago
    Flickr's main competition isn't with services that have feature parity with them, it's more with the growing list of free image sharing sites. Imgur, twitpic, facebook, picassa web, even dropbox. Flickr's value add is rapidly becoming smaller and smaller, which could very easily undermine their business model.
    - xyzzyb 16y ago
      Exactly. I used to have a flickr account but since dropbox/facebook came along if I want to share anything it goes in dropbox (public) or facebook (public, but it's easy for my family to see the pics).

citadrianne 16y ago

It sounds like Flickr is throwing engineers at the problem now--they responded with a statement which is in the updated post: http://www.observer.com/2011/tech/flickr-accidentally-deletes-users-4000-photos-and-cant-get-them-back http://www.observer.com/2011/tech/flickr-accidentally-delete...

callmeed 16y ago

Are you saying that a freemium service should never delete data—or that a customer service rep shouldn't be able to do it? I can see why Flickr might engineer things this way. Most of their users don't pay (granted, this guy was a pro user). Deleting images when an account is deleted saves money.

random42 16y ago

How should the deletion system be designed?