21 ms·

Facebook's e-mail debacle: One 'bug' fix, but rollback impossible

175 points by wyclif 14y ago

51Cards 14y ago
Sometimes it seems to me that FB doesn't have an impact assessment process. This is a very important data change you're making to 900-ish million records on a system that is inter-woven with and syncing to... everything. I understand bugs happening, but really, the cascade effect of this email change had to have been foreseen by someone... no? Bueller? Bueller? I really don't mean to trash FB as a lot of good people work there. However... when I read about something like this, and then I see a company like Google figuring out a time smear function to seamlessly account for a leap second in their servers to avoid cascading system changes... it's hard to feel the two companies are functioning on the same level. (I know, a mis-balanced and irrelevant comparison, but it's just for a 'gut feeling' sense)
- rhizome 14y ago
  They knew what they were doing. Not having a complete rollback was intentional.
  - eli 14y ago
    I believe Facebook is capable of doing sneaky things to advance their business goals, but Hanlon's razor [1] comes to mind here. [1] http://www.jargon.net/jargonfile/h/HanlonsRazor.html http://www.jargon.net/jargonfile/h/HanlonsRazor.html
    - rhizome 14y ago
      I know what Hanlon's razor is, and I can guarantee you FB Engineering knows what a rollback is and how to know whether you need one or not. I'm not sure where you're getting the idea that they're that stupid.
      - keithpeter 14y ago
        Would applying such a 'rollback' involve replacing facebook.com email addresses with the original external domain addresses in the contacts list on the devices affected by the write-over issue? If so: Facebook have taken private data and placed it in a central location If not: it isn't a rollback that I understand Disclaimer: I'm not a programmer
        rhizome 14y ago
        Yes, a (hypothetical) "complete rollback" would involve replacing removed data. Sure, data privacy would be a concern, but it's beside the point that rolling this back would necessitate un-overwriting.
      - Cloven 14y ago
        how about 'the evidence on display in front of us'?
    - Devilboy 14y ago
      Hanlon's razor is not a law of nature.
      - eli 14y ago
        Neither is the opposite.
  - j_s 14y ago
    Only Violet Blue has said that they don't have a complete rollback as best I can tell from the CNET article. I would guess the only thing they can't undo would be the time that has passed. If any data Facebook ever had on any of their users (such as their primary email address [edit: at any point in time]) was ever lost, they wouldn't have as much to sell to their customers! Read me blathering on further about this: http://news.ycombinator.com/item?id=4192299 http://news.ycombinator.com/item?id=4192299
    - rhizome 14y ago
      Yeah, I thought about leaving that second sentence off. Point remains.
      - j_s 14y ago
        Thanks for sharing your perspective. Just to make sure we're on the same page, you are arguing that FB engineering was 100% aware of the consequences of automatically switching primary email address to @facebook.com for users who had setup that functionality, including overwriting email addresses in phone contact lists and capturing email unintentionally sent via those modified contact lists? This is certainly within the realm of possibility and indicates that you hold the team's technical ability in high regard. If it is indeed true, I would wonder what percentage of the folks involved in the implementation actually thought about it and how high/wide the issue was escalated. I have certainly been in situations where after-the-fact I found out that something I had been a part of shipped with known bugs/features I didn't know about that I would have raised a lot more concern over.
        rhizome 14y ago
        My line of thought is that they knew it would change phonebooks and that they weren't putting whatever resources necessary into creating a process for rollback. Given the nature of Facebook and it's technology, I'm not even sure there is anything they could have done, but it's de rigeur to know if this is the case. It's not even about high regard, but by minimum standards. If you know you can't rollback you go full steam ahead. If you've never worked someplace where they have dedicated staff for source control, that fact alone might give you an idea at the level of expertise involved here. It's nobody's first time at the rodeo, and setting the sync API on the change (as described below) wasn't just slipped in by an intern. You finding out after-the-fact and still having an opinion about bugs shipping in code you contributed to only reinforces my point: someone made the decision to go ahead anyway.
- tedunangst 14y ago
  google buzz?
- dasil003 14y ago
  > and then I see a company like Google figuring out a time smear function to seamlessly account for a leap second in their servers to avoid cascading system changes... it's hard to feel the two companies are functioning on the same level. Sorta makes you wish Google would stop chasing Facebook and just do their thing.
  - bgilroy26 14y ago
    Connecting site recommendations to user profiles has the potential to produce better, unspammable Google search results. Google cannot allow other companies to get ahead of them on social search and remain competitive in their most important vertical.
ChuckMcM 14y ago
Ouch. Talk about not thinking it through. Relating back to a common topic here, one of the things that distinguishes between senior engineers and not senior engineers in my experience is the ability to visualize both secondary and tertiary consequences of a particular change to the system. They stop and say "Ok, we change this, that means ..." and then walk forward past the change in their mind exploring the ramifications. In order to do that they have to have a broad understanding of the system involved from both a technical and operational perspective. They come back with "Oh if we do that then we'll get this effect and that would be bad, how can we prevent that from happening?" and if the answer was "We can't." then they spike the feature. One of the most frustrating things for me as an engineer at Sun early on was bouncing some whiz bang idea off Bill Joy only to get one of his one line zinger 'but that breaks X' type replies. Always annoying but often right as he would take what was proposed, extrapolate two, three, or a half dozen steps and then point out the now 'obvious' flaw.
- rhubarbquid 14y ago
  "move fast and break things"
  - Schwolop 14y ago
    Let's hope that changes to "hurry up and fix things".
    - markokocic 14y ago
      Please don't. If they really hurry up to fix things they could just create another breakage ;) Let them time to really think through it and fix the damage, instead of trying to do a "quick fix in production".
- jmspring 14y ago
  Facebook certainly has a reputation in engineering circles as following the "move fast and break things" model or touting "eventual consistency" or similar. That works for certain areas, as others have pointed out, but once you start manipulating customer data that they have set up, it crosses a line. I'm a strong believer in confining Windows to a fixed set of resources -- i.e., a VM with fixed memory and disk space. This incident, though I don't seem to be directly impacted (checked contacts, account settings, etc), Facebook has crossed that line into Windows equivalency for me. It needs to be confined where possible. Mobile app deleted on all my iOS devices and FB account disassociated on my WP7 device (most used for testing).
dmethvin 14y ago
"Move fast and break things" should still require that you think through what you're doing. The problem seems to be that the complexity of Facebook's system is beyond the understanding of anyone who reviews these changes.
- jerrya 14y ago
  "Move fast and break things" should refer to your servers and your data, not your customers and users devices and their data.
- tobtoh 14y ago
  I'm pretty sure that FB knew exactly what they were doing. They did think things through and decided the the short term PR loss was worth the benefits of gaining access to the email of 90% of their user base.
  - alan_cx 14y ago
    Exactly. Users are stupid. Not because they cant read minds and find hidden features, but because they take it. Its the same thing with things like internet censorship. Every one bleats on and on, and they just do it any way.
- dserodio 14y ago
  "Move fast and break things" applies to their website. They may break facebook.com all they want, but breaking MY contact list is unacceptable.
pixelcort 14y ago
Quoting: > Today Facebook also admitted that its API for contact sync on phone and device apps was set -- on which devices and systems we're not told -- to take an individual's most recently added e-mail address and overwrite their correct contact e-mail in everyone's address books with the new e-mail. and >> Since Facebook changed the email addresses on contacts, I've actually lost every single one of my email addresses including those for work. From this article it sounds like actual data loss occurred on some address books, where non-facebook.com email addresses were permanently deleted from contacts. For those people who don't have a backup of their address books, it might now be impossible to recover those lost email addresses. Isn't this more worrisome than any issues around how Facebook handles incoming messages?
- TeMPOraL 14y ago
  It is. Everything else seemed to me to be whining about how someone who saw you on Facebook will maybe not be able to contact you because your default mail is a Facebook one (but why would anyone contact you by mail if Facebook messages are better? They both a) show notifications on FB and b) send notification e-mails to your real address). But this, this is real, and worrisome, and it has done real damage, and it will be damn hard if not impossible to reverse.
  - s_henry_paulson 14y ago
    So Facebook messages are better because instead of getting your e-mail, you now have to jump through hoops to read it? Got it.
    - TeMPOraL 14y ago
      Facebook messages are better when the person who wants to contact you found you on Facebook. They are better because they show in two places: your normal, original (not the Facebook one) mailbox AND in Facebook itself. So it is at least as good as sending an e-mail, and likely better.
      - 14y ago
reneherse 14y ago
More good reasons to never: a) Associate your phone number with your FB account. b) Download the FB app. The more you can keep Facebook sandboxed on your devices (and in your life) the better.
- JaggedJax 14y ago
  I've been using "Tinfoil for Facebook" on Android for this very reason. https://play.google.com/store/apps/details?id=com.danvelazco.fbwrapper https://play.google.com/store/apps/details?id=com.danvelazco...
- bsg75 14y ago
  c) Use FB
- Ogre 14y ago
  c) Associate your real email address with your FB account Facebook gets an email address I don't use anywhere else. No one has me listed in their contacts under the only email address facebook knows for me (well, I presume. It's not hard to guess the address I use for facebook if you know my real address, but there's no good reason anyone but facebook would use it). I've done that on general principle for more than a decade any time I have to give my email address to anyone other than an individual. This instance is the first time in a while where it's actually felt worthwhile. Originally I did it more for spam tracking. GMail's excellent spam filter fixed that problem for me years ago. But I never stopped doing it.
  - reneherse 14y ago
    Indeed, I do the same. And I'll also add: d) Never associate your real name with your FB account. I use my first name plus a family name that matches my middle initial. It's generally recognizable and accepted by friends. When I first set up my account this way (items A through D) it did seem a little "tinfoilly", and this last measure still does to some extent. But since Facebook doesn't actually let you delete your account, I'd rather err on the side of prudence. It will be nice to at least approximate a hole in their databases if their policies ever get too onerous.
deleted 14y ago
[deleted]
j_s 14y ago
Can anyone point to a Facebook representative saying that rollback is impossible? (I don't care at all what the CNET author says.) It seems like they should just put everything back to the email address they had before it was auto-updated to @ facebook.com. Then their apps would put them back to what they were on people's phones. Lastly, if they were truly going to fix the problem, they would pump through all the emails they got a chance to spy on, this time to the correct address, maybe with an attachment or sending the email as an attachment with their official apology. Can't rollback the delay and that impacts a lot, but email is theoretically unreliable anyway. Hurry up guys and gals... show me someone at Facebook actually cares about the product (their users) they are selling to their customers (their advertisers).
- jaylevitt 14y ago
  There's no reason to assume that the old addresses in people's address books were the same ones Facebook had as primaries. If my co-worker was in my phone as tom@patientslikeme.com, but signed up for Facebook as tom@gmail.com, and Facebook tries to roll back? Data loss still, and possibly even a privacy leak.
  - j_s 14y ago
    I was basing this on the fact that the facebook app/sync would have pushed tom@gmail.com into the contact info in the same way that it is now in trouble for overwriting with @facebook.com. Thanks for bringing this up as it would not work if they only push changes, and no change had been made facebook-side prior to the 'bug'. In that case, they should only rollback those that had been pushed out and the majority of email addresses would be lost. However, since users can go into the settings and restore their primary email address, it seems like it would be best for Facebook to encourage them to do so in the same way they've nagged me to switch to timeline forever.
- 51Cards 14y ago
  A couple issues pop to mind. My understanding was that this update took a week(+?) to roll out and may still be in progress. It could take them awhile to even start to roll back. Then if you do roll back what do you do about all the people that have changed their settings in the meantime? They have updated their email addresses, perhaps with different ones, so do those get over-written with the old data in all cases? I'm sure it could be figured out but how timely it would be is questionable... not to mention that FB may still be trying to salvage the bump in use for @Facebook.com they were aiming for in the beginning.
antidoh 14y ago
"Apple might be able to salvage the relationship with Facebook, but it's pretty clear that Facebook's relationship with users has hit the rocks." I very much doubt that. Statistically no one will leave FBook over this.
- pdonis 14y ago
  >>> Statistically no one will leave FBook over this. If that's really true, it's very unsettling. If this doesn't trip a significant number of users' "not worth it" detectors, what would?
  - alttab 14y ago
    Maybe we see that 38 dollar stock price now.
  - tomrod 14y ago
    And thus I feel compelled to start a discussion of the German zeitgeist in the mid 1930s, thus causing myself to bring about Godwin's law.
  - deleted 14y ago
    [deleted]
  - blhack 14y ago
    Um. Hi. I use facebook, this pissed me off. Then I changed my email address back to what I want it to be, and I have ceased caring. I will continue using facebook to coordinate bar meetups with my friends.
    - glesica 14y ago
      I think you kind of hit the nail on the head with your last sentence. You will continue to use the service for fairly meaningless tasks. It seems Facebook wants its users to use their service for more than that. So while this debacle may not make many people delete their accounts, Facebook may find its apparent goal more difficult to achieve going forward.
uptown 14y ago
I used to have an Android device, and let the thing sync my Facebook contacts into my phone. What I wound up with was a mix of people I'm actually friends with, and also a big collection on "Facebook friends" ... those people I'll never ever call, but knew from back-in-the-day and wanted to connect with on Facebook. I don't need them in my phone today because the extend of my correspondence with them will be what I consider "two-way read-only". They can look at my FB stuff. I can look at their FB stuff, but that's pretty much the extent of our relationship. I'm on an iPhone now, and know that iOS6 will give users the same option ... to link your device w/ Facebook ... sucking in all your "FB friends". I won't be enabling that feature, and while the software apparently indicates you can disable this part of the connectivity, I'm reluctant to even link it for fear that it'll immediately hose my contacts in a way that requires me to manually clean out that kid I went to middle-school with many years ago.
- deleted 14y ago
  [deleted]
leeoniya 14y ago
i uninstalled FB app about 8 months ago, never used sync on it anyways since i use FB more like a glorified twitter. been using mobile site since then (which is quite good), posting photos by sending an email to a special status address. so much more privacy. if they ever get rid of the mobile site to force me to install the app, byebye facebook on mobile.
acchow 14y ago
I can't stop my @facebook.com address from appearing on my timeline. The option shows up but clicking on it does nothing. Similarly, I can't make my @gmail.com address show up on my timeline. The mouse listener is also missing.
ricardobeat 14y ago
I'm still confused. Did the address book trashing affect only iOS 6 beta users?
- s_henry_paulson 14y ago
  It affected anyone on Android and Blackberry using contact sync, IOS6, and Windows Phone 8 http://www.bbc.com/news/technology-18687044 http://www.bbc.com/news/technology-18687044
AznHisoka 14y ago
There were so many obvious problems with associating a @facebook with everyone's account. As soon as I heard the news, I thought.. hmm.. someone could easily scrape 1 million facebook usernames, and be able to send 1 million emails to all of them using their facebook email address. And make it all uber-personal because they know their full name AND all their friend's names AND what they liked AND their hometown AND the college they went to, etc, etc,. That's a spammer's wet dream.
- entropy_ 14y ago
  The @facebook was always there and could always have been used to reach a person's FB messages via email. Also, most of that other information would probably be marked as only available to friends or friends of friends. So you wouldn't be able to scrape it. Don't get me wrong, this is a horrible move by facebook, but spam isn't the reason why.
bschlinker 14y ago
"Facebook is indeed intercepting messages -- the e-mail is clearly passing through Facebook's servers, but this seems to be where it stays, as neither sender nor receiver are getting a copy." Is this really appropriate use of the word "intercepting"? Typically intercepting carries a negative meaning -- like "the FBI is intercepting my postal mail and opening it before it reaches me". However, with their use of the word, every email I send is "intercepted" by multiple servers before it reaches its destination....
- j_s 14y ago
  Facebook is intercepting messages by replacing the destination address (which email clients often do not readily display [opting instead for full names, even photos, etc.]), and now claims to have done so inadvertently. Multiple servers relay messages before they reach their intended destination, yes, but that is by design. To continue stretching your analogy, I could claim that the post office x-ray/bomb scanners are 'intercepting' mail sent to me when in fact it is an understood part of the process to anyone who cares enough to research how things work. Someone moving my mailbox [edit: filling out a 'change of address'] and photocopying everything sent there is pretty clearly a reasonable use of the word 'intercepting'.
melvinmt 14y ago
Let's say 1% of all their users is fed up because of this and 0.1% actually closes their account. This leaves Zuckerberg with 98.9% control over e-mail traffic from 900+ million users. Well played, Zuck, well played.
- f0untain 14y ago
  Nope, it translates into 900 million bad karma..eventually.
- entropy_ 14y ago
  Couldn't this be construed as an abuse of a monopoly(antitrust)? They have a monopoly on social networking(there are alternatives, but they are very very small in comparison) and they are leveraging that to expand massively in another sphere, namely as email providers.
reidmain 14y ago
"Since Facebook changed the email addresses on contacts, I've actually lost every single one of my email addresses including those for work." -Complaint from iOS 6 user My suggestion, don't use the very first beta release of a operating system on anything important. Digital darwinism at work there.
- kalleboo 14y ago
  Indeed, Apple have a very explicit message on the download page for all iOS betas to not use it on a device that's not solely dedicated to testing.
- s_henry_paulson 14y ago
  What about all the Android and Blackberry users who had their addresses replaced? What's the lesson there?
  - reidmain 14y ago
    Oh that is Facebook screwing up. I'm not trying to pardon that. I just hate people who complain about stuff when they take a well defined risk and it backfires on them.
danso 14y ago
I like FB so I won't be doing one of those "hey everyone, look at me I'm quitting FB" acts. But I highly recommend that you don't put FB on your phone...it's amazing how even a five second glance to check for new notifications will ruin your train of thought. And also, not having it mess up your phone account is nice
dm8 14y ago
Why don't FB openly admit their mistake on their engineering blogs? At least they can salvage already messy situation
- Devilboy 14y ago
  They don't see it as a mistake.
  - rhizome 14y ago
    Or a mess.
SoftwareMaven 14y ago
Given FB is already having a hard time generating revenue from mobile clients, doing things that make mobile users afraid to use their service seems like a really bad idea.
- wutbrodo 14y ago
  Well, given that they can't monetize mobile usage and they can monetize desktop usage (if somewhat poorly), technically driving users back to desktop usage from mobile would be good for them.
andrewfelix 14y ago
I've been using a great work around to the numerous Facebook issues, including the lack of meaningful interaction with Friends and Family; I use a combination of email, telephone and physical interaction to maintain and form social connections.
luminaobscura 14y ago
this is why i have never sync'ed to fb or hotmail.
febeling 14y ago
There is the product life-cycle which has the phases: introduction, growth, maturity, and decline. The point is that facebook needs to understand that "Move fast, and break things" is an appropriate approach in the earlier phases, but it is quite incompatible with later phases, when people start to rely on a service.
dsirijus 14y ago
Of course, they do thoroughly test their new ad options, limiting them to region first, then measuring impact.
grey-area 14y ago
The problem here is not one of competence or failure to think through implications. Facebook know exactly what they are doing, and you can be sure the potential implications of these changes were debated at length within the company at a high level. They were taken precisely because of their implications for non-technical users, not in spite of them - FB has replaced users contact details because they want Facebook to be the only way most people can contact others online. Facebook have always wanted to replace the internet entirely with a system they control. Hence the walled garden approach with incessant prompts to signup to view content, the use of address book import to spread virally, the use of beacon to track and announce users activities, the use of like button js to track activity, the launch of Facebook apps and single sign-on to attempt to corral web developers within their ecosystem and make them dependent, the launch of a competitor to email/IM etc etc. This latest move with email is simply the latest in a long line of moves consistently attempting to steer users into spending all of their online time within the confines of Facebook, it is entirely consistent with their past behaviour and we should expect similar moves in future. So the arguments about how they could have done this differently, or whether Facebook has made some mistakes here are largely irrelevant - they will always push to own your data, email, photos, and digital life, because it's in their DNA, this is not a mistake, it's a pattern of behaviour - they can and will control your digital life if you choose to hand it over to them. Consider the quote from FB on the way their FB email works: Regarding the "email loss" this may actually just be confusion around the Messages Inbox: By default, messages from friends or friends of friends go into your Inbox. Everything else goes to your Other folder. All messages which do not come from FB contacts go into a folder which is effectively a junk folder - they want you to use their email for everything (hence the recent changes), they want your address book to contain nothing but FB contacts, they want you to forget email even existed, and they want you to encourage your friends to join FB so that you actually receive their messages. That won't work for all their users of course, but their hope is that they keep enough of the user base on board that they can dominate the internet and replace email with FB and the web with a network of sites where they control login and user info and all roads lead to FB. So replacing email with FB mail wherever they can, hiding other contacts and trashing user address books is not a mistake, it's part of a broader strategy, one which is not in the best interests of their users, but which they hope most will go along with anyway.
GBKS 14y ago
Now what happens if other sites and apps do the same? Will our address books change every day and contacts fill up with dozens of unused email addresses? Many of Facebook's growth techniques have been imitated by other startups so going into the territory of manipulating data that has always been off-limits is a big step. If another social site uses those techniques against to manipulate data on Facebook profiles, will Facebook change it's mind?
gitarr 14y ago
If lawyers are reading this, a question: Our governments are treating our emails more and more like real mail (aka. letters). Is there not a law prohibiting third party people or companies to mess with(fake) my mail addresses? Fact is, I do not want a facebook email address. And I have given no consent for my email address to be changed.
- rhizome 14y ago
  It's more like someone sending a change-of-address card to the post office to have your mail delivered to their Mailboxes Etc. store.
  - tfb 14y ago
    That's actually a really good analogy. And the GP is right. There should definitely be some laws to prohibit these kinds of things that Facebook is attempting (doing), now that electronic mail/communication is becoming standard.
    - rhizome 14y ago
      As a matter of fact, the (US) Postal Inspector considers this to be Identity Theft.
      - tfb 14y ago
        Interesting. I wonder why no legal action has been taken against Facebook for this recent fiasco... or has there?
crazygringo 14y ago
So this whole debacle has taught me that I have an "Other" folder, which I never saw before, and it turns out I have 20+ messages from people from the past few months which I didn't know about! I'm trying to find a way to get Facebook to send me an e-mail notification when I receive a message that goes into "Other", the same way it sends me an e-mail notification when I receive a message that goes into "Inbox". But I can't find any option for this anywhere. Does anyone know? Otherwise, how am I ever supposed to know I have a message from someone on Facebook who I'm not already friends with?!
- crazygringo 14y ago
  Wow, this is infuriating. According to Slate: So how can you make sure you find your lost laptop, appear on Israeli TV, and respond immediately to job offers? Unfortunately, the Facebook rep informed me, you can’t change your settings to get email notifications for your Other messages, the way you can for your main messages. Your best bet: Make checking the Other tab part of your daily Facebook routine. So since I never log into Facebook, but use it as a place to receive messages, which I find out about via e-mail... I am doomed to never find out about possibly important messages from people who are not already my Facebook friends. This is possibly the worst design decision I've ever come across on the Internet. Facebook has created a black hole for messages. Even when you log into Facebook, it doesn't show a number next to "Other", so you can't even find out if you have messages there without navigating to the "Other" page. I can't wrap my mind around how insanely stupid this is. WTF, Facebook?
  - Tyr42 14y ago
    Time to use selenium [1] to automate it and send you emails? [1] : http://seleniumhq.org/ http://seleniumhq.org/
mariuolo 14y ago
Why anyone would delegate handling of their address book to Facebook is beyond me.

175 points by wyclif 14y ago

Sometimes it seems to me that FB doesn't have an impact assessment process. This is a very important data change you're making to 900-ish million records on a system that is inter-woven with and syncing to... everything. I understand bugs happening, but really, the cascade effect of this email change had to have been foreseen by someone... no? Bueller? Bueller? I really don't mean to trash FB as a lot of good people work there. However... when I read about something like this, and then I see a company like Google figuring out a time smear function to seamlessly account for a leap second in their servers to avoid cascading system changes... it's hard to feel the two companies are functioning on the same level. (I know, a mis-balanced and irrelevant comparison, but it's just for a 'gut feeling' sense)

rhizome 14y ago
They knew what they were doing. Not having a complete rollback was intentional.
- eli 14y ago
  I believe Facebook is capable of doing sneaky things to advance their business goals, but Hanlon's razor [1] comes to mind here. [1] http://www.jargon.net/jargonfile/h/HanlonsRazor.html http://www.jargon.net/jargonfile/h/HanlonsRazor.html
  - rhizome 14y ago
    I know what Hanlon's razor is, and I can guarantee you FB Engineering knows what a rollback is and how to know whether you need one or not. I'm not sure where you're getting the idea that they're that stupid.
    - keithpeter 14y ago
      Would applying such a 'rollback' involve replacing facebook.com email addresses with the original external domain addresses in the contacts list on the devices affected by the write-over issue? If so: Facebook have taken private data and placed it in a central location If not: it isn't a rollback that I understand Disclaimer: I'm not a programmer
      - rhizome 14y ago
        Yes, a (hypothetical) "complete rollback" would involve replacing removed data. Sure, data privacy would be a concern, but it's beside the point that rolling this back would necessitate un-overwriting.
    - Cloven 14y ago
      how about 'the evidence on display in front of us'?
  - Devilboy 14y ago
    Hanlon's razor is not a law of nature.
    - eli 14y ago
      Neither is the opposite.
- j_s 14y ago
  Only Violet Blue has said that they don't have a complete rollback as best I can tell from the CNET article. I would guess the only thing they can't undo would be the time that has passed. If any data Facebook ever had on any of their users (such as their primary email address [edit: at any point in time]) was ever lost, they wouldn't have as much to sell to their customers! Read me blathering on further about this: http://news.ycombinator.com/item?id=4192299 http://news.ycombinator.com/item?id=4192299
  - rhizome 14y ago
    Yeah, I thought about leaving that second sentence off. Point remains.
    - j_s 14y ago
      Thanks for sharing your perspective. Just to make sure we're on the same page, you are arguing that FB engineering was 100% aware of the consequences of automatically switching primary email address to @facebook.com for users who had setup that functionality, including overwriting email addresses in phone contact lists and capturing email unintentionally sent via those modified contact lists? This is certainly within the realm of possibility and indicates that you hold the team's technical ability in high regard. If it is indeed true, I would wonder what percentage of the folks involved in the implementation actually thought about it and how high/wide the issue was escalated. I have certainly been in situations where after-the-fact I found out that something I had been a part of shipped with known bugs/features I didn't know about that I would have raised a lot more concern over.
      - rhizome 14y ago
        My line of thought is that they knew it would change phonebooks and that they weren't putting whatever resources necessary into creating a process for rollback. Given the nature of Facebook and it's technology, I'm not even sure there is anything they could have done, but it's de rigeur to know if this is the case. It's not even about high regard, but by minimum standards. If you know you can't rollback you go full steam ahead. If you've never worked someplace where they have dedicated staff for source control, that fact alone might give you an idea at the level of expertise involved here. It's nobody's first time at the rodeo, and setting the sync API on the change (as described below) wasn't just slipped in by an intern. You finding out after-the-fact and still having an opinion about bugs shipping in code you contributed to only reinforces my point: someone made the decision to go ahead anyway.
tedunangst 14y ago
google buzz?
dasil003 14y ago
> and then I see a company like Google figuring out a time smear function to seamlessly account for a leap second in their servers to avoid cascading system changes... it's hard to feel the two companies are functioning on the same level. Sorta makes you wish Google would stop chasing Facebook and just do their thing.
- bgilroy26 14y ago
  Connecting site recommendations to user profiles has the potential to produce better, unspammable Google search results. Google cannot allow other companies to get ahead of them on social search and remain competitive in their most important vertical.

ChuckMcM 14y ago

Ouch. Talk about not thinking it through. Relating back to a common topic here, one of the things that distinguishes between senior engineers and not senior engineers in my experience is the ability to visualize both secondary and tertiary consequences of a particular change to the system. They stop and say "Ok, we change this, that means ..." and then walk forward past the change in their mind exploring the ramifications. In order to do that they have to have a broad understanding of the system involved from both a technical and operational perspective. They come back with "Oh if we do that then we'll get this effect and that would be bad, how can we prevent that from happening?" and if the answer was "We can't." then they spike the feature. One of the most frustrating things for me as an engineer at Sun early on was bouncing some whiz bang idea off Bill Joy only to get one of his one line zinger 'but that breaks X' type replies. Always annoying but often right as he would take what was proposed, extrapolate two, three, or a half dozen steps and then point out the now 'obvious' flaw.

rhubarbquid 14y ago
"move fast and break things"
- Schwolop 14y ago
  Let's hope that changes to "hurry up and fix things".
  - markokocic 14y ago
    Please don't. If they really hurry up to fix things they could just create another breakage ;) Let them time to really think through it and fix the damage, instead of trying to do a "quick fix in production".
jmspring 14y ago
Facebook certainly has a reputation in engineering circles as following the "move fast and break things" model or touting "eventual consistency" or similar. That works for certain areas, as others have pointed out, but once you start manipulating customer data that they have set up, it crosses a line. I'm a strong believer in confining Windows to a fixed set of resources -- i.e., a VM with fixed memory and disk space. This incident, though I don't seem to be directly impacted (checked contacts, account settings, etc), Facebook has crossed that line into Windows equivalency for me. It needs to be confined where possible. Mobile app deleted on all my iOS devices and FB account disassociated on my WP7 device (most used for testing).

dmethvin 14y ago

"Move fast and break things" should still require that you think through what you're doing. The problem seems to be that the complexity of Facebook's system is beyond the understanding of anyone who reviews these changes.

jerrya 14y ago
"Move fast and break things" should refer to your servers and your data, not your customers and users devices and their data.
tobtoh 14y ago
I'm pretty sure that FB knew exactly what they were doing. They did think things through and decided the the short term PR loss was worth the benefits of gaining access to the email of 90% of their user base.
- alan_cx 14y ago
  Exactly. Users are stupid. Not because they cant read minds and find hidden features, but because they take it. Its the same thing with things like internet censorship. Every one bleats on and on, and they just do it any way.
dserodio 14y ago
"Move fast and break things" applies to their website. They may break facebook.com all they want, but breaking MY contact list is unacceptable.

pixelcort 14y ago

Quoting: > Today Facebook also admitted that its API for contact sync on phone and device apps was set -- on which devices and systems we're not told -- to take an individual's most recently added e-mail address and overwrite their correct contact e-mail in everyone's address books with the new e-mail. and >> Since Facebook changed the email addresses on contacts, I've actually lost every single one of my email addresses including those for work. From this article it sounds like actual data loss occurred on some address books, where non-facebook.com email addresses were permanently deleted from contacts. For those people who don't have a backup of their address books, it might now be impossible to recover those lost email addresses. Isn't this more worrisome than any issues around how Facebook handles incoming messages?

TeMPOraL 14y ago
It is. Everything else seemed to me to be whining about how someone who saw you on Facebook will maybe not be able to contact you because your default mail is a Facebook one (but why would anyone contact you by mail if Facebook messages are better? They both a) show notifications on FB and b) send notification e-mails to your real address). But this, this is real, and worrisome, and it has done real damage, and it will be damn hard if not impossible to reverse.
- s_henry_paulson 14y ago
  So Facebook messages are better because instead of getting your e-mail, you now have to jump through hoops to read it? Got it.
  - TeMPOraL 14y ago
    Facebook messages are better when the person who wants to contact you found you on Facebook. They are better because they show in two places: your normal, original (not the Facebook one) mailbox AND in Facebook itself. So it is at least as good as sending an e-mail, and likely better.
    - 14y ago

reneherse 14y ago

More good reasons to never: a) Associate your phone number with your FB account. b) Download the FB app. The more you can keep Facebook sandboxed on your devices (and in your life) the better.

JaggedJax 14y ago
I've been using "Tinfoil for Facebook" on Android for this very reason. https://play.google.com/store/apps/details?id=com.danvelazco.fbwrapper https://play.google.com/store/apps/details?id=com.danvelazco...
bsg75 14y ago
c) Use FB
Ogre 14y ago
c) Associate your real email address with your FB account Facebook gets an email address I don't use anywhere else. No one has me listed in their contacts under the only email address facebook knows for me (well, I presume. It's not hard to guess the address I use for facebook if you know my real address, but there's no good reason anyone but facebook would use it). I've done that on general principle for more than a decade any time I have to give my email address to anyone other than an individual. This instance is the first time in a while where it's actually felt worthwhile. Originally I did it more for spam tracking. GMail's excellent spam filter fixed that problem for me years ago. But I never stopped doing it.
- reneherse 14y ago
  Indeed, I do the same. And I'll also add: d) Never associate your real name with your FB account. I use my first name plus a family name that matches my middle initial. It's generally recognizable and accepted by friends. When I first set up my account this way (items A through D) it did seem a little "tinfoilly", and this last measure still does to some extent. But since Facebook doesn't actually let you delete your account, I'd rather err on the side of prudence. It will be nice to at least approximate a hole in their databases if their policies ever get too onerous.

deleted 14y ago

[deleted]

j_s 14y ago

Can anyone point to a Facebook representative saying that rollback is impossible? (I don't care at all what the CNET author says.) It seems like they should just put everything back to the email address they had before it was auto-updated to @ facebook.com. Then their apps would put them back to what they were on people's phones. Lastly, if they were truly going to fix the problem, they would pump through all the emails they got a chance to spy on, this time to the correct address, maybe with an attachment or sending the email as an attachment with their official apology. Can't rollback the delay and that impacts a lot, but email is theoretically unreliable anyway. Hurry up guys and gals... show me someone at Facebook actually cares about the product (their users) they are selling to their customers (their advertisers).

jaylevitt 14y ago
There's no reason to assume that the old addresses in people's address books were the same ones Facebook had as primaries. If my co-worker was in my phone as tom@patientslikeme.com, but signed up for Facebook as tom@gmail.com, and Facebook tries to roll back? Data loss still, and possibly even a privacy leak.
- j_s 14y ago
  I was basing this on the fact that the facebook app/sync would have pushed tom@gmail.com into the contact info in the same way that it is now in trouble for overwriting with @facebook.com. Thanks for bringing this up as it would not work if they only push changes, and no change had been made facebook-side prior to the 'bug'. In that case, they should only rollback those that had been pushed out and the majority of email addresses would be lost. However, since users can go into the settings and restore their primary email address, it seems like it would be best for Facebook to encourage them to do so in the same way they've nagged me to switch to timeline forever.
51Cards 14y ago
A couple issues pop to mind. My understanding was that this update took a week(+?) to roll out and may still be in progress. It could take them awhile to even start to roll back. Then if you do roll back what do you do about all the people that have changed their settings in the meantime? They have updated their email addresses, perhaps with different ones, so do those get over-written with the old data in all cases? I'm sure it could be figured out but how timely it would be is questionable... not to mention that FB may still be trying to salvage the bump in use for @Facebook.com they were aiming for in the beginning.

antidoh 14y ago

"Apple might be able to salvage the relationship with Facebook, but it's pretty clear that Facebook's relationship with users has hit the rocks." I very much doubt that. Statistically no one will leave FBook over this.

pdonis 14y ago
>>> Statistically no one will leave FBook over this. If that's really true, it's very unsettling. If this doesn't trip a significant number of users' "not worth it" detectors, what would?
- alttab 14y ago
  Maybe we see that 38 dollar stock price now.
- tomrod 14y ago
  And thus I feel compelled to start a discussion of the German zeitgeist in the mid 1930s, thus causing myself to bring about Godwin's law.
- deleted 14y ago
  [deleted]
- blhack 14y ago
  Um. Hi. I use facebook, this pissed me off. Then I changed my email address back to what I want it to be, and I have ceased caring. I will continue using facebook to coordinate bar meetups with my friends.
  - glesica 14y ago
    I think you kind of hit the nail on the head with your last sentence. You will continue to use the service for fairly meaningless tasks. It seems Facebook wants its users to use their service for more than that. So while this debacle may not make many people delete their accounts, Facebook may find its apparent goal more difficult to achieve going forward.

uptown 14y ago

I used to have an Android device, and let the thing sync my Facebook contacts into my phone. What I wound up with was a mix of people I'm actually friends with, and also a big collection on "Facebook friends" ... those people I'll never ever call, but knew from back-in-the-day and wanted to connect with on Facebook. I don't need them in my phone today because the extend of my correspondence with them will be what I consider "two-way read-only". They can look at my FB stuff. I can look at their FB stuff, but that's pretty much the extent of our relationship. I'm on an iPhone now, and know that iOS6 will give users the same option ... to link your device w/ Facebook ... sucking in all your "FB friends". I won't be enabling that feature, and while the software apparently indicates you can disable this part of the connectivity, I'm reluctant to even link it for fear that it'll immediately hose my contacts in a way that requires me to manually clean out that kid I went to middle-school with many years ago.

deleted 14y ago
[deleted]

leeoniya 14y ago

i uninstalled FB app about 8 months ago, never used sync on it anyways since i use FB more like a glorified twitter. been using mobile site since then (which is quite good), posting photos by sending an email to a special status address. so much more privacy. if they ever get rid of the mobile site to force me to install the app, byebye facebook on mobile.

acchow 14y ago

I can't stop my @facebook.com address from appearing on my timeline. The option shows up but clicking on it does nothing. Similarly, I can't make my @gmail.com address show up on my timeline. The mouse listener is also missing.

ricardobeat 14y ago

I'm still confused. Did the address book trashing affect only iOS 6 beta users?

s_henry_paulson 14y ago
It affected anyone on Android and Blackberry using contact sync, IOS6, and Windows Phone 8 http://www.bbc.com/news/technology-18687044 http://www.bbc.com/news/technology-18687044

AznHisoka 14y ago

There were so many obvious problems with associating a @facebook with everyone's account. As soon as I heard the news, I thought.. hmm.. someone could easily scrape 1 million facebook usernames, and be able to send 1 million emails to all of them using their facebook email address. And make it all uber-personal because they know their full name AND all their friend's names AND what they liked AND their hometown AND the college they went to, etc, etc,. That's a spammer's wet dream.

entropy_ 14y ago
The @facebook was always there and could always have been used to reach a person's FB messages via email. Also, most of that other information would probably be marked as only available to friends or friends of friends. So you wouldn't be able to scrape it. Don't get me wrong, this is a horrible move by facebook, but spam isn't the reason why.

bschlinker 14y ago

"Facebook is indeed intercepting messages -- the e-mail is clearly passing through Facebook's servers, but this seems to be where it stays, as neither sender nor receiver are getting a copy." Is this really appropriate use of the word "intercepting"? Typically intercepting carries a negative meaning -- like "the FBI is intercepting my postal mail and opening it before it reaches me". However, with their use of the word, every email I send is "intercepted" by multiple servers before it reaches its destination....

j_s 14y ago
Facebook is intercepting messages by replacing the destination address (which email clients often do not readily display [opting instead for full names, even photos, etc.]), and now claims to have done so inadvertently. Multiple servers relay messages before they reach their intended destination, yes, but that is by design. To continue stretching your analogy, I could claim that the post office x-ray/bomb scanners are 'intercepting' mail sent to me when in fact it is an understood part of the process to anyone who cares enough to research how things work. Someone moving my mailbox [edit: filling out a 'change of address'] and photocopying everything sent there is pretty clearly a reasonable use of the word 'intercepting'.

melvinmt 14y ago

Let's say 1% of all their users is fed up because of this and 0.1% actually closes their account. This leaves Zuckerberg with 98.9% control over e-mail traffic from 900+ million users. Well played, Zuck, well played.

f0untain 14y ago
Nope, it translates into 900 million bad karma..eventually.
entropy_ 14y ago
Couldn't this be construed as an abuse of a monopoly(antitrust)? They have a monopoly on social networking(there are alternatives, but they are very very small in comparison) and they are leveraging that to expand massively in another sphere, namely as email providers.

reidmain 14y ago

"Since Facebook changed the email addresses on contacts, I've actually lost every single one of my email addresses including those for work." -Complaint from iOS 6 user My suggestion, don't use the very first beta release of a operating system on anything important. Digital darwinism at work there.

kalleboo 14y ago
Indeed, Apple have a very explicit message on the download page for all iOS betas to not use it on a device that's not solely dedicated to testing.
s_henry_paulson 14y ago
What about all the Android and Blackberry users who had their addresses replaced? What's the lesson there?
- reidmain 14y ago
  Oh that is Facebook screwing up. I'm not trying to pardon that. I just hate people who complain about stuff when they take a well defined risk and it backfires on them.

danso 14y ago

I like FB so I won't be doing one of those "hey everyone, look at me I'm quitting FB" acts. But I highly recommend that you don't put FB on your phone...it's amazing how even a five second glance to check for new notifications will ruin your train of thought. And also, not having it mess up your phone account is nice

dm8 14y ago

Why don't FB openly admit their mistake on their engineering blogs? At least they can salvage already messy situation

Devilboy 14y ago
They don't see it as a mistake.
- rhizome 14y ago
  Or a mess.

SoftwareMaven 14y ago

Given FB is already having a hard time generating revenue from mobile clients, doing things that make mobile users afraid to use their service seems like a really bad idea.

wutbrodo 14y ago
Well, given that they can't monetize mobile usage and they can monetize desktop usage (if somewhat poorly), technically driving users back to desktop usage from mobile would be good for them.

andrewfelix 14y ago

I've been using a great work around to the numerous Facebook issues, including the lack of meaningful interaction with Friends and Family; I use a combination of email, telephone and physical interaction to maintain and form social connections.

luminaobscura 14y ago

this is why i have never sync'ed to fb or hotmail.

febeling 14y ago

There is the product life-cycle which has the phases: introduction, growth, maturity, and decline. The point is that facebook needs to understand that "Move fast, and break things" is an appropriate approach in the earlier phases, but it is quite incompatible with later phases, when people start to rely on a service.

dsirijus 14y ago

Of course, they do thoroughly test their new ad options, limiting them to region first, then measuring impact.

grey-area 14y ago

The problem here is not one of competence or failure to think through implications. Facebook know exactly what they are doing, and you can be sure the potential implications of these changes were debated at length within the company at a high level. They were taken precisely because of their implications for non-technical users, not in spite of them - FB has replaced users contact details because they want Facebook to be the only way most people can contact others online. Facebook have always wanted to replace the internet entirely with a system they control. Hence the walled garden approach with incessant prompts to signup to view content, the use of address book import to spread virally, the use of beacon to track and announce users activities, the use of like button js to track activity, the launch of Facebook apps and single sign-on to attempt to corral web developers within their ecosystem and make them dependent, the launch of a competitor to email/IM etc etc. This latest move with email is simply the latest in a long line of moves consistently attempting to steer users into spending all of their online time within the confines of Facebook, it is entirely consistent with their past behaviour and we should expect similar moves in future. So the arguments about how they could have done this differently, or whether Facebook has made some mistakes here are largely irrelevant - they will always push to own your data, email, photos, and digital life, because it's in their DNA, this is not a mistake, it's a pattern of behaviour - they can and will control your digital life if you choose to hand it over to them. Consider the quote from FB on the way their FB email works: Regarding the "email loss" this may actually just be confusion around the Messages Inbox: By default, messages from friends or friends of friends go into your Inbox. Everything else goes to your Other folder. All messages which do not come from FB contacts go into a folder which is effectively a junk folder - they want you to use their email for everything (hence the recent changes), they want your address book to contain nothing but FB contacts, they want you to forget email even existed, and they want you to encourage your friends to join FB so that you actually receive their messages. That won't work for all their users of course, but their hope is that they keep enough of the user base on board that they can dominate the internet and replace email with FB and the web with a network of sites where they control login and user info and all roads lead to FB. So replacing email with FB mail wherever they can, hiding other contacts and trashing user address books is not a mistake, it's part of a broader strategy, one which is not in the best interests of their users, but which they hope most will go along with anyway.

GBKS 14y ago

Now what happens if other sites and apps do the same? Will our address books change every day and contacts fill up with dozens of unused email addresses? Many of Facebook's growth techniques have been imitated by other startups so going into the territory of manipulating data that has always been off-limits is a big step. If another social site uses those techniques against to manipulate data on Facebook profiles, will Facebook change it's mind?

gitarr 14y ago

If lawyers are reading this, a question: Our governments are treating our emails more and more like real mail (aka. letters). Is there not a law prohibiting third party people or companies to mess with(fake) my mail addresses? Fact is, I do not want a facebook email address. And I have given no consent for my email address to be changed.

rhizome 14y ago
It's more like someone sending a change-of-address card to the post office to have your mail delivered to their Mailboxes Etc. store.
- tfb 14y ago
  That's actually a really good analogy. And the GP is right. There should definitely be some laws to prohibit these kinds of things that Facebook is attempting (doing), now that electronic mail/communication is becoming standard.
  - rhizome 14y ago
    As a matter of fact, the (US) Postal Inspector considers this to be Identity Theft.
    - tfb 14y ago
      Interesting. I wonder why no legal action has been taken against Facebook for this recent fiasco... or has there?

crazygringo 14y ago

So this whole debacle has taught me that I have an "Other" folder, which I never saw before, and it turns out I have 20+ messages from people from the past few months which I didn't know about! I'm trying to find a way to get Facebook to send me an e-mail notification when I receive a message that goes into "Other", the same way it sends me an e-mail notification when I receive a message that goes into "Inbox". But I can't find any option for this anywhere. Does anyone know? Otherwise, how am I ever supposed to know I have a message from someone on Facebook who I'm not already friends with?!

crazygringo 14y ago
Wow, this is infuriating. According to Slate: So how can you make sure you find your lost laptop, appear on Israeli TV, and respond immediately to job offers? Unfortunately, the Facebook rep informed me, you can’t change your settings to get email notifications for your Other messages, the way you can for your main messages. Your best bet: Make checking the Other tab part of your daily Facebook routine. So since I never log into Facebook, but use it as a place to receive messages, which I find out about via e-mail... I am doomed to never find out about possibly important messages from people who are not already my Facebook friends. This is possibly the worst design decision I've ever come across on the Internet. Facebook has created a black hole for messages. Even when you log into Facebook, it doesn't show a number next to "Other", so you can't even find out if you have messages there without navigating to the "Other" page. I can't wrap my mind around how insanely stupid this is. WTF, Facebook?
- Tyr42 14y ago
  Time to use selenium [1] to automate it and send you emails? [1] : http://seleniumhq.org/ http://seleniumhq.org/

mariuolo 14y ago

Why anyone would delegate handling of their address book to Facebook is beyond me.