9 ms·

Sometimes it seems to me that FB doesn't have an impact assessment process. This is a very important data change you're making to 900-ish million records on a

by 51Cards 14y ago

Sometimes it seems to me that FB doesn't have an impact assessment process. This is a very important data change you're making to 900-ish million records on a system that is inter-woven with and syncing to... everything. I understand bugs happening, but really, the cascade effect of this email change had to have been foreseen by someone... no? Bueller? Bueller? I really don't mean to trash FB as a lot of good people work there. However... when I read about something like this, and then I see a company like Google figuring out a time smear function to seamlessly account for a leap second in their servers to avoid cascading system changes... it's hard to feel the two companies are functioning on the same level. (I know, a mis-balanced and irrelevant comparison, but it's just for a 'gut feeling' sense)

rhizome 14y ago
They knew what they were doing. Not having a complete rollback was intentional.
- eli 14y ago
  I believe Facebook is capable of doing sneaky things to advance their business goals, but Hanlon's razor [1] comes to mind here. [1] http://www.jargon.net/jargonfile/h/HanlonsRazor.html http://www.jargon.net/jargonfile/h/HanlonsRazor.html
  - rhizome 14y ago
    I know what Hanlon's razor is, and I can guarantee you FB Engineering knows what a rollback is and how to know whether you need one or not. I'm not sure where you're getting the idea that they're that stupid.
    - keithpeter 14y ago
      Would applying such a 'rollback' involve replacing facebook.com email addresses with the original external domain addresses in the contacts list on the devices affected by the write-over issue? If so: Facebook have taken private data and placed it in a central location If not: it isn't a rollback that I understand Disclaimer: I'm not a programmer
      - rhizome 14y ago
        Yes, a (hypothetical) "complete rollback" would involve replacing removed data. Sure, data privacy would be a concern, but it's beside the point that rolling this back would necessitate un-overwriting.
    - Cloven 14y ago
      how about 'the evidence on display in front of us'?
  - Devilboy 14y ago
    Hanlon's razor is not a law of nature.
    - eli 14y ago
      Neither is the opposite.
- j_s 14y ago
  Only Violet Blue has said that they don't have a complete rollback as best I can tell from the CNET article. I would guess the only thing they can't undo would be the time that has passed. If any data Facebook ever had on any of their users (such as their primary email address [edit: at any point in time]) was ever lost, they wouldn't have as much to sell to their customers! Read me blathering on further about this: http://news.ycombinator.com/item?id=4192299 http://news.ycombinator.com/item?id=4192299
  - rhizome 14y ago
    Yeah, I thought about leaving that second sentence off. Point remains.
    - j_s 14y ago
      Thanks for sharing your perspective. Just to make sure we're on the same page, you are arguing that FB engineering was 100% aware of the consequences of automatically switching primary email address to @facebook.com for users who had setup that functionality, including overwriting email addresses in phone contact lists and capturing email unintentionally sent via those modified contact lists? This is certainly within the realm of possibility and indicates that you hold the team's technical ability in high regard. If it is indeed true, I would wonder what percentage of the folks involved in the implementation actually thought about it and how high/wide the issue was escalated. I have certainly been in situations where after-the-fact I found out that something I had been a part of shipped with known bugs/features I didn't know about that I would have raised a lot more concern over.
      - rhizome 14y ago
        My line of thought is that they knew it would change phonebooks and that they weren't putting whatever resources necessary into creating a process for rollback. Given the nature of Facebook and it's technology, I'm not even sure there is anything they could have done, but it's de rigeur to know if this is the case. It's not even about high regard, but by minimum standards. If you know you can't rollback you go full steam ahead. If you've never worked someplace where they have dedicated staff for source control, that fact alone might give you an idea at the level of expertise involved here. It's nobody's first time at the rodeo, and setting the sync API on the change (as described below) wasn't just slipped in by an intern. You finding out after-the-fact and still having an opinion about bugs shipping in code you contributed to only reinforces my point: someone made the decision to go ahead anyway.
tedunangst 14y ago
google buzz?
dasil003 14y ago
> and then I see a company like Google figuring out a time smear function to seamlessly account for a leap second in their servers to avoid cascading system changes... it's hard to feel the two companies are functioning on the same level. Sorta makes you wish Google would stop chasing Facebook and just do their thing.
- bgilroy26 14y ago
  Connecting site recommendations to user profiles has the potential to produce better, unspammable Google search results. Google cannot allow other companies to get ahead of them on social search and remain competitive in their most important vertical.

by 51Cards 14y ago

rhizome 14y ago

They knew what they were doing. Not having a complete rollback was intentional.

eli 14y ago
I believe Facebook is capable of doing sneaky things to advance their business goals, but Hanlon's razor [1] comes to mind here. [1] http://www.jargon.net/jargonfile/h/HanlonsRazor.html http://www.jargon.net/jargonfile/h/HanlonsRazor.html
- rhizome 14y ago
  I know what Hanlon's razor is, and I can guarantee you FB Engineering knows what a rollback is and how to know whether you need one or not. I'm not sure where you're getting the idea that they're that stupid.
  - keithpeter 14y ago
    Would applying such a 'rollback' involve replacing facebook.com email addresses with the original external domain addresses in the contacts list on the devices affected by the write-over issue? If so: Facebook have taken private data and placed it in a central location If not: it isn't a rollback that I understand Disclaimer: I'm not a programmer
    - rhizome 14y ago
      Yes, a (hypothetical) "complete rollback" would involve replacing removed data. Sure, data privacy would be a concern, but it's beside the point that rolling this back would necessitate un-overwriting.
  - Cloven 14y ago
    how about 'the evidence on display in front of us'?
- Devilboy 14y ago
  Hanlon's razor is not a law of nature.
  - eli 14y ago
    Neither is the opposite.
j_s 14y ago
Only Violet Blue has said that they don't have a complete rollback as best I can tell from the CNET article. I would guess the only thing they can't undo would be the time that has passed. If any data Facebook ever had on any of their users (such as their primary email address [edit: at any point in time]) was ever lost, they wouldn't have as much to sell to their customers! Read me blathering on further about this: http://news.ycombinator.com/item?id=4192299 http://news.ycombinator.com/item?id=4192299
- rhizome 14y ago
  Yeah, I thought about leaving that second sentence off. Point remains.
  - j_s 14y ago
    Thanks for sharing your perspective. Just to make sure we're on the same page, you are arguing that FB engineering was 100% aware of the consequences of automatically switching primary email address to @facebook.com for users who had setup that functionality, including overwriting email addresses in phone contact lists and capturing email unintentionally sent via those modified contact lists? This is certainly within the realm of possibility and indicates that you hold the team's technical ability in high regard. If it is indeed true, I would wonder what percentage of the folks involved in the implementation actually thought about it and how high/wide the issue was escalated. I have certainly been in situations where after-the-fact I found out that something I had been a part of shipped with known bugs/features I didn't know about that I would have raised a lot more concern over.
    - rhizome 14y ago
      My line of thought is that they knew it would change phonebooks and that they weren't putting whatever resources necessary into creating a process for rollback. Given the nature of Facebook and it's technology, I'm not even sure there is anything they could have done, but it's de rigeur to know if this is the case. It's not even about high regard, but by minimum standards. If you know you can't rollback you go full steam ahead. If you've never worked someplace where they have dedicated staff for source control, that fact alone might give you an idea at the level of expertise involved here. It's nobody's first time at the rodeo, and setting the sync API on the change (as described below) wasn't just slipped in by an intern. You finding out after-the-fact and still having an opinion about bugs shipping in code you contributed to only reinforces my point: someone made the decision to go ahead anyway.

tedunangst 14y ago

google buzz?

dasil003 14y ago

> and then I see a company like Google figuring out a time smear function to seamlessly account for a leap second in their servers to avoid cascading system changes... it's hard to feel the two companies are functioning on the same level. Sorta makes you wish Google would stop chasing Facebook and just do their thing.

bgilroy26 14y ago
Connecting site recommendations to user profiles has the potential to produce better, unspammable Google search results. Google cannot allow other companies to get ahead of them on social search and remain competitive in their most important vertical.