6 ms·

What a crock of shit: after boot camp, all engineers get access to live DB I can understand on a startup or small org but an organisation of that size, there

What a crock of shit: after boot camp, all engineers get access to live DB I can understand on a startup or small org but an organisation of that size, there should be very tight access control. Despite what anyone says, the probability that someone does something bad increases in larger groups. Security should be on a simple need-to-know basis and nothing else. I build BIG financial software and we have certain audit requirements, access control requirements, data protection requirements etc and that is exactly how it should be. I'd never put my data near FB. They are simply irresponsible.

TazeTSchnitzel 14y ago
It's a little irresponsible, but how else do you expect them to debug live code?
- bnr 14y ago
  You don't. Reproduce the issue on a development instance.
  - damncabbage 14y ago
    I first took TazeTSchnitzel's comment to be sarcasm, but now I'm not so sure. (Poe's Law says hi.)
    - TazeTSchnitzel 14y ago
      Not sarcasm. Of course you should reproduce in a development environment, but for some issues, looking at live DB is the only way to see what is wrong.
      - mgkimsal 14y ago
        A system I work on has grown a lot over the last few years (data for 2000 users is now data for 50,000, rules have been added, etc). Example story: We had an issue that was only coming up on production. I could not reproduce it in a dev environment. Worse, we didn't even notice it for a long time because it was a nightly job, and people were not reporting an absence of their notifications. (nightly job to email reports). Finally got a dump of live data to dev system (it's a lot of data, so I don't pull it all the time). Someone had updated their email address to something invalid, and the system threw an uncaught exception in the middle of the loop (yay java). So... half the people were getting their stuff, the second half didn't - guilty record was a user with last name starting with L. Yes, we should have prevented a bad email from going in with validation, but it slipped my testing (and the client's). It's just shocking to me today that professional people whose job it is to send and receive email can mistype their email. "john smith @yaho" is not valid, yes I should catch that, but someone typed that in. Adding on top of that was my own dynamic language background not mapping well to the JVM - one bad address in the middle of a loop doesn't just get skipped and logged, but the entire process now stops. Multiple lessons learned from that one certainly (logging, validation, exception handling, etc) but... it would have been a lot longer for me to even consider putting in an invalid email address (it worked in dev - it was working for end users, etc) - pulling live data was the only thing that made it apparent.
        wpietri 14y ago
        I agree totally with your broad point, but wanted to respond to the "I can't believe a user typed..." Even if your 50,000 people are all pretty smart, you're well out into the range of exceptional circumstances. If I do something once a day my whole life, that's still only 25,000 times. It's pretty easy to imagine somebody on their worst day doing something like that. E.g., You go to visit family, so you're jet lagged. You had a couple of late beers with your brother, so you're hung over. His baby is screaming with colic and the toddler is banging pots together. You're VPNing in from their kitchen table, trying to fill out some form, and in the middle somebody knocks over a glass of juice. You yank your laptop up, help clean up the mess, and then sit back down to finish. While the pot-banging proceeds at full volume. Madness, sure, but common madness.
        gouranga 14y ago
        > Adding on top of that was my own dynamic language background not mapping well to the JVM There's your problem...
        mgkimsal 14y ago
        It's a problem, but it certainly wasn't my only - groovy, to be precise.
        tedunangst 14y ago
        Adding on top of that was my own dynamic language background not mapping well to the JVM - one bad address in the middle of a loop doesn't just get skipped and logged, but the entire process now stops. Makes me wonder what dynamic language you're used to. An uncaught exception in both python and ruby aborts the program, not just the tiny inner loop that happens to be running.
        mgkimsal 14y ago
        Primarily PHP, although I seem to remember classic ASP VBScript not dying if you passed a bad email to CDO (but that's been a long time, so i might not remember that correctly). PHP would not throw an exception if the mail function got a bad email address. well, using some mailing libraries might, but the core mailing function (basically just a wrapper for sendmail/postfix CLI stuff) wouldn't.
  - rimantas 14y ago
    How do you do that, if issue is caused by some corner case with user data you did not foresee and hence not covered?
    - Goladus 14y ago
      For some applications, the number of corner cases that can not be resolved by reading log files is minimal. (you are writing logs, right?) That is, too, assuming the problem is actually in the code itself and not the infrastructure.
- arethuza 14y ago
  "how else do you expect them to debug live code" I've actually spent quite a long time writing code where, for various legal reasons, you are never going to get direct access to the production data. (And indeed some cases where accessing the data in question would be a serious criminal offence). Makes you really keen on defensive programming and comprehensive logging - although you do then have to be careful about what you log. I actally found it kind of fun to try and debug complex systems based only on a log file and QA with production operations folks. [NB Of course, none of this would be applicable to Facebook]
- gouranga 14y ago
  Thou who uses unit tests, UI tests, precondition, postcondition checks, heavy logging, knows the language and toolchain and knows arse from elbow NEVER debugs in production out of necessity or desire. It just works. In the last 5 years, we've never hooked a debugger to production instance or given access to a developer. In that time, we've pushed approximately 912 billion SQL transactions through the system and 24 billion page hits (most of our stuff is backend non OLTP). That's how it's done.
  - TazeTSchnitzel 14y ago
    You're right. Now that I think about it, I doubt Facebook is that careful, considering how often they break things.
  - rimantas 14y ago
    No please tell me, how did you learn to write unit test which cover all the crazy data configurations users are able to come with.
    - alttab 14y ago
      By being able to design software that doesn't need or allow crazy data configurations. My whole mantra is don't manage complexity: avoid it.
      - gouranga 14y ago
        You nailed it - thanks for the absolutely spot on comment. Design is the key word here. It requires thought, it requires intelligence and it requires multiple people's input.
    - gioele 14y ago
      Boundary value analysis? Machine state transitions testing? Equivalence partitioning? There are heaps of techniques one can use to identify which data points and combinations are useful to test. Some of these techniques even take a peek at your code to highlight the possible pain points.
jschuur 14y ago
Note that it doesn't say whether they all have write access, or whether all the data they have access to is encrypted in some way. And any of those statements should be interpreted as one response he got from talking to lots of different people, and be subject to some amount of skepticism.
- gouranga 14y ago
  Either way, it simply doesn't matter.
piggity 14y ago
You finance and billing guys will never understand. These are rockstars. They work for facebook. They would never ever type UPDATE users SET email = username || '@facebook.com'; WHERE username == 'john.smith';
- jacques_chester 14y ago
  Of course not. Much faster to type UPDATE users SET email = username || '@facebook.com'; Then let the users sort out the exceptions.
  - alttab 14y ago
    I think he was being sarcastic - you can see the semicolon in the middle of the command which looks like pretty much what they did with the whole e-mail debacle. You simply removed the second incomplete statement.
    - jacques_chester 14y ago
      I now realise that I am underqualified to work at Facebook. :(
- deleted 14y ago
  [deleted]
- getsat 14y ago
  >implying this was an engineering team decision
jdsemrau 14y ago
>I build BIG financial software and we have certain audit requirements, access control requirements, data protection requirements etc and that is exactly how it should be. Funny, I had the same argument with a colleague today. Since FB is listed and their business model is based on customer data they should follow the same legal requirements as customer finance companies.
- yashchandra 14y ago
  "they should follow the same legal requirements" Perhaps. But they certainly do not have the restrictions of financial audit requirements and things like Dodd Frank etc. which is overwhelming all the major banks right now and is a big way to earn money for consultants if they know ABC of audit/compliance in 2012. Anyway, the point is that fb even though not ideal in how they store/use user data (which I personally am not a fan of as well), it does not matter as much as it matters to a bank/financial institution. I say this while I work for clients that are major banks.
jorgeleo 14y ago
But Facebook is not a financial institution, and I would put the responsability in the user if financial data is on the news feed. One size (of engineering practices) fit all mentality is not the correct approach either.
- gouranga 14y ago
  You are correct - it is not a financial organsiation. It does know who you are, what you eat, where you've been, who you're friends with, what you're interested in, what your political allegiances are, what your bowel movements are like etc. A list of investments or your mortgage statement is way less important.
gaius 14y ago
Right. FB doesn't get that all they have is the user experience, and it is bad user experience if I drop out of a conversation I'm having because their DBA-free database has decided to simply, silently drop my last comment. Or if it takes me several attempts to post a photo. Or if I invite someone to an event, and they never get the invite. You can do all sorts of amazing things if you just don't care if your code actually works or not. Grown-up companies have full-time professional DBAs, and not just for separation of concerns.
- darkarmani 14y ago
  > Grown-up companies have full-time professional DBAs I don't know if I'd call them "professional," but they definitely have full-time DBAs. ;)
  - mkjones 14y ago
    What do you mean? I've worked with some of our DBAs, and they're quite good. In fact, I can't think of a single site issue that was caused by a DBA. I work on fighting spam at FB, and we make use of mysql quite a bit.

TazeTSchnitzel 14y ago

It's a little irresponsible, but how else do you expect them to debug live code?

bnr 14y ago
You don't. Reproduce the issue on a development instance.
- damncabbage 14y ago
  I first took TazeTSchnitzel's comment to be sarcasm, but now I'm not so sure. (Poe's Law says hi.)
  - TazeTSchnitzel 14y ago
    Not sarcasm. Of course you should reproduce in a development environment, but for some issues, looking at live DB is the only way to see what is wrong.
    - mgkimsal 14y ago
      A system I work on has grown a lot over the last few years (data for 2000 users is now data for 50,000, rules have been added, etc). Example story: We had an issue that was only coming up on production. I could not reproduce it in a dev environment. Worse, we didn't even notice it for a long time because it was a nightly job, and people were not reporting an absence of their notifications. (nightly job to email reports). Finally got a dump of live data to dev system (it's a lot of data, so I don't pull it all the time). Someone had updated their email address to something invalid, and the system threw an uncaught exception in the middle of the loop (yay java). So... half the people were getting their stuff, the second half didn't - guilty record was a user with last name starting with L. Yes, we should have prevented a bad email from going in with validation, but it slipped my testing (and the client's). It's just shocking to me today that professional people whose job it is to send and receive email can mistype their email. "john smith @yaho" is not valid, yes I should catch that, but someone typed that in. Adding on top of that was my own dynamic language background not mapping well to the JVM - one bad address in the middle of a loop doesn't just get skipped and logged, but the entire process now stops. Multiple lessons learned from that one certainly (logging, validation, exception handling, etc) but... it would have been a lot longer for me to even consider putting in an invalid email address (it worked in dev - it was working for end users, etc) - pulling live data was the only thing that made it apparent.
      - wpietri 14y ago
        I agree totally with your broad point, but wanted to respond to the "I can't believe a user typed..." Even if your 50,000 people are all pretty smart, you're well out into the range of exceptional circumstances. If I do something once a day my whole life, that's still only 25,000 times. It's pretty easy to imagine somebody on their worst day doing something like that. E.g., You go to visit family, so you're jet lagged. You had a couple of late beers with your brother, so you're hung over. His baby is screaming with colic and the toddler is banging pots together. You're VPNing in from their kitchen table, trying to fill out some form, and in the middle somebody knocks over a glass of juice. You yank your laptop up, help clean up the mess, and then sit back down to finish. While the pot-banging proceeds at full volume. Madness, sure, but common madness.
      - gouranga 14y ago
        > Adding on top of that was my own dynamic language background not mapping well to the JVM There's your problem...
        mgkimsal 14y ago
        It's a problem, but it certainly wasn't my only - groovy, to be precise.
      - tedunangst 14y ago
        Adding on top of that was my own dynamic language background not mapping well to the JVM - one bad address in the middle of a loop doesn't just get skipped and logged, but the entire process now stops. Makes me wonder what dynamic language you're used to. An uncaught exception in both python and ruby aborts the program, not just the tiny inner loop that happens to be running.
        mgkimsal 14y ago
        Primarily PHP, although I seem to remember classic ASP VBScript not dying if you passed a bad email to CDO (but that's been a long time, so i might not remember that correctly). PHP would not throw an exception if the mail function got a bad email address. well, using some mailing libraries might, but the core mailing function (basically just a wrapper for sendmail/postfix CLI stuff) wouldn't.
- rimantas 14y ago
  How do you do that, if issue is caused by some corner case with user data you did not foresee and hence not covered?
  - Goladus 14y ago
    For some applications, the number of corner cases that can not be resolved by reading log files is minimal. (you are writing logs, right?) That is, too, assuming the problem is actually in the code itself and not the infrastructure.
arethuza 14y ago
"how else do you expect them to debug live code" I've actually spent quite a long time writing code where, for various legal reasons, you are never going to get direct access to the production data. (And indeed some cases where accessing the data in question would be a serious criminal offence). Makes you really keen on defensive programming and comprehensive logging - although you do then have to be careful about what you log. I actally found it kind of fun to try and debug complex systems based only on a log file and QA with production operations folks. [NB Of course, none of this would be applicable to Facebook]
gouranga 14y ago
Thou who uses unit tests, UI tests, precondition, postcondition checks, heavy logging, knows the language and toolchain and knows arse from elbow NEVER debugs in production out of necessity or desire. It just works. In the last 5 years, we've never hooked a debugger to production instance or given access to a developer. In that time, we've pushed approximately 912 billion SQL transactions through the system and 24 billion page hits (most of our stuff is backend non OLTP). That's how it's done.
- TazeTSchnitzel 14y ago
  You're right. Now that I think about it, I doubt Facebook is that careful, considering how often they break things.
- rimantas 14y ago
  No please tell me, how did you learn to write unit test which cover all the crazy data configurations users are able to come with.
  - alttab 14y ago
    By being able to design software that doesn't need or allow crazy data configurations. My whole mantra is don't manage complexity: avoid it.
    - gouranga 14y ago
      You nailed it - thanks for the absolutely spot on comment. Design is the key word here. It requires thought, it requires intelligence and it requires multiple people's input.
  - gioele 14y ago
    Boundary value analysis? Machine state transitions testing? Equivalence partitioning? There are heaps of techniques one can use to identify which data points and combinations are useful to test. Some of these techniques even take a peek at your code to highlight the possible pain points.

jschuur 14y ago

Note that it doesn't say whether they all have write access, or whether all the data they have access to is encrypted in some way. And any of those statements should be interpreted as one response he got from talking to lots of different people, and be subject to some amount of skepticism.

gouranga 14y ago
Either way, it simply doesn't matter.

piggity 14y ago

You finance and billing guys will never understand. These are rockstars. They work for facebook. They would never ever type UPDATE users SET email = username || '@facebook.com'; WHERE username == 'john.smith';

jacques_chester 14y ago
Of course not. Much faster to type UPDATE users SET email = username || '@facebook.com'; Then let the users sort out the exceptions.
- alttab 14y ago
  I think he was being sarcastic - you can see the semicolon in the middle of the command which looks like pretty much what they did with the whole e-mail debacle. You simply removed the second incomplete statement.
  - jacques_chester 14y ago
    I now realise that I am underqualified to work at Facebook. :(
deleted 14y ago
[deleted]
getsat 14y ago
>implying this was an engineering team decision

jdsemrau 14y ago

>I build BIG financial software and we have certain audit requirements, access control requirements, data protection requirements etc and that is exactly how it should be. Funny, I had the same argument with a colleague today. Since FB is listed and their business model is based on customer data they should follow the same legal requirements as customer finance companies.

yashchandra 14y ago
"they should follow the same legal requirements" Perhaps. But they certainly do not have the restrictions of financial audit requirements and things like Dodd Frank etc. which is overwhelming all the major banks right now and is a big way to earn money for consultants if they know ABC of audit/compliance in 2012. Anyway, the point is that fb even though not ideal in how they store/use user data (which I personally am not a fan of as well), it does not matter as much as it matters to a bank/financial institution. I say this while I work for clients that are major banks.

jorgeleo 14y ago

But Facebook is not a financial institution, and I would put the responsability in the user if financial data is on the news feed. One size (of engineering practices) fit all mentality is not the correct approach either.

gouranga 14y ago
You are correct - it is not a financial organsiation. It does know who you are, what you eat, where you've been, who you're friends with, what you're interested in, what your political allegiances are, what your bowel movements are like etc. A list of investments or your mortgage statement is way less important.

gaius 14y ago

Right. FB doesn't get that all they have is the user experience, and it is bad user experience if I drop out of a conversation I'm having because their DBA-free database has decided to simply, silently drop my last comment. Or if it takes me several attempts to post a photo. Or if I invite someone to an event, and they never get the invite. You can do all sorts of amazing things if you just don't care if your code actually works or not. Grown-up companies have full-time professional DBAs, and not just for separation of concerns.

darkarmani 14y ago
> Grown-up companies have full-time professional DBAs I don't know if I'd call them "professional," but they definitely have full-time DBAs. ;)
- mkjones 14y ago
  What do you mean? I've worked with some of our DBAs, and they're quite good. In fact, I can't think of a single site issue that was caused by a DBA. I work on fighting spam at FB, and we make use of mysql quite a bit.