5 ms·

How does this happen? I realize it's a small percentage, but this is one of the first tests you build. Even 1 photo (not to mention this is regarding videos) sh

How does this happen? I realize it's a small percentage, but this is one of the first tests you build. Even 1 photo (not to mention this is regarding videos) should never make it to another non-authenticated user. This is a massive mistake.

beamatronic 7y ago
They had one job!
whatisthiseven 7y ago
I think, when discussing software, the phrase "should never" just doesn't belong. I have only a short software career, but of all the bugs and unintentional behavior I have read about failures small and big companies make, I am no longer surprised. For that matter, software itself is full of surprising and undefined behavior, so it really shouldn't be a surprise that large corporations sometimes have "simple" mistakes that appear really big. That is just software in motion.
- jrowley 7y ago
  Unintended software behavior exists on a spectrum and a great deal of it is preventable by learning from past mistakes. It’s unfortunate we don’t have more information about the nature of the underlying this issue.
thedance 7y ago
I don't work at Google (any more) but I have seen bugs in large-scale production that served one user's data to another user, and both times it had the same cause: a developer stored user-specific data into a process-global singleton because the consequence of some java decorator was non-obvious. When the next user request came along they were served the previous user's information.
- nappy-doo 7y ago
  This guy injects @ Google.
  - thedance 7y ago
    Guice is evil and should be eradicated, but it's probably easier to get rid of Java itself.
dehrmann 7y ago
One way I've seen this happen is images have UUIDs (or just incrementing), they're base-64 encoded somewhere after security checks, and someone accidentally called `toLower()` on the id.
- KMag 7y ago
  Going from 22 base-64 characters to 22 base-36 characters still only drops you to about 113 bits of entropy. Unless we're talking significantly more than a quadrillion images in the system, getting a collision on a 113-bit random ID is exceedingly unlikely. I strongly suspect there were more problems in the system than just toLower(), or else the system was hosting significantly more than a quadrillion images. Your mention of "or just incrementing" sounds much closer to the mark.
gwbas1c 7y ago
I once witnessed a very similar bug. The story was a comedy of errors: 1: I interviewed a HORRIBLE candidate and told my boss in no uncertain terms not to hire the bozo. Then, I look at my colleagues and explain how poorly the interview went. 2: I go away for two weeks to have surgery 3: I come back and learn that we're hiring the bozo 4: My boss asks me to do some pair programming with the bozo, and he doesn't understand some very basic concepts 5: I hear the bozo is debugging webservice code in production 6: We had a data leak from one of the bozo's bugs From what I remember, it was a very dumb bug based on clear misunderstanding of fundamentals.
bouke 7y ago
I once witnessed something similar. Deep down in the repository code there was a field marked `static`, which should've been an instance property. The value would be overwritten by the last user to access the repository; so this resulted in a race condition where one user might see the other's data.

beamatronic 7y ago

They had one job!

whatisthiseven 7y ago

I think, when discussing software, the phrase "should never" just doesn't belong. I have only a short software career, but of all the bugs and unintentional behavior I have read about failures small and big companies make, I am no longer surprised. For that matter, software itself is full of surprising and undefined behavior, so it really shouldn't be a surprise that large corporations sometimes have "simple" mistakes that appear really big. That is just software in motion.

jrowley 7y ago
Unintended software behavior exists on a spectrum and a great deal of it is preventable by learning from past mistakes. It’s unfortunate we don’t have more information about the nature of the underlying this issue.

thedance 7y ago

I don't work at Google (any more) but I have seen bugs in large-scale production that served one user's data to another user, and both times it had the same cause: a developer stored user-specific data into a process-global singleton because the consequence of some java decorator was non-obvious. When the next user request came along they were served the previous user's information.

nappy-doo 7y ago
This guy injects @ Google.
- thedance 7y ago
  Guice is evil and should be eradicated, but it's probably easier to get rid of Java itself.

dehrmann 7y ago

One way I've seen this happen is images have UUIDs (or just incrementing), they're base-64 encoded somewhere after security checks, and someone accidentally called `toLower()` on the id.

KMag 7y ago
Going from 22 base-64 characters to 22 base-36 characters still only drops you to about 113 bits of entropy. Unless we're talking significantly more than a quadrillion images in the system, getting a collision on a 113-bit random ID is exceedingly unlikely. I strongly suspect there were more problems in the system than just toLower(), or else the system was hosting significantly more than a quadrillion images. Your mention of "or just incrementing" sounds much closer to the mark.

gwbas1c 7y ago

I once witnessed a very similar bug. The story was a comedy of errors: 1: I interviewed a HORRIBLE candidate and told my boss in no uncertain terms not to hire the bozo. Then, I look at my colleagues and explain how poorly the interview went. 2: I go away for two weeks to have surgery 3: I come back and learn that we're hiring the bozo 4: My boss asks me to do some pair programming with the bozo, and he doesn't understand some very basic concepts 5: I hear the bozo is debugging webservice code in production 6: We had a data leak from one of the bozo's bugs From what I remember, it was a very dumb bug based on clear misunderstanding of fundamentals.

bouke 7y ago

I once witnessed something similar. Deep down in the repository code there was a field marked `static`, which should've been an instance property. The value would be overwritten by the last user to access the repository; so this resulted in a race condition where one user might see the other's data.