5 ms·

Aside from the Google domain issue (which is obviously a problem) -- why on earth would GitBook be hosting user-generated content under their corporate gitbook.

Aside from the Google domain issue (which is obviously a problem) -- why on earth would GitBook be hosting user-generated content under their corporate gitbook.com domain? The registrar issue is one problem here -- but I don't see this addressed in their post-mortem. I thought this was a well-known issue. You don't host user-generated content on the same domain that handles your corporate email. Because things like this can and do happen, so you want to make sure that your company isn't also down while you're busy fixing a problem with your customers. I know the exposure risks are different, but isn't this is one of the reasons why Github moved hosting Github Pages content to github.io from their primary github.com site? Or why raw data is hosted from githubusercontent.com (in addition to mitigating cookie security issues). The registrar was an issue that was largely out of their hands. But this was something that they could control. And I think it's something that is missing from their post-mortem. If they had split their domains, then while serving user-content was disrupted, accepting users to gitbook.com and their email (!?!?) would have still been working. Also, depending on if they had a 3rd domain for hosting their CDN, users that had custom domains would also have been protected.This goes along with the idea that you don't use email from your primary domain (cto@gitbook.com) to register your domain (gitbook.com). Or host your status page on the same infrastructure as your site. If I were them, after migrating registrars, this would be the next engineering change I'd make. (Maybe they do this, I don't know enough about GitBook to know. But based on the thread here, I don't think they do, or at least it isn't mentioned in their post-mortem that I saw.)

jtl999 6y ago
Where do you draw the line? I mean github.com is the corporate email domain of the GitHub company, it also hosts repositories (user generated content)
- fred256 6y ago
  But not for any binary content as uploaded by the user, that's served from raw.githubusercontent.com.
  - jtl999 6y ago
    I'm aware, but I could just as easily see a lesser registrar getting confused by an abuse report and suspending github.com because it embeds content from githubusercontent.com.
    - mbreese 6y ago
      So you also use two different registrars (the cheap option)? Or the more expensive option where you use a registrar that gives you get a direct phone number to an account manager.
  - ken 6y ago
    Not all of it. Any bug attachments which don't have previews are served from https://github.com https://github.com directly, for example. Here, I uploaded that image from the other day that crashes some phones. I gzipped it so it wouldn't generate a preview, and attached it to a bug. When you click the "github.com" link, it downloads the file, and (at least with my web browser) uncompresses it and opens it with your default application. It's bit-for-bit the same as what I uploaded. https://github.com/kengruven/strukt-bugs/issues/40 https://github.com/kengruven/strukt-bugs/issues/40 I don't know if this is exploitable. I haven't spent any time trying to break GitHub. This is just something I happened to notice once.
    - mbreese 6y ago
      That's interesting. It looks like the link is active, but the request to github.com results in a 302 redirect to S3. I don't know what that would mean in this type of scenario (phishing) either. I wonder what an html attachment would look like...
- mbreese 6y ago
  But it isn't... directly. Github.com is their corporate site, hosts their application, and is how we all interact with repositories (via https or git://). But, the only data you get from that site has been processed through their application and sanitized. The only way to get access to the raw user-generated data is through raw.githubusercontent.com or Github Pages which hosted on github.io. And the data from raw.githubusercontent.com has the MIME types set so that you don't get HTML rendered -- only the raw plaintext (I think). So, in this specific example, if there was someone hosting a phishing site in a github account, it would have been active only through github.io, not the main github.com site. (You could have likely seen the source code from the main site, but it would not have actually generated an HTML form). For historical views, here is the Github blog post that details the change (2013): https://github.blog/2013-04-05-new-github-pages-domain-github-io/ https://github.blog/2013-04-05-new-github-pages-domain-githu...
  - pmontra 6y ago
    Nevertheless some countries blocked github.com, not (only?) githubusercontent.com or github.io https://en.wikipedia.org/wiki/Censorship_of_GitHub https://en.wikipedia.org/wiki/Censorship_of_GitHub Probably they think that GitHub didn't sanitize the content on github.com well enough.
  - jasonlotito 6y ago
    That’s exactly what they are doing, too. Google effectively shut down github.com for the issues in github.io that didn’t exist anymore, in your example. https://news.ycombinator.com/item?id=23425569 https://news.ycombinator.com/item?id=23425569
waon 6y ago
Assumingly because GitBook didn't expect Google to be so reckless and tone-deaf.
anoncareer0212 6y ago
because tech culture permits you to blame Google and their support team instead of knowing basic domain operation practices
acituan 6y ago
Does it really make it less of a problem if they had a separate domain for corporate and it was only the user content going down? Serving user content can as well be part of a service’s core product and registrar induced downtime is a big enough problem. Building and running anti-abuse tech works in proportion to the money put in. Can a small player really compete with Google’s anti-abuse investments? No. Save for a couple of exceptions, this means Google could always find abusive user content in its clients’ domains that the clients themselves couldn’t catch. I am not suggesting small players should be able to freeride Google’s anti-abuse capabilities, but if leads to enforcement and downtime of this sort, there needs to be a better protocol for the client to be able to respond or it paints a very anti-competitive picture.
- shkkmo 6y ago
  > Does it really make it less of a problem if they had a separate domain for corporate and it was only the user content going down? Serving user content can as well be part of a service’s core product and registrar induced downtime is a big enough problem. Yes, it really does. Not losing your ability to receive email will help resolve the problem. > Serving user content can as well be part of a service’s core product and registrar induced downtime is a big enough problem. If that is the case, then choosing a registrar that provides quality, timely and responsive support when dealing abusive content complaints should be on the top of your priorities. > Building and running anti-abuse tech works in proportion to the money put in. Can a small player really compete with Google’s anti-abuse investments? In this case the small company did find the content first through their own anti-abuse systems. I'm not sure how this become anti-competitive. Google may have a monopoly on many things, but DNS registry is definitely not one of them.
  - acituan 6y ago
    > Not losing your ability to receive email will help resolve the problem. Sure, I'm trying to prevent this stealing focus from the fact that this shouldn't have been a problem to begin with. The argument for separation is not made for defending against a SLA limitation or some other technical reason. It is made to accommodate a faulty policy problem. It is equal to saying "best practice: all big-co domains users should have corporate and production domain separation in case big-co's policy layer messes up. oh well". I'd rather see big-co held accountable not to mess up. > If that is the case, then choosing a registrar that provides quality, timely and responsive support when dealing abusive content complaints should be on the top of your priorities. Hindsight is perfect, while those qualities are rarely perfectly symmetrical information. "They should have known this could have happened" sounds like victim blaming to me. They were entitled to reasonable policies and reasonable application of those policies. > I'm not sure how this become anti-competitive Monopoly is not the only machinery of anti-competition. Creating a landscape demanding anti-abuse parity is essentially creating a barrier of entry. They are going to cloudflare next but there is no telling a similar scenario won't happen. It is important to call out the potential of big-co's putting capital-intensive demands on little players to participate in the web.
thayne 6y ago
I can say from experience that moving content to a different domain without breaking things for users can be atrociously difficult. So if you make a mistake in how you split domains in the beginning, it can be very expensive to rectify later. Can you imagine going to all your customers and saying "you know all those links you have to example.com? You're going to have to change all of them to point to example.org now." They _should_ split their domains, and given this recent issue, it would make sense for them to prioritize that. But, well, easier said than done.
- mbreese 6y ago
  I imagine it will be painful to split their domains. But, I was surprised to see that wasn't mentioned in their post-mortem. Even if they didn't plan on doing it, I think that it would have made sense to mention. I imagine their main paying customers would start demanding this change though. They are probably using a custom domain, so whatever domain GitBook chooses to use should be irrelevant for them (in theory, but in practice, who knows...).
grey-area 6y ago
why on earth would GitBook be hosting user-generated content under their corporate gitbook.com domain? You mean like news.ycombinator.com, nytimes.com, facebook.com or most blogs or forums ever created? Also their entire business is serving user domains so if that goes down customers don't care about their marketing site, though the email vanishing would be annoying. What's to stop google doing this with any regular domain with user comments or content though? I blame the registrar - having heard this horror story I'd never host with google domains.
- otterley 6y ago
  Facebook uses fb.com as their corporate domain.
kohtatsu 6y ago
It most often has to do with browser sandboxing. See CORS, cookies, etc. It was a security flaw that began being addressed circa 2010.
SamyPesse 6y ago
We don't host user content under the gitbook.com, or at least we've stopped doing it a few years ago. User content is stored under *.gitbook.io, similar to GitHub. Google blocked all domains that contained "gitbook" in our account, even ones that are used for some infrastructure and are not accessible by the public. We don't know the exact reason for this, maybe they've blocked gitbook.com because we still have some redirect for content that was hosted under it years ago. And yes we are going to make changes to host our status page under another domain.
- mbreese 6y ago
  That’s really important to add to the story! I think that makes the rest of the story more troubling. Especially knowing that domains unrelated to the apparent user phishing code were affected. You might think of adding it to the post-mortem doc so that others don’t assume what I did.

jtl999 6y ago

Where do you draw the line? I mean github.com is the corporate email domain of the GitHub company, it also hosts repositories (user generated content)

fred256 6y ago
But not for any binary content as uploaded by the user, that's served from raw.githubusercontent.com.
- jtl999 6y ago
  I'm aware, but I could just as easily see a lesser registrar getting confused by an abuse report and suspending github.com because it embeds content from githubusercontent.com.
  - mbreese 6y ago
    So you also use two different registrars (the cheap option)? Or the more expensive option where you use a registrar that gives you get a direct phone number to an account manager.
- ken 6y ago
  Not all of it. Any bug attachments which don't have previews are served from https://github.com https://github.com directly, for example. Here, I uploaded that image from the other day that crashes some phones. I gzipped it so it wouldn't generate a preview, and attached it to a bug. When you click the "github.com" link, it downloads the file, and (at least with my web browser) uncompresses it and opens it with your default application. It's bit-for-bit the same as what I uploaded. https://github.com/kengruven/strukt-bugs/issues/40 https://github.com/kengruven/strukt-bugs/issues/40 I don't know if this is exploitable. I haven't spent any time trying to break GitHub. This is just something I happened to notice once.
  - mbreese 6y ago
    That's interesting. It looks like the link is active, but the request to github.com results in a 302 redirect to S3. I don't know what that would mean in this type of scenario (phishing) either. I wonder what an html attachment would look like...
mbreese 6y ago
But it isn't... directly. Github.com is their corporate site, hosts their application, and is how we all interact with repositories (via https or git://). But, the only data you get from that site has been processed through their application and sanitized. The only way to get access to the raw user-generated data is through raw.githubusercontent.com or Github Pages which hosted on github.io. And the data from raw.githubusercontent.com has the MIME types set so that you don't get HTML rendered -- only the raw plaintext (I think). So, in this specific example, if there was someone hosting a phishing site in a github account, it would have been active only through github.io, not the main github.com site. (You could have likely seen the source code from the main site, but it would not have actually generated an HTML form). For historical views, here is the Github blog post that details the change (2013): https://github.blog/2013-04-05-new-github-pages-domain-github-io/ https://github.blog/2013-04-05-new-github-pages-domain-githu...
- pmontra 6y ago
  Nevertheless some countries blocked github.com, not (only?) githubusercontent.com or github.io https://en.wikipedia.org/wiki/Censorship_of_GitHub https://en.wikipedia.org/wiki/Censorship_of_GitHub Probably they think that GitHub didn't sanitize the content on github.com well enough.
- jasonlotito 6y ago
  That’s exactly what they are doing, too. Google effectively shut down github.com for the issues in github.io that didn’t exist anymore, in your example. https://news.ycombinator.com/item?id=23425569 https://news.ycombinator.com/item?id=23425569

waon 6y ago

Assumingly because GitBook didn't expect Google to be so reckless and tone-deaf.

anoncareer0212 6y ago

because tech culture permits you to blame Google and their support team instead of knowing basic domain operation practices

acituan 6y ago

Does it really make it less of a problem if they had a separate domain for corporate and it was only the user content going down? Serving user content can as well be part of a service’s core product and registrar induced downtime is a big enough problem. Building and running anti-abuse tech works in proportion to the money put in. Can a small player really compete with Google’s anti-abuse investments? No. Save for a couple of exceptions, this means Google could always find abusive user content in its clients’ domains that the clients themselves couldn’t catch. I am not suggesting small players should be able to freeride Google’s anti-abuse capabilities, but if leads to enforcement and downtime of this sort, there needs to be a better protocol for the client to be able to respond or it paints a very anti-competitive picture.

shkkmo 6y ago
> Does it really make it less of a problem if they had a separate domain for corporate and it was only the user content going down? Serving user content can as well be part of a service’s core product and registrar induced downtime is a big enough problem. Yes, it really does. Not losing your ability to receive email will help resolve the problem. > Serving user content can as well be part of a service’s core product and registrar induced downtime is a big enough problem. If that is the case, then choosing a registrar that provides quality, timely and responsive support when dealing abusive content complaints should be on the top of your priorities. > Building and running anti-abuse tech works in proportion to the money put in. Can a small player really compete with Google’s anti-abuse investments? In this case the small company did find the content first through their own anti-abuse systems. I'm not sure how this become anti-competitive. Google may have a monopoly on many things, but DNS registry is definitely not one of them.
- acituan 6y ago
  > Not losing your ability to receive email will help resolve the problem. Sure, I'm trying to prevent this stealing focus from the fact that this shouldn't have been a problem to begin with. The argument for separation is not made for defending against a SLA limitation or some other technical reason. It is made to accommodate a faulty policy problem. It is equal to saying "best practice: all big-co domains users should have corporate and production domain separation in case big-co's policy layer messes up. oh well". I'd rather see big-co held accountable not to mess up. > If that is the case, then choosing a registrar that provides quality, timely and responsive support when dealing abusive content complaints should be on the top of your priorities. Hindsight is perfect, while those qualities are rarely perfectly symmetrical information. "They should have known this could have happened" sounds like victim blaming to me. They were entitled to reasonable policies and reasonable application of those policies. > I'm not sure how this become anti-competitive Monopoly is not the only machinery of anti-competition. Creating a landscape demanding anti-abuse parity is essentially creating a barrier of entry. They are going to cloudflare next but there is no telling a similar scenario won't happen. It is important to call out the potential of big-co's putting capital-intensive demands on little players to participate in the web.

thayne 6y ago

I can say from experience that moving content to a different domain without breaking things for users can be atrociously difficult. So if you make a mistake in how you split domains in the beginning, it can be very expensive to rectify later. Can you imagine going to all your customers and saying "you know all those links you have to example.com? You're going to have to change all of them to point to example.org now." They _should_ split their domains, and given this recent issue, it would make sense for them to prioritize that. But, well, easier said than done.

mbreese 6y ago
I imagine it will be painful to split their domains. But, I was surprised to see that wasn't mentioned in their post-mortem. Even if they didn't plan on doing it, I think that it would have made sense to mention. I imagine their main paying customers would start demanding this change though. They are probably using a custom domain, so whatever domain GitBook chooses to use should be irrelevant for them (in theory, but in practice, who knows...).

grey-area 6y ago

why on earth would GitBook be hosting user-generated content under their corporate gitbook.com domain? You mean like news.ycombinator.com, nytimes.com, facebook.com or most blogs or forums ever created? Also their entire business is serving user domains so if that goes down customers don't care about their marketing site, though the email vanishing would be annoying. What's to stop google doing this with any regular domain with user comments or content though? I blame the registrar - having heard this horror story I'd never host with google domains.

otterley 6y ago
Facebook uses fb.com as their corporate domain.

kohtatsu 6y ago

It most often has to do with browser sandboxing. See CORS, cookies, etc. It was a security flaw that began being addressed circa 2010.

SamyPesse 6y ago

We don't host user content under the gitbook.com, or at least we've stopped doing it a few years ago. User content is stored under *.gitbook.io, similar to GitHub. Google blocked all domains that contained "gitbook" in our account, even ones that are used for some infrastructure and are not accessible by the public. We don't know the exact reason for this, maybe they've blocked gitbook.com because we still have some redirect for content that was hosted under it years ago. And yes we are going to make changes to host our status page under another domain.

mbreese 6y ago
That’s really important to add to the story! I think that makes the rest of the story more troubling. Especially knowing that domains unrelated to the apparent user phishing code were affected. You might think of adding it to the post-mortem doc so that others don’t assume what I did.