10 ms·

Google on today’s massive Google+ spam influx: “We ran out of disk space”

67 points by suneliot 15y ago

staunch 15y ago
Running out of disk space is probably the single biggest category of server problem that occurs.
benologist 15y ago
I still have this happen, and every time I tell myself right, today I'm going to put in a warning system that'll stop this happening .... and then I clear up a ton of space and move on. HTTP.sys logging has been a real pain in my ass, every time I deploy a new server I forget to disable it and we do so much traffic it fills the drive completely overnight.
- deleted 15y ago
  [deleted]
- jackowayed 15y ago
  Sounds like you need to automate setting up a new server
  - benologist 15y ago
    That also doesn't quite happen enough to make it a priority!
  - pavel_lishin 15y ago
    Or at least automate the logging process. 5-minute log monitor that compresses old logs, and maybe moves them off the system onto a cloud server once space starts to run out, while also e-mailing the people responsible.
    - benologist 15y ago
      Yep that would help, notifications are built into about 1/2 my platform now ... gradually getting them more robust. :)
ChuckMcM 15y ago
Interesting. Given the minimum size of a Colossus cluster I find the explanation Vic gives unsatisfying. That being said I'm pretty impressed with the overall product, it is the best attempt yet to unseat Facebook, and I predict it will if Facebook can't come up with a credible response quickly. The killer feature is that its blended with Gmail, and since a lot of people keep gmail open all the time it means you get notified and you see stuff. Not as common to keep one's facebook page open.
- deleted 15y ago
  [deleted]
- mlinsey 15y ago
  since a lot of people keep gmail open ... Not as common to keep one's facebook page open. Do you have data on this? I'd think it would be close, if not the opposite. Edit for clarification: In my social circles, what you said is probably true, but I know many others use FB messages more than email (any email, not just gmail). Facebook has more pageviews total, and Facebook has probably around 3X as many active users (I couldn't find numbers at the same point in time, Gmail was at around 200 million last November as per the WSJ, Facebook was at 500 million last July and 750 million last week) Don't get me wrong, I agree with you that G+ notifications in Gmail (and on Google Search!) is hugely powerful, and will mean G+ engagement among its users will stay quite high. But when you talk about "unseating" Facebook, you have to first come to grips with just how entrenched it is, compared to social networks that rose and fell before it. It is so entrenched that Gmail integration alone will not be enough - Facebook is substantially bigger than Gmail.
  - ChuckMcM 15y ago
    You are correct in that I was generalizing when I should not have been. I've used Google Apps for work and since a lot of communication comes through email that means Gmail is open (or at least getting notifications with the talk gadget). I would not be surprised if you were correct that many folks leave Facebook open all the time.
    - 15y ago
craigmccaskill 15y ago
Some quick napkin math on numbers: If you're to believe Eric Schmidt when he says 'millions' and put that at 3 million users (being generous), guessing that each user uploads 20 megabytes of content (again, generous) thats: 3x10^6 × 20 MB 6x10^7 MB or 60TB Sanity check or does that not seem like a lot of resources to allocate to a project of this size? Edit: formatting
- starwed 15y ago
  I don't think this has anything to do with how much content a user uploads. They specifically said that “For about 80 minutes we ran out of disk space on the service that keeps track of notifications.” So probably they had allocated some amount that made perfect sense while testing, but was too low for the full rollout that seems to (effectively) be happening now.
- Maakuth 15y ago
  Maybe some heavy logging in this field test phase could explain it? Indeed 60TB sounds like very little. I thought Google would use their GFS system to back this kind of things so the disk space wouldn't be an issue. What do I know.
- carbonica 15y ago
  60TB? That's about a couple grand worth of space. Why would that be a lot of space to Google?
  - whiskers 15y ago
    Even using 3TB consumer grade SATA disks you'd need 20 of them, an enclosure for 20 disks costs a lot more than a couple of grand.
    - moe 15y ago
      Not really. http://shopping.yahoo.com/721280117-supermicro-sc216-e2-r900ub-system-case/ http://shopping.yahoo.com/721280117-supermicro-sc216-e2-r900...
oflannabhra 15y ago
So I guess "field testing" is the new beta testing? Google watered down beta by applying it to finished products to the degree that they've had to invent a new term to fill its function.
adaml_623 15y ago
Shouldn't there be a Beta tag on the Google Plus icon. Perhaps they left it on Gmail for so long they didn't think anybody would pay attention to it anymore.
robryan 15y ago
Seems today and yesterday (for me and those I have invited anyway) there has been no issue with creating an account straight away. So I'm guessing this has sent the total users on the service up pretty fast.
senthilnayagam 15y ago
running out of diskspace is devops 101, it is high time these colossus startups start publishing which CMM level standard they meet
- vegai 15y ago
  Are you joking?
  - senthilnayagam 15y ago
    no, I am pretty serious, down votes dont change reality. there are apps/services which are signing on 99.999% availability. if after having a expertise in managing 200million+ gmail accounts, I would consider it a bad fumble at 5 million+ users
    - laz 15y ago
      Mentioning devops undermines your credibility. What service is 5 nines? How is that measured? In general, Google SRE just gets it done. Sometimes people screw up. It happens.
    - vegai 15y ago
      Oh, I was referring to you asking about so-called "CMM level" when we're talking about Google, not some second-hand suit-run IT company.
deleted 15y ago
[deleted]
tybris 15y ago
Not so long ago hard disk space was abundant, then programmers realized hard disk space was abundant.

67 points by suneliot 15y ago

Running out of disk space is probably the single biggest category of server problem that occurs.

I still have this happen, and every time I tell myself right, today I'm going to put in a warning system that'll stop this happening .... and then I clear up a ton of space and move on. HTTP.sys logging has been a real pain in my ass, every time I deploy a new server I forget to disable it and we do so much traffic it fills the drive completely overnight.

deleted 15y ago
[deleted]
jackowayed 15y ago
Sounds like you need to automate setting up a new server
- benologist 15y ago
  That also doesn't quite happen enough to make it a priority!
- pavel_lishin 15y ago
  Or at least automate the logging process. 5-minute log monitor that compresses old logs, and maybe moves them off the system onto a cloud server once space starts to run out, while also e-mailing the people responsible.
  - benologist 15y ago
    Yep that would help, notifications are built into about 1/2 my platform now ... gradually getting them more robust. :)

ChuckMcM 15y ago

Interesting. Given the minimum size of a Colossus cluster I find the explanation Vic gives unsatisfying. That being said I'm pretty impressed with the overall product, it is the best attempt yet to unseat Facebook, and I predict it will if Facebook can't come up with a credible response quickly. The killer feature is that its blended with Gmail, and since a lot of people keep gmail open all the time it means you get notified and you see stuff. Not as common to keep one's facebook page open.

deleted 15y ago
[deleted]
mlinsey 15y ago
since a lot of people keep gmail open ... Not as common to keep one's facebook page open. Do you have data on this? I'd think it would be close, if not the opposite. Edit for clarification: In my social circles, what you said is probably true, but I know many others use FB messages more than email (any email, not just gmail). Facebook has more pageviews total, and Facebook has probably around 3X as many active users (I couldn't find numbers at the same point in time, Gmail was at around 200 million last November as per the WSJ, Facebook was at 500 million last July and 750 million last week) Don't get me wrong, I agree with you that G+ notifications in Gmail (and on Google Search!) is hugely powerful, and will mean G+ engagement among its users will stay quite high. But when you talk about "unseating" Facebook, you have to first come to grips with just how entrenched it is, compared to social networks that rose and fell before it. It is so entrenched that Gmail integration alone will not be enough - Facebook is substantially bigger than Gmail.
- ChuckMcM 15y ago
  You are correct in that I was generalizing when I should not have been. I've used Google Apps for work and since a lot of communication comes through email that means Gmail is open (or at least getting notifications with the talk gadget). I would not be surprised if you were correct that many folks leave Facebook open all the time.
  - 15y ago

craigmccaskill 15y ago

Some quick napkin math on numbers: If you're to believe Eric Schmidt when he says 'millions' and put that at 3 million users (being generous), guessing that each user uploads 20 megabytes of content (again, generous) thats: 3x10^6 × 20 MB 6x10^7 MB or 60TB Sanity check or does that not seem like a lot of resources to allocate to a project of this size? Edit: formatting

starwed 15y ago
I don't think this has anything to do with how much content a user uploads. They specifically said that “For about 80 minutes we ran out of disk space on the service that keeps track of notifications.” So probably they had allocated some amount that made perfect sense while testing, but was too low for the full rollout that seems to (effectively) be happening now.
Maakuth 15y ago
Maybe some heavy logging in this field test phase could explain it? Indeed 60TB sounds like very little. I thought Google would use their GFS system to back this kind of things so the disk space wouldn't be an issue. What do I know.
carbonica 15y ago
60TB? That's about a couple grand worth of space. Why would that be a lot of space to Google?
- whiskers 15y ago
  Even using 3TB consumer grade SATA disks you'd need 20 of them, an enclosure for 20 disks costs a lot more than a couple of grand.
  - moe 15y ago
    Not really. http://shopping.yahoo.com/721280117-supermicro-sc216-e2-r900ub-system-case/ http://shopping.yahoo.com/721280117-supermicro-sc216-e2-r900...

oflannabhra 15y ago

So I guess "field testing" is the new beta testing? Google watered down beta by applying it to finished products to the degree that they've had to invent a new term to fill its function.

adaml_623 15y ago

Shouldn't there be a Beta tag on the Google Plus icon. Perhaps they left it on Gmail for so long they didn't think anybody would pay attention to it anymore.

robryan 15y ago

Seems today and yesterday (for me and those I have invited anyway) there has been no issue with creating an account straight away. So I'm guessing this has sent the total users on the service up pretty fast.

senthilnayagam 15y ago

running out of diskspace is devops 101, it is high time these colossus startups start publishing which CMM level standard they meet

vegai 15y ago
Are you joking?
- senthilnayagam 15y ago
  no, I am pretty serious, down votes dont change reality. there are apps/services which are signing on 99.999% availability. if after having a expertise in managing 200million+ gmail accounts, I would consider it a bad fumble at 5 million+ users
  - laz 15y ago
    Mentioning devops undermines your credibility. What service is 5 nines? How is that measured? In general, Google SRE just gets it done. Sometimes people screw up. It happens.
  - vegai 15y ago
    Oh, I was referring to you asking about so-called "CMM level" when we're talking about Google, not some second-hand suit-run IT company.

deleted 15y ago

[deleted]

tybris 15y ago

Not so long ago hard disk space was abundant, then programmers realized hard disk space was abundant.