5 ms·

This is pretty impressive. Keep in mind most of the team is on the west coast so this happened at 1am on a Sunday and they put up a post mortem within hours.

This is pretty impressive. Keep in mind most of the team is on the west coast so this happened at 1am on a Sunday and they put up a post mortem within hours. Obviously you would prefer it not happen at all, but that is a great response imo.

jgrahamc 14y ago
Everyone who was needed (management, network operations, and the technical support team) were woken up as soon as this happened. The small team who monitor things during the night were quickly calling people. Some folks physically went to the office to help answers phones; others drove to one of our data centers.
- larrys 14y ago
  "The small team" Didn't see this when I posted my comment. So there is a team monitoring not a single person?
  - TazeTSchnitzel 14y ago
    Uptime is a fundamental part of their business, of course it's a team.
    - larrys 14y ago
      What do you mean "of course it's a team" because "uptime is a fundamental part of their business"? That's an assumption on your part. Many negative events happen because of assumptions people make and things they take for granted. I ask a valid question. Do you know for a fact how many people monitor their network or what systems they have in place? Or whether they even simulate events like this to see if other members of the team are even reachable? Have you ever seen the constant testing that goes on at large organizations (such as the military) to make sure battle systems are ready to deploy when necessary? Note also that cloudflare charges $3000 per month for enterprise plans (in addition to free) and that their own website was taken offline.
    - david_shaw 14y ago
      > Uptime is a fundamental part of their business, of course it's a team Many businesses call roles a "team," even when it's just one person. I believe this is the case here, since the linked CloudFlare blog post states: > Someone from our operations team is monitoring our network 24/7.
- deleted 14y ago
  [deleted]
  - Dylan16807 14y ago
    Well I was in the process of upvoting you to a less faded gray and suggesting you be less rude when you're making a valid point, but "edit: downvotes? truth hurts." tips the line to where you just go ahead and keep on keeping on and get yourself banned have a nice day see you never.
  - deleted 14y ago
    [deleted]
larrys 14y ago
But this is not impressive: "Someone from our operations team is monitoring our network 24/7." "Someone" seems to indicate "1 person". Not "people are monitoring" but "someone". That's it, one person monitors the network? Like the single night guard at the warehouse?
- deleted 14y ago
  [deleted]
  - larrys 14y ago
    It's not an issue of whether they can get backup or not. (And I don't agree with that anyway a night guard can call 911 and get the police pretty quick.) The issue is whether there is a single person monitoring the network or several or even two. And is the coverage different during "working" hours? And what about the skills of the person monitoring at 1am vs. during the day? Seems that after something happens people wise up to the weak points. Remembering the case of a single air traffic controller in some towers and after something went wrong there was such shock that only one person was on duty with no backup.
- dandelany 14y ago
  Why would you assume this and judge the company on it, based entirely on an offhanded comment? You couldnt have given them the benefit of the doubt long enough to find another of several comments which clearly indicate there was a team of people working on the issue?
  - larrys 14y ago
    The question remains how many people actually are monitoring the network in order to call the first responders. Is it one person, two or five? And my use of "not impressive" was in reply to someone who said "impressive" but more importantly thought it was "impressive" that they put up a post mortem within hours. That's nice but it doesn't answer the question that I had. I stand behind my comment and re ask the question (since the info is ambiguous we have jgramhmc saying "small team who monitor things" and we have the blog post saying "Someone from our operations team is monitoring our network 24/7." I don't think it's unreasonable (in the interest of transparency) to know exactly the structure and # bodies of who monitors the network at any given time. What is the human point of failure in the system? I don't depend on cloudflare. But if I was running a mission critical operation and depended on them I might setup a site visit to actually get a feel of what is going on. As an aside back when the .org registry got started one of the dns servers sat in an open unguarded office under a desk accessible by the cleaning person. I saw it when I did a site visit. And of course if you've been around long enough you know there was a time when the root dns servers sat unguarded in university offices.

jgrahamc 14y ago

Everyone who was needed (management, network operations, and the technical support team) were woken up as soon as this happened. The small team who monitor things during the night were quickly calling people. Some folks physically went to the office to help answers phones; others drove to one of our data centers.

larrys 14y ago
"The small team" Didn't see this when I posted my comment. So there is a team monitoring not a single person?
- TazeTSchnitzel 14y ago
  Uptime is a fundamental part of their business, of course it's a team.
  - larrys 14y ago
    What do you mean "of course it's a team" because "uptime is a fundamental part of their business"? That's an assumption on your part. Many negative events happen because of assumptions people make and things they take for granted. I ask a valid question. Do you know for a fact how many people monitor their network or what systems they have in place? Or whether they even simulate events like this to see if other members of the team are even reachable? Have you ever seen the constant testing that goes on at large organizations (such as the military) to make sure battle systems are ready to deploy when necessary? Note also that cloudflare charges $3000 per month for enterprise plans (in addition to free) and that their own website was taken offline.
  - david_shaw 14y ago
    > Uptime is a fundamental part of their business, of course it's a team Many businesses call roles a "team," even when it's just one person. I believe this is the case here, since the linked CloudFlare blog post states: > Someone from our operations team is monitoring our network 24/7.
deleted 14y ago
[deleted]
- Dylan16807 14y ago
  Well I was in the process of upvoting you to a less faded gray and suggesting you be less rude when you're making a valid point, but "edit: downvotes? truth hurts." tips the line to where you just go ahead and keep on keeping on and get yourself banned have a nice day see you never.
- deleted 14y ago
  [deleted]

larrys 14y ago

But this is not impressive: "Someone from our operations team is monitoring our network 24/7." "Someone" seems to indicate "1 person". Not "people are monitoring" but "someone". That's it, one person monitors the network? Like the single night guard at the warehouse?

deleted 14y ago
[deleted]
- larrys 14y ago
  It's not an issue of whether they can get backup or not. (And I don't agree with that anyway a night guard can call 911 and get the police pretty quick.) The issue is whether there is a single person monitoring the network or several or even two. And is the coverage different during "working" hours? And what about the skills of the person monitoring at 1am vs. during the day? Seems that after something happens people wise up to the weak points. Remembering the case of a single air traffic controller in some towers and after something went wrong there was such shock that only one person was on duty with no backup.
dandelany 14y ago
Why would you assume this and judge the company on it, based entirely on an offhanded comment? You couldnt have given them the benefit of the doubt long enough to find another of several comments which clearly indicate there was a team of people working on the issue?
- larrys 14y ago
  The question remains how many people actually are monitoring the network in order to call the first responders. Is it one person, two or five? And my use of "not impressive" was in reply to someone who said "impressive" but more importantly thought it was "impressive" that they put up a post mortem within hours. That's nice but it doesn't answer the question that I had. I stand behind my comment and re ask the question (since the info is ambiguous we have jgramhmc saying "small team who monitor things" and we have the blog post saying "Someone from our operations team is monitoring our network 24/7." I don't think it's unreasonable (in the interest of transparency) to know exactly the structure and # bodies of who monitors the network at any given time. What is the human point of failure in the system? I don't depend on cloudflare. But if I was running a mission critical operation and depended on them I might setup a site visit to actually get a feel of what is going on. As an aside back when the .org registry got started one of the dns servers sat in an open unguarded office under a desk accessible by the cleaning person. I saw it when I did a site visit. And of course if you've been around long enough you know there was a time when the root dns servers sat unguarded in university offices.