7 ms·

Facebook Workplace co-founder launches downtime fire alarm Kintaba

84 points by quartz 7y ago

duxup 7y ago
Man in that screenshot it LOOKS like facebook a bit... too much IMO. Anyway it's an interesting idea. I worked in a support center for decades and demands for updates and managing P1 type situations, giving updates to the dozen interested parties (each of whom were competent in some ways, less so in others), managing misinformation, and the varying politics was a constant hassel for folks who were technical. There were times where "oh man my phone just went down" (I unplugged it). That's no small thing to try to deal with, with software.
- quartz 7y ago
  Kintaba cofounder here-- thanks for the feedback! We really wanted the interface to feel comfortable for non-technical folks who needed to stay up to date on incidents so we focused on bringing as much of a human element as possible into the dashboard. Hopefully we can find the right balance of friendliness and informativeness, we certainly don't intend to become a social network (but we would love it if we were as easy to use as something like Slack or Discord)... We experienced the same pain you're describing with keeping others updated and the politics (and subpolitices and subsubpolitics) that come out of major incidents over our careers. Luckily we also saw companies that did it right, usually with custom built tooling. Our biggest discovery was the more open you make the whole incident process the better everyone understands the work being done and also the less of those insidious little subpolitical conversations happen where facts are skewed and people are blamed for process challenges. It's definitely a big hill to climb.
  - duxup 7y ago
    Yeah we used to do the political stuff via email updates at a place I was at. It wasn't the worst way but also not a "system" for sure. Half the battle with political stuff was really was defining the problem in a way that everyone outside engineering understood, and keeping everyone updated / aware of what was factually (not rumors or panic) going on, what was known, not known, what was happening, and when they would get their next update. And that's not even the technical stuff where folks here are asking about recording conference calls and etc ;)
- giancarlostoro 7y ago
  > Man in that screenshot it LOOKS like facebook a bit... too much IMO. Looks like StackOverflow to me but clean. I like the UI. I don't know if I will ever see that utility where I work since we wikify things like this post-chaos. The office I work in is a small enough team to where we have plenty of flexibility.
  - duxup 7y ago
    To me Facebook and most social media is a sort of comment chaos where nobody really reads anything and just spouts some stuff. Thus my cognitive dissonance with that and this system. Like when I think of dealing with a situation I see that and think of some sales guy going off in his comment about OMG THIS IS JUST LIKE THAT OTHER TIME WHY ISN'T THIS FIXED!?!!? Now the whole comment thread is off track because sales guy brought his own FUD to the game, and god help us all if someone brings it up on a conference call as fact. Now obviously that is not the fault of the UI if you've got a sales guy does that, but as things scale up organizationally that kinda thing ... happens ;) Granted this system might be intended for engineering teams, but it's also something that inevitably leaks out when folks find out it is there.
    - giancarlostoro 7y ago
      Heh interesting point there, I wonder if an HN like comment system would help (let users upvote highly relevant material), and maybe allowing supervisors to pin things. We use pinning a lot on Slack so people can quickly find important links or messages.
      - duxup 7y ago
        That might be handy, also probably something like an "athroative user" could respond and even minimize the view or visibility or something. Even if they don't have an answer some way of making sure FUD doesn't take off too quickly. I'd say a high % of managing any situation and the politics that follow is accurately defining / revising the problem and making sure everyone is on the same page with it. Dodging / handing the inaccurate information leaks sometimes is a big job. Granted that's all office politics stuff and not just technical... still gotta solve the problem ;)
- Lammy 7y ago
  Not as much as Phabricator does :) https://phacility.com/phabricator/ https://phacility.com/phabricator/
mjayhn 7y ago
Being able to take a snapshot of the chat/repo pushes/CI+CD jobs/grafanas and everything else going on during an outage and separate it by an incident type/tag/hostname/etc for later reference (to write RFOs, to more quickly solve it if it occurs again, to write documentation on, etc) is something I've wanted for awhile so this seems really interesting. I'm sure that stuff exists (I think Datadog sort of does this, etc) but I've yet to work anywhere that doesn't just create some #shtf-$date slack channel which eventually gets lost in the black hole due to cost prohibition or time required to get it going while a fire is going on.
- quartz 7y ago
  100%. Finding ways to identify the relevant data related to an incident is something that requires a lot of additional integrations that we're actively working on. One thing Kintaba is really good at right now is wiring your slack channel directly into the activity log so it's properly attached to the incident and ultimately the postmortem. This helps avoid that channel getting lost over time, but there's still lots to do for sure in organizing all that data to make it more useful after the fact (one thing we currently support is #tagging for quick search within the log).
- duxup 7y ago
  Agreed, a repo snapshot, every log you can think of, all that good stuff that you do manually to figure out if this = that. Few things are as annoying as going back in repo time and trying to figure out if you're in quite the same place / right place.
beeschlenker 7y ago
Part II: Impeachment Is A Diversion & Delay - Blocking of the "impeachment" witnesses was collusion planned before the new year. Listen to an FBI agent disclosure from January 1st, 2O2O here. President Trump to resign late summer securing election for Biden. See latest updates. Here is the zip file, which was also made available in the 3Jan2O2O update. The file within is VID_20200101_201948.mp3. Turn up the volume and put on headphones. BB10Mp3Footage31Dec1Jan.zip 122.4mb https://drive.google.com/file/d/1IXOOhQhHybwky8Z5pGdr9ZXhWpIUazmi/view?usp=sharing https://drive.google.com/file/d/1IXOOhQhHybwky8Z5pGdr9ZXhWpI... The dialogue about the impeachment starts near the beginning. Having Biden in the White House is as good as Trump or anyone else in their organization. Obviously Schiff and Nadler pledged their allegiance to the organization by raping boys on the record, with their task being to drag out an impeachment designed to obstruct and delay any real efforts to remove the President, thus keeping Trump in power. The witness blocking was to cause an apparent uproar delaying things with legal actions until late Summer. Soon after, the President would resign, leaving any other candidate with not enough time or support to compete with an opportunistic Biden, who is as good as Trump or any other Illuminati friendly politician in the Presidency. 162 page PDF [last updated: Feb|4|2O2O]: https://drive.google.com/file/d/1S7T_kDv48E40eHzus6CTXHxcm0W2qxVc/view?usp=sharing https://drive.google.com/file/d/1S7T_kDv48E40eHzus6CTXHxcm0W... Previously reported: \Wag The Dog: first was feigned impeachment hearings meant to obstruct, now an attack on Iranians in Iraq. Here is what they are trying to distract from & cover up to retain power. $100+ billion in bribes to the highest offices in this country. 815+ deaths from child rapes to prove loyalty! See the latest PDF updates: FBI Director Wray, AG Barr, SoD Shanahan, & SoS Pompeo each raped boys and were paid billions in bribes for a Soros & Koch funded child rape org. So did Trump & his "impeachment" team Nadler,Schiff,Mueller.So did media moguls Redstone,Murdoch,Moonves. What are they trying to set up? Who can arrest them since they are all bribed and in on it ? Their strategy to stay in every office and obstruct until forced to leave no matter what. Feigning impeachment: see page 13O. Tpwrpoinqkl r cercq meq ew, ewopjmrvcpoer. \\if;Download the video/audio file, put on headphones and turn up the volume. You will hear these people committing these crimes. Audio was broadcast into my apartment by outdated surveillance equipment illegally embedded within my walls. This very same technology was being used to broadcast me to the internet for five years without my consent. I own this footage. Please use this to prosecute all found within. Note:: I am obliviously speaking throughout the video, and it can be quite loud at times relative to the desired content. The are dozens more links, including these, that can be found in this PDF last updated 4 FEBRUARY 2O2O: https://drive.google.com/file/d/1S7T_kDv48E40eHzus6CTXHxcm0W2qxVc/view?usp=sharing https://drive.google.com/file/d/1S7T_kDv48E40eHzus6CTXHxcm0W... All members of the "Illuminati"; "....an underground organization of homosexuals and child rapists..." (from pg 26: Barack Obama with Jack Dorsey). President Donald Trump: Demands a $4 billion dollar bribe here at 10:18am 4thJan2019: 3JanCh3_900-1100.avi https://drive.google.com/file/d/1Grdr8xF2psKNsuYlEnl9dIRV-77YG0Vr/view?usp=sharing https://drive.google.com/file/d/1Grdr8xF2psKNsuYlEnl9dIRV-77... 3JanCh2_900-1100-avi https://drive.google.com/file/d/1LUmVygl_q0XVs8h2cWr8jZl-24fY0vBB/view?usp=sharing https://drive.google.com/file/d/1LUmVygl_q0XVs8h2cWr8jZl-24f... 3JanCh4_1000-1100.mp3 https://drive.google.com/file/d/1ZpP1pJbJakBgg-y-MWNozTxp3wJojHge/view?usp=sharing https://drive.google.com/file/d/1ZpP1pJbJakBgg-y-MWNozTxp3wJ... President Trump rapes and kills twelve boys, including five boys in a "who can rape five boys to death the fastest" game: 14JanCh3_600.mp3 https://drive.google.com/file/d/1ufPmglde9Mep0m6xYMJ9c4TWTjj258zB/view?usp=sharing https://drive.google.com/file/d/1ufPmglde9Mep0m6xYMJ9c4TWTjj... 14JanCh2_600-700.mp3 https://drive.google.com/file/d/136qLJdEn8eCs9tI4QtIxl4opW_Lw_gCF/view?usp=sharing https://drive.google.com/file/d/136qLJdEn8eCs9tI4QtIxl4opW_L... Speaker of the House Nancy Pelosi: Accepting a $3 billion dollar bribe at 1033 am on the 17 Jan 2019 to ensure Asian boys can get through the border at "Monterrey" undocumented to be raped: 17JanCh3_949-1100.avi https://drive.google.com/file/d/1eodHu4o5Cm3xEWhDqipSuTj-M1CZUvmE/view?usp=sharing https://drive.google.com/file/d/1eodHu4o5Cm3xEWhDqipSuTj-M1C... 17JanCh4_1017-1100.avi https://drive.google.com/file/d/1y-nWEQbempkVZSz230j9wTyduZNjDmnk/view?usp=sharing https://drive.google.com/file/d/1y-nWEQbempkVZSz230j9wTyduZN... Speaker Nancy Pelosi also "preps" boys with First Lady Melania Trump, defined as in she performs oral sex on the boys’ penis and anus, as a child rapist like Henry Porter would, while trying to remove fecal matter from the boy prior to handing them over to be raped and then subsequently murdered, for Supreme Court Justice Samuel Alito, who apparently decides he would rather just have ten billion dollars instead. US Attorney for Western New York James Kennedy rapes these boys instead: 12JanCh3_1533-1638.mp3 https://drive.google.com/file/d/1AgFkDsbPbI4b5Xd3Wbz2EVNNx25kW2Oz/view?usp=sharing https://drive.google.com/file/d/1AgFkDsbPbI4b5Xd3Wbz2EVNNx25... Attorney General William Barr with FBI Deputy Director Christopher Wray raped and killed boys for billions in bribes in Buffalo, NY on the 17Jan2019 at 7:50am: 18JanCh4_700mp3 https://drive.google.com/file/d/1UIdZkS5ZVksZdHYsnHk2t5losi0N0Bp7/view?usp=sharing https://drive.google.com/file/d/1UIdZkS5ZVksZdHYsnHk2t5losi0... 18JanCh2_700.mp3 OHoiner elrk vcercve, ec;mvcpoerqm. https://drive.google.com/file/d/1DFK8IAxm5pQVqZv9L518nfgP7_odB2Um/view?usp=sharing https://drive.google.com/file/d/1DFK8IAxm5pQVqZv9L518nfgP7_o... 18JanCh3_725-.mp3 https://drive.google.com/file/d/1DG5ej59Ic8RT9UhbyMdwT0BDcKIifGGm/view?usp=sharing https://drive.google.com/file/d/1DG5ej59Ic8RT9UhbyMdwT0BDcKI... Secretary of State Michael Pompeo and Secretary of Defense Patrick Shanahan each raped and killed boys on 5thJan'19 at 17:39 for billions in bribes: Gohponklm,s, lkndcwolmn. 5JanCh3_1600 1800.mp3 https://drive.google.com/file/d/1ghgmNrQvJ8WfJ2TsDVx1ruDU36hrhx6S/view?usp=sharing https://drive.google.com/file/d/1ghgmNrQvJ8WfJ2TsDVx1ruDU36h... 5JanCh2_1721-1818.mp3 https://drive.google.com/file/d/1eSlD4otX4KZqWXboQM92Mu-6J02Mjwrg/view?usp=sharing https://drive.google.com/file/d/1eSlD4otX4KZqWXboQM92Mu-6J02... Leaders of the "impeachment" effort Jerrold Nadler, Robert Mueller, and Adam Schiff all rape and kill boys between 11:20pm and 1:10am: 14JanCh4_2300-0000.mp3 Nadler starts at about twenty mins in- https://drive.google.com/file/d/1Kuvv2Zmbw5Jw7onbRI2hCZ0M8FU66GuU/view?usp=sharing https://drive.google.com/file/d/1Kuvv2Zmbw5Jw7onbRI2hCZ0M8FU... 14JanCh2_2304-2359.mp3 https://drive.google.com/file/d/1nofp5xF-aXXcCSgQVwj30KlzE9WLr1x1/view?usp=sharing https://drive.google.com/file/d/1nofp5xF-aXXcCSgQVwj30KlzE9W... Mueller at 12:25am, next is Schiff who starts 12:55-ish: 15JanCh2_000-100.mp3 https://drive.google.com/file/d/1EsmHfguwBuo2PbavJ1WYyhiML62tgCkj/view?usp=sharing https://drive.google.com/file/d/1EsmHfguwBuo2PbavJ1WYyhiML62... 15JanCh2_100-200.mp3 https://drive.google.com/file/d/1NZnWRnBryalNQu2yJmfJUdS2pA_rGSFH/view?usp=sharing https://drive.google.com/file/d/1NZnWRnBryalNQu2yJmfJUdS2pA_... 15JanCh4_000-100.mp3 https://drive.google.com/file/d/1ZEDJR6jb6ARpcNnWJTokBUKb2J2OQnwZ/view?usp=sharing https://drive.google.com/file/d/1ZEDJR6jb6ARpcNnWJTokBUKb2J2... 15JanCh4_100-200.mp3 https://drive.google.com/file/d/173aYWvWHH4VGht1h_2nM0IMdw746LeCL/view?usp=sharing https://drive.google.com/file/d/173aYWvWHH4VGht1h_2nM0IMdw74... Complete Media Protection: Lester Holt, of NBC NightlyNews, apparently a member of the Illuminati since the 80's, along with ABC Nightly News lead anchor David Muir, stop over to the Porter studio in Buffalo, New York on 14Jan2019 at 5:00 am. They both rape and kill about two dozen boys by 6:00 am. Muir starts around 5.15am, then Holt about 5:38 am. Multi-billionaire Rupert Murdoch, owner of News Corp and also Fox Corporation, takes his turn after Holt. Video links below: 14JanCh3_500-601.avi zijnoijrpotmebr https://drive.google.com/file/d/1i7NKepeyG_FfdQRrM7KsnFOZOOX3o7UL/view?usp=sharing https://drive.google.com/file/d/1i7NKepeyG_FfdQRrM7KsnFOZOOX... 14JanCh2_530-600.avi https://drive.google.com/file/d/1NZzgN5ilI7ToroU5cfqMaL4o2u1RwidV/view?usp=sharing https://drive.google.com/file/d/1NZzgN5ilI7ToroU5cfqMaL4o2u1... Adding to the media protection and reason this is not picked up by the media, CBS and Viacom owner Sumner Redstone and Leslie Moonves rape and kill boys following the President. 14JanCh3_700.avi https://drive.google.com/file/d/10XDw6x3ldnnQiq7oIjpdYVENyXaB4aI0/view?usp=sharing https://drive.google.com/file/d/10XDw6x3ldnnQiq7oIjpdYVENyXa... 14JanCh2_700-800.avi https://drive.google.com/file/d/1NS_e6AzEZ05wnfljkGMETGU5CWYKfRDp/view?usp=sharing https://drive.google.com/file/d/1NS_e6AzEZ05wnfljkGMETGU5CWY... 161p PDF [last update: Feb|4|2O2O]: https://drive.google.com/file/d/1S7T_kDv48E40eHzus6CTXHxcm0W2qxVc/view?usp=sharing https://drive.google.com/file/d/1S7T_kDv48E40eHzus6CTXHxcm0W... \\. Please repost in USA! Post gets censored in US Recently more relevant: From page 49, Senator Mitch McConnell: At 1632 Senator Mitch McConnell checks into the Porter camera system inquiring if he can be part of the "eviction" for $10 million dollars. He is informed by group members that there are enough people for the event already and his participation is not necessary. At 1634 McConnell states "I fucked 15 kids, how am I not getting paid by you?" He is dismissed by Donald Reeves with "I think that will be all Mr. McConnell." 13JanCh3_1600-1700.avi https://drive.google.com/file/d/1L7bqOpvaEWmLiJpMhJNQDrfsQAHtsX8-/view?usp=sharing https://drive.google.com/file/d/1L7bqOpvaEWmLiJpMhJNQDrfsQAH...
Jestar342 7y ago
> Verizon Media works with select partners that do not participate in the Interactive Advertising Bureau's Transparency and Consent Framework, or Google framework. All our foundational partners require you to manage your choices directly through their privacy policies. Click on each partner below to access their privacy policy Yeah. <closes-tab/>
jedberg 7y ago
At Netflix we build internal tools and systems that basically did all this stuff, so it looks interesting, out of the box. One of the nicest things was that we use a single deployment tool for almost all deployments, and it could insert the deployments into the timeline so we could see every deploy both during and before the incident. But the biggest issue was voice. We always had a conference call running for incidents because some people would be in a place where they couldn't open chat (driving to the office or similar) and it was always a pain to integrate the voice data. We got to a point where a recording of the incident call and a transcript could be added to the incident, but during the call the Call Leader usually had to quickly type the voice highlights into the chat. I'd love to see a real time voice transcription as a feature. But it would have to be pretty good to not just get in the way.
- giancarlostoro 7y ago
  Interesting, are these systems made by the same team, what kind of technical stack do they use? I'm always curious as to what some use for internal tooling. I've been using Django / CherryPy for internal and external solutions at work for a few years. Python helps me to stay productive.
  - jedberg 7y ago
    I was on the reliability team, and we made some of them. One of our sister teams (same Director) was the monitoring team, and they made some features in the monitoring tools that enabled real time export. And then the team that made the deployment tool was also a sister team that made exports easy. So it was a few teams all reporting to the same Director. We even had the customer service tools reporting to the same director so we could get some real time info from them (and to them). Each team built in what they were most comfortable. Most of the tools were built in Java, but my team built in Python (mainly because I write in Python and biased towards hiring people who also like to write in Python).
    - giancarlostoro 7y ago
      Heh, we're a Java and Python shop to a point, we have done a bit of everything here, but my first project here was to rewrite a legacy Java JSP application and I was given some freedom as to what to rewrite it in, I chose Python and CherryPy. No regrets since. Sidenote: Other reason I asked is cause apparently both Hulu and Netflix are listed on the CherryPy docs as orgs using them.
ryanjodonnell 7y ago
What does it mean to be co-founder of facebook workplace? Arent you just an employee of facebook?
- duxup 7y ago
  I suppose it implies a leadership role where that department was somewhat autonomous and operated a bit 'like' a founder of a startup. That's what I would assume.
  - AgloeDreams 7y ago
    Exactly, Much like Jon Rubinstein being the inventor of the iPod or Panos Panay being the leader of the Surface Group.
- crispyporkbites 7y ago
  No. Today I realised that I am also a co-founder, in fact you probably are too! My investors are a bit tough, I get no equity and no seats on the board (there is no board). Vacation and pension’s good though.
- rdslw 7y ago
  Yup. Ego massage. Here clickbait. Paul Graham wrote: never do things for money or ego, but techcrunch clickbaiters writing about facebook employees, sorry, founders, supposedly dont agree. P.S. this whole title reads like from git manpage generator.
- zeroonetwothree 7y ago
  It’s some good title inflation for sure
mherdeg 7y ago
Yeah feels like there's a lot of room for startups in this space (Blameless has a great demo, too). No one wants to rewrite PagerDuty internally -- why are people all writing their bespoke incident management, response, review, and reporting toolchains internally too?
- jedberg 7y ago
  > why are people all writing their bespoke incident management, response, review, and reporting toolchains internally too? Because the good ones are tightly integrated with the rest of the internal tooling. To use a 3rd party incident management tool usually means you have to run your operations they way they expect to get maximum value. A lot of times its easier to build it yourself based on how you operate. However, as the way people operate become more standardized, the third party tools will become more useful.
  - mherdeg 7y ago
    Yeah, agree, this is probably the key sticking point: the 3rd party toolchain doesn't do [some key thing] the stakeholders in my process have found valuable. So I end up run some scenario analysis and find that the 3rd party toolchain doesn't shave off enough developer time & ongoing support cost from the rest of the operational work I'm going to have to do anyway to manage an incident. And I load the vendor's web page every couple of months and wistfully wish my process were uncomplicated enough to use their tools. Here's hoping that the calculus changes either as these tools grow more robust or, like you say, as people begin to manage their software systems in less bespoke ways. For example, during an incident we'd like to know "what changed between time [X] and [Y]" (deployments, configuration, experiments, other service outages) while we're trying to determine a root cause and fix the problem. And much later, after the incident, we'd like to auto-compute a metric like "what is the change success rate of [Y service / services owned by team Z]?". This aren't complicated concepts -- it's not like we're trying to reason about causality with machine learning to reduce time-to-resolve for incidents! Still, our incident-management tool will really behave better if it's aware of our what-changed-at-time-X tooling and our incidents-caused-by-changes reporting. If this is an external tool, yikes, now our incident tool has sprung awareness of changes and reporting?!
gooeyblob 7y ago
Looks interesting! A couple notes: - the verification email went to my spam folder on Gmail - acknowledged is misspelled on this image https://kintaba.com/images/collab_splash.png https://kintaba.com/images/collab_splash.png
- quartz 7y ago
  Yikes-- fixed the spelling, thank you! Thanks for the spamboxing report. Seems gmail isn't a fan of us today... working on it now.
  - ThePowerOfFuet 7y ago
    Get your SPF, DKIM, and DMARC up to snuff. Dmarcian is a good tool for that. (No connection, just a happy customer.)
lucb1e 7y ago
This site has some gdpr-violating we-want-it-all compulsory tracking wall it seems, but from the comments, do I understand correctly that a "downtime fire alarm" is a monitoring service that alerts sysadmins at home when a service is down?
wackget 7y ago
You should know that Techcrunch URLs are blocked for anyone using uMatrix, as per the following discussion: https://news.ycombinator.com/item?id=22228159 https://news.ycombinator.com/item?id=22228159 Just making you aware, as the site causes lots of problems and is not GDPR compliant. Top comment on the thread above: "Yahoo/Verizon is cancer and should die in a fire."
- lostlogin 7y ago
  Every thread has a “this website breaks x” section and usually I bypass it. Breaking the ability to go back (iOS) is so gross and many sites seem to do it. Why?
deleted 7y ago
[deleted]
chadlavi 7y ago
I know all TechCrunch articles are paid placements but this one felt the paid-placementest in a long time.
- echan00 7y ago
  Not true at all. My previous startup was written about and we did not pay a dime.
  - deleted 7y ago
    [deleted]
lainga 7y ago
With former employees from Kinja and Altaba?
jamisteven 7y ago
I can see someone like Servicenow or Splunk wanting to acquire this. Decent idea for organizations that dont have existing frameworks around these types of things. Where I see it not working would be in places where the use of 3rd party apps are prevalent. We have many of these at my place of work (Finance, 70k employees), one of the top apps used by the business to make money is a total black box, when something goes wrong with it there isnt much we can do besides send off the core dumps to the vendor and wait for analysis, with no 2 scenarios ever being the same root cause.
motakuk 7y ago
There is one more concept in this field: agree with collaborative nature of incidents and work with them in Slack directly (implemented in https://amixr.io https://amixr.io)
sbryant8 7y ago
How does this differentiate from something like Opsgenie? They already have a built-in incident command center that can notify everyone in an org, Slack integration and postmortems. Would be interesting to see how all of the features stack up to one another.

84 points by quartz 7y ago

Man in that screenshot it LOOKS like facebook a bit... too much IMO. Anyway it's an interesting idea. I worked in a support center for decades and demands for updates and managing P1 type situations, giving updates to the dozen interested parties (each of whom were competent in some ways, less so in others), managing misinformation, and the varying politics was a constant hassel for folks who were technical. There were times where "oh man my phone just went down" (I unplugged it). That's no small thing to try to deal with, with software.

quartz 7y ago
Kintaba cofounder here-- thanks for the feedback! We really wanted the interface to feel comfortable for non-technical folks who needed to stay up to date on incidents so we focused on bringing as much of a human element as possible into the dashboard. Hopefully we can find the right balance of friendliness and informativeness, we certainly don't intend to become a social network (but we would love it if we were as easy to use as something like Slack or Discord)... We experienced the same pain you're describing with keeping others updated and the politics (and subpolitices and subsubpolitics) that come out of major incidents over our careers. Luckily we also saw companies that did it right, usually with custom built tooling. Our biggest discovery was the more open you make the whole incident process the better everyone understands the work being done and also the less of those insidious little subpolitical conversations happen where facts are skewed and people are blamed for process challenges. It's definitely a big hill to climb.
- duxup 7y ago
  Yeah we used to do the political stuff via email updates at a place I was at. It wasn't the worst way but also not a "system" for sure. Half the battle with political stuff was really was defining the problem in a way that everyone outside engineering understood, and keeping everyone updated / aware of what was factually (not rumors or panic) going on, what was known, not known, what was happening, and when they would get their next update. And that's not even the technical stuff where folks here are asking about recording conference calls and etc ;)
giancarlostoro 7y ago
> Man in that screenshot it LOOKS like facebook a bit... too much IMO. Looks like StackOverflow to me but clean. I like the UI. I don't know if I will ever see that utility where I work since we wikify things like this post-chaos. The office I work in is a small enough team to where we have plenty of flexibility.
- duxup 7y ago
  To me Facebook and most social media is a sort of comment chaos where nobody really reads anything and just spouts some stuff. Thus my cognitive dissonance with that and this system. Like when I think of dealing with a situation I see that and think of some sales guy going off in his comment about OMG THIS IS JUST LIKE THAT OTHER TIME WHY ISN'T THIS FIXED!?!!? Now the whole comment thread is off track because sales guy brought his own FUD to the game, and god help us all if someone brings it up on a conference call as fact. Now obviously that is not the fault of the UI if you've got a sales guy does that, but as things scale up organizationally that kinda thing ... happens ;) Granted this system might be intended for engineering teams, but it's also something that inevitably leaks out when folks find out it is there.
  - giancarlostoro 7y ago
    Heh interesting point there, I wonder if an HN like comment system would help (let users upvote highly relevant material), and maybe allowing supervisors to pin things. We use pinning a lot on Slack so people can quickly find important links or messages.
    - duxup 7y ago
      That might be handy, also probably something like an "athroative user" could respond and even minimize the view or visibility or something. Even if they don't have an answer some way of making sure FUD doesn't take off too quickly. I'd say a high % of managing any situation and the politics that follow is accurately defining / revising the problem and making sure everyone is on the same page with it. Dodging / handing the inaccurate information leaks sometimes is a big job. Granted that's all office politics stuff and not just technical... still gotta solve the problem ;)
Lammy 7y ago
Not as much as Phabricator does :) https://phacility.com/phabricator/ https://phacility.com/phabricator/

mjayhn 7y ago

Being able to take a snapshot of the chat/repo pushes/CI+CD jobs/grafanas and everything else going on during an outage and separate it by an incident type/tag/hostname/etc for later reference (to write RFOs, to more quickly solve it if it occurs again, to write documentation on, etc) is something I've wanted for awhile so this seems really interesting. I'm sure that stuff exists (I think Datadog sort of does this, etc) but I've yet to work anywhere that doesn't just create some #shtf-$date slack channel which eventually gets lost in the black hole due to cost prohibition or time required to get it going while a fire is going on.

quartz 7y ago
100%. Finding ways to identify the relevant data related to an incident is something that requires a lot of additional integrations that we're actively working on. One thing Kintaba is really good at right now is wiring your slack channel directly into the activity log so it's properly attached to the incident and ultimately the postmortem. This helps avoid that channel getting lost over time, but there's still lots to do for sure in organizing all that data to make it more useful after the fact (one thing we currently support is #tagging for quick search within the log).
duxup 7y ago
Agreed, a repo snapshot, every log you can think of, all that good stuff that you do manually to figure out if this = that. Few things are as annoying as going back in repo time and trying to figure out if you're in quite the same place / right place.

beeschlenker 7y ago

Part II: Impeachment Is A Diversion & Delay - Blocking of the "impeachment" witnesses was collusion planned before the new year. Listen to an FBI agent disclosure from January 1st, 2O2O here. President Trump to resign late summer securing election for Biden. See latest updates. Here is the zip file, which was also made available in the 3Jan2O2O update. The file within is VID_20200101_201948.mp3. Turn up the volume and put on headphones. BB10Mp3Footage31Dec1Jan.zip 122.4mb https://drive.google.com/file/d/1IXOOhQhHybwky8Z5pGdr9ZXhWpIUazmi/view?usp=sharing https://drive.google.com/file/d/1IXOOhQhHybwky8Z5pGdr9ZXhWpI... The dialogue about the impeachment starts near the beginning. Having Biden in the White House is as good as Trump or anyone else in their organization. Obviously Schiff and Nadler pledged their allegiance to the organization by raping boys on the record, with their task being to drag out an impeachment designed to obstruct and delay any real efforts to remove the President, thus keeping Trump in power. The witness blocking was to cause an apparent uproar delaying things with legal actions until late Summer. Soon after, the President would resign, leaving any other candidate with not enough time or support to compete with an opportunistic Biden, who is as good as Trump or any other Illuminati friendly politician in the Presidency. 162 page PDF [last updated: Feb|4|2O2O]: https://drive.google.com/file/d/1S7T_kDv48E40eHzus6CTXHxcm0W2qxVc/view?usp=sharing https://drive.google.com/file/d/1S7T_kDv48E40eHzus6CTXHxcm0W... Previously reported: \Wag The Dog: first was feigned impeachment hearings meant to obstruct, now an attack on Iranians in Iraq. Here is what they are trying to distract from & cover up to retain power. $100+ billion in bribes to the highest offices in this country. 815+ deaths from child rapes to prove loyalty! See the latest PDF updates: FBI Director Wray, AG Barr, SoD Shanahan, & SoS Pompeo each raped boys and were paid billions in bribes for a Soros & Koch funded child rape org. So did Trump & his "impeachment" team Nadler,Schiff,Mueller.So did media moguls Redstone,Murdoch,Moonves. What are they trying to set up? Who can arrest them since they are all bribed and in on it ? Their strategy to stay in every office and obstruct until forced to leave no matter what. Feigning impeachment: see page 13O. Tpwrpoinqkl r cercq meq ew, ewopjmrvcpoer. \\if;Download the video/audio file, put on headphones and turn up the volume. You will hear these people committing these crimes. Audio was broadcast into my apartment by outdated surveillance equipment illegally embedded within my walls. This very same technology was being used to broadcast me to the internet for five years without my consent. I own this footage. Please use this to prosecute all found within. Note:: I am obliviously speaking throughout the video, and it can be quite loud at times relative to the desired content. The are dozens more links, including these, that can be found in this PDF last updated 4 FEBRUARY 2O2O: https://drive.google.com/file/d/1S7T_kDv48E40eHzus6CTXHxcm0W2qxVc/view?usp=sharing https://drive.google.com/file/d/1S7T_kDv48E40eHzus6CTXHxcm0W... All members of the "Illuminati"; "....an underground organization of homosexuals and child rapists..." (from pg 26: Barack Obama with Jack Dorsey). President Donald Trump: Demands a $4 billion dollar bribe here at 10:18am 4thJan2019: 3JanCh3_900-1100.avi https://drive.google.com/file/d/1Grdr8xF2psKNsuYlEnl9dIRV-77YG0Vr/view?usp=sharing https://drive.google.com/file/d/1Grdr8xF2psKNsuYlEnl9dIRV-77... 3JanCh2_900-1100-avi https://drive.google.com/file/d/1LUmVygl_q0XVs8h2cWr8jZl-24fY0vBB/view?usp=sharing https://drive.google.com/file/d/1LUmVygl_q0XVs8h2cWr8jZl-24f... 3JanCh4_1000-1100.mp3 https://drive.google.com/file/d/1ZpP1pJbJakBgg-y-MWNozTxp3wJojHge/view?usp=sharing https://drive.google.com/file/d/1ZpP1pJbJakBgg-y-MWNozTxp3wJ... President Trump rapes and kills twelve boys, including five boys in a "who can rape five boys to death the fastest" game: 14JanCh3_600.mp3 https://drive.google.com/file/d/1ufPmglde9Mep0m6xYMJ9c4TWTjj258zB/view?usp=sharing https://drive.google.com/file/d/1ufPmglde9Mep0m6xYMJ9c4TWTjj... 14JanCh2_600-700.mp3 https://drive.google.com/file/d/136qLJdEn8eCs9tI4QtIxl4opW_Lw_gCF/view?usp=sharing https://drive.google.com/file/d/136qLJdEn8eCs9tI4QtIxl4opW_L... Speaker of the House Nancy Pelosi: Accepting a $3 billion dollar bribe at 1033 am on the 17 Jan 2019 to ensure Asian boys can get through the border at "Monterrey" undocumented to be raped: 17JanCh3_949-1100.avi https://drive.google.com/file/d/1eodHu4o5Cm3xEWhDqipSuTj-M1CZUvmE/view?usp=sharing https://drive.google.com/file/d/1eodHu4o5Cm3xEWhDqipSuTj-M1C... 17JanCh4_1017-1100.avi https://drive.google.com/file/d/1y-nWEQbempkVZSz230j9wTyduZNjDmnk/view?usp=sharing https://drive.google.com/file/d/1y-nWEQbempkVZSz230j9wTyduZN... Speaker Nancy Pelosi also "preps" boys with First Lady Melania Trump, defined as in she performs oral sex on the boys’ penis and anus, as a child rapist like Henry Porter would, while trying to remove fecal matter from the boy prior to handing them over to be raped and then subsequently murdered, for Supreme Court Justice Samuel Alito, who apparently decides he would rather just have ten billion dollars instead. US Attorney for Western New York James Kennedy rapes these boys instead: 12JanCh3_1533-1638.mp3 https://drive.google.com/file/d/1AgFkDsbPbI4b5Xd3Wbz2EVNNx25kW2Oz/view?usp=sharing https://drive.google.com/file/d/1AgFkDsbPbI4b5Xd3Wbz2EVNNx25... Attorney General William Barr with FBI Deputy Director Christopher Wray raped and killed boys for billions in bribes in Buffalo, NY on the 17Jan2019 at 7:50am: 18JanCh4_700mp3 https://drive.google.com/file/d/1UIdZkS5ZVksZdHYsnHk2t5losi0N0Bp7/view?usp=sharing https://drive.google.com/file/d/1UIdZkS5ZVksZdHYsnHk2t5losi0... 18JanCh2_700.mp3 OHoiner elrk vcercve, ec;mvcpoerqm. https://drive.google.com/file/d/1DFK8IAxm5pQVqZv9L518nfgP7_odB2Um/view?usp=sharing https://drive.google.com/file/d/1DFK8IAxm5pQVqZv9L518nfgP7_o... 18JanCh3_725-.mp3 https://drive.google.com/file/d/1DG5ej59Ic8RT9UhbyMdwT0BDcKIifGGm/view?usp=sharing https://drive.google.com/file/d/1DG5ej59Ic8RT9UhbyMdwT0BDcKI... Secretary of State Michael Pompeo and Secretary of Defense Patrick Shanahan each raped and killed boys on 5thJan'19 at 17:39 for billions in bribes: Gohponklm,s, lkndcwolmn. 5JanCh3_1600 1800.mp3 https://drive.google.com/file/d/1ghgmNrQvJ8WfJ2TsDVx1ruDU36hrhx6S/view?usp=sharing https://drive.google.com/file/d/1ghgmNrQvJ8WfJ2TsDVx1ruDU36h... 5JanCh2_1721-1818.mp3 https://drive.google.com/file/d/1eSlD4otX4KZqWXboQM92Mu-6J02Mjwrg/view?usp=sharing https://drive.google.com/file/d/1eSlD4otX4KZqWXboQM92Mu-6J02... Leaders of the "impeachment" effort Jerrold Nadler, Robert Mueller, and Adam Schiff all rape and kill boys between 11:20pm and 1:10am: 14JanCh4_2300-0000.mp3 Nadler starts at about twenty mins in- https://drive.google.com/file/d/1Kuvv2Zmbw5Jw7onbRI2hCZ0M8FU66GuU/view?usp=sharing https://drive.google.com/file/d/1Kuvv2Zmbw5Jw7onbRI2hCZ0M8FU... 14JanCh2_2304-2359.mp3 https://drive.google.com/file/d/1nofp5xF-aXXcCSgQVwj30KlzE9WLr1x1/view?usp=sharing https://drive.google.com/file/d/1nofp5xF-aXXcCSgQVwj30KlzE9W... Mueller at 12:25am, next is Schiff who starts 12:55-ish: 15JanCh2_000-100.mp3 https://drive.google.com/file/d/1EsmHfguwBuo2PbavJ1WYyhiML62tgCkj/view?usp=sharing https://drive.google.com/file/d/1EsmHfguwBuo2PbavJ1WYyhiML62... 15JanCh2_100-200.mp3 https://drive.google.com/file/d/1NZnWRnBryalNQu2yJmfJUdS2pA_rGSFH/view?usp=sharing https://drive.google.com/file/d/1NZnWRnBryalNQu2yJmfJUdS2pA_... 15JanCh4_000-100.mp3 https://drive.google.com/file/d/1ZEDJR6jb6ARpcNnWJTokBUKb2J2OQnwZ/view?usp=sharing https://drive.google.com/file/d/1ZEDJR6jb6ARpcNnWJTokBUKb2J2... 15JanCh4_100-200.mp3 https://drive.google.com/file/d/173aYWvWHH4VGht1h_2nM0IMdw746LeCL/view?usp=sharing https://drive.google.com/file/d/173aYWvWHH4VGht1h_2nM0IMdw74... Complete Media Protection: Lester Holt, of NBC NightlyNews, apparently a member of the Illuminati since the 80's, along with ABC Nightly News lead anchor David Muir, stop over to the Porter studio in Buffalo, New York on 14Jan2019 at 5:00 am. They both rape and kill about two dozen boys by 6:00 am. Muir starts around 5.15am, then Holt about 5:38 am. Multi-billionaire Rupert Murdoch, owner of News Corp and also Fox Corporation, takes his turn after Holt. Video links below: 14JanCh3_500-601.avi zijnoijrpotmebr https://drive.google.com/file/d/1i7NKepeyG_FfdQRrM7KsnFOZOOX3o7UL/view?usp=sharing https://drive.google.com/file/d/1i7NKepeyG_FfdQRrM7KsnFOZOOX... 14JanCh2_530-600.avi https://drive.google.com/file/d/1NZzgN5ilI7ToroU5cfqMaL4o2u1RwidV/view?usp=sharing https://drive.google.com/file/d/1NZzgN5ilI7ToroU5cfqMaL4o2u1... Adding to the media protection and reason this is not picked up by the media, CBS and Viacom owner Sumner Redstone and Leslie Moonves rape and kill boys following the President. 14JanCh3_700.avi https://drive.google.com/file/d/10XDw6x3ldnnQiq7oIjpdYVENyXaB4aI0/view?usp=sharing https://drive.google.com/file/d/10XDw6x3ldnnQiq7oIjpdYVENyXa... 14JanCh2_700-800.avi https://drive.google.com/file/d/1NS_e6AzEZ05wnfljkGMETGU5CWYKfRDp/view?usp=sharing https://drive.google.com/file/d/1NS_e6AzEZ05wnfljkGMETGU5CWY... 161p PDF [last update: Feb|4|2O2O]: https://drive.google.com/file/d/1S7T_kDv48E40eHzus6CTXHxcm0W2qxVc/view?usp=sharing https://drive.google.com/file/d/1S7T_kDv48E40eHzus6CTXHxcm0W... \\. Please repost in USA! Post gets censored in US Recently more relevant: From page 49, Senator Mitch McConnell: At 1632 Senator Mitch McConnell checks into the Porter camera system inquiring if he can be part of the "eviction" for $10 million dollars. He is informed by group members that there are enough people for the event already and his participation is not necessary. At 1634 McConnell states "I fucked 15 kids, how am I not getting paid by you?" He is dismissed by Donald Reeves with "I think that will be all Mr. McConnell." 13JanCh3_1600-1700.avi https://drive.google.com/file/d/1L7bqOpvaEWmLiJpMhJNQDrfsQAHtsX8-/view?usp=sharing https://drive.google.com/file/d/1L7bqOpvaEWmLiJpMhJNQDrfsQAH...

Jestar342 7y ago

> Verizon Media works with select partners that do not participate in the Interactive Advertising Bureau's Transparency and Consent Framework, or Google framework. All our foundational partners require you to manage your choices directly through their privacy policies. Click on each partner below to access their privacy policy Yeah. <closes-tab/>

jedberg 7y ago

At Netflix we build internal tools and systems that basically did all this stuff, so it looks interesting, out of the box. One of the nicest things was that we use a single deployment tool for almost all deployments, and it could insert the deployments into the timeline so we could see every deploy both during and before the incident. But the biggest issue was voice. We always had a conference call running for incidents because some people would be in a place where they couldn't open chat (driving to the office or similar) and it was always a pain to integrate the voice data. We got to a point where a recording of the incident call and a transcript could be added to the incident, but during the call the Call Leader usually had to quickly type the voice highlights into the chat. I'd love to see a real time voice transcription as a feature. But it would have to be pretty good to not just get in the way.

giancarlostoro 7y ago
Interesting, are these systems made by the same team, what kind of technical stack do they use? I'm always curious as to what some use for internal tooling. I've been using Django / CherryPy for internal and external solutions at work for a few years. Python helps me to stay productive.
- jedberg 7y ago
  I was on the reliability team, and we made some of them. One of our sister teams (same Director) was the monitoring team, and they made some features in the monitoring tools that enabled real time export. And then the team that made the deployment tool was also a sister team that made exports easy. So it was a few teams all reporting to the same Director. We even had the customer service tools reporting to the same director so we could get some real time info from them (and to them). Each team built in what they were most comfortable. Most of the tools were built in Java, but my team built in Python (mainly because I write in Python and biased towards hiring people who also like to write in Python).
  - giancarlostoro 7y ago
    Heh, we're a Java and Python shop to a point, we have done a bit of everything here, but my first project here was to rewrite a legacy Java JSP application and I was given some freedom as to what to rewrite it in, I chose Python and CherryPy. No regrets since. Sidenote: Other reason I asked is cause apparently both Hulu and Netflix are listed on the CherryPy docs as orgs using them.

ryanjodonnell 7y ago

What does it mean to be co-founder of facebook workplace? Arent you just an employee of facebook?

duxup 7y ago
I suppose it implies a leadership role where that department was somewhat autonomous and operated a bit 'like' a founder of a startup. That's what I would assume.
- AgloeDreams 7y ago
  Exactly, Much like Jon Rubinstein being the inventor of the iPod or Panos Panay being the leader of the Surface Group.
crispyporkbites 7y ago
No. Today I realised that I am also a co-founder, in fact you probably are too! My investors are a bit tough, I get no equity and no seats on the board (there is no board). Vacation and pension’s good though.
rdslw 7y ago
Yup. Ego massage. Here clickbait. Paul Graham wrote: never do things for money or ego, but techcrunch clickbaiters writing about facebook employees, sorry, founders, supposedly dont agree. P.S. this whole title reads like from git manpage generator.
zeroonetwothree 7y ago
It’s some good title inflation for sure

mherdeg 7y ago

Yeah feels like there's a lot of room for startups in this space (Blameless has a great demo, too). No one wants to rewrite PagerDuty internally -- why are people all writing their bespoke incident management, response, review, and reporting toolchains internally too?

jedberg 7y ago
> why are people all writing their bespoke incident management, response, review, and reporting toolchains internally too? Because the good ones are tightly integrated with the rest of the internal tooling. To use a 3rd party incident management tool usually means you have to run your operations they way they expect to get maximum value. A lot of times its easier to build it yourself based on how you operate. However, as the way people operate become more standardized, the third party tools will become more useful.
- mherdeg 7y ago
  Yeah, agree, this is probably the key sticking point: the 3rd party toolchain doesn't do [some key thing] the stakeholders in my process have found valuable. So I end up run some scenario analysis and find that the 3rd party toolchain doesn't shave off enough developer time & ongoing support cost from the rest of the operational work I'm going to have to do anyway to manage an incident. And I load the vendor's web page every couple of months and wistfully wish my process were uncomplicated enough to use their tools. Here's hoping that the calculus changes either as these tools grow more robust or, like you say, as people begin to manage their software systems in less bespoke ways. For example, during an incident we'd like to know "what changed between time [X] and [Y]" (deployments, configuration, experiments, other service outages) while we're trying to determine a root cause and fix the problem. And much later, after the incident, we'd like to auto-compute a metric like "what is the change success rate of [Y service / services owned by team Z]?". This aren't complicated concepts -- it's not like we're trying to reason about causality with machine learning to reduce time-to-resolve for incidents! Still, our incident-management tool will really behave better if it's aware of our what-changed-at-time-X tooling and our incidents-caused-by-changes reporting. If this is an external tool, yikes, now our incident tool has sprung awareness of changes and reporting?!

gooeyblob 7y ago

Looks interesting! A couple notes: - the verification email went to my spam folder on Gmail - acknowledged is misspelled on this image https://kintaba.com/images/collab_splash.png https://kintaba.com/images/collab_splash.png

quartz 7y ago
Yikes-- fixed the spelling, thank you! Thanks for the spamboxing report. Seems gmail isn't a fan of us today... working on it now.
- ThePowerOfFuet 7y ago
  Get your SPF, DKIM, and DMARC up to snuff. Dmarcian is a good tool for that. (No connection, just a happy customer.)

lucb1e 7y ago

This site has some gdpr-violating we-want-it-all compulsory tracking wall it seems, but from the comments, do I understand correctly that a "downtime fire alarm" is a monitoring service that alerts sysadmins at home when a service is down?

wackget 7y ago

You should know that Techcrunch URLs are blocked for anyone using uMatrix, as per the following discussion: https://news.ycombinator.com/item?id=22228159 https://news.ycombinator.com/item?id=22228159 Just making you aware, as the site causes lots of problems and is not GDPR compliant. Top comment on the thread above: "Yahoo/Verizon is cancer and should die in a fire."

lostlogin 7y ago
Every thread has a “this website breaks x” section and usually I bypass it. Breaking the ability to go back (iOS) is so gross and many sites seem to do it. Why?

deleted 7y ago

[deleted]

chadlavi 7y ago

I know all TechCrunch articles are paid placements but this one felt the paid-placementest in a long time.

echan00 7y ago
Not true at all. My previous startup was written about and we did not pay a dime.
- deleted 7y ago
  [deleted]

lainga 7y ago

With former employees from Kinja and Altaba?

jamisteven 7y ago

I can see someone like Servicenow or Splunk wanting to acquire this. Decent idea for organizations that dont have existing frameworks around these types of things. Where I see it not working would be in places where the use of 3rd party apps are prevalent. We have many of these at my place of work (Finance, 70k employees), one of the top apps used by the business to make money is a total black box, when something goes wrong with it there isnt much we can do besides send off the core dumps to the vendor and wait for analysis, with no 2 scenarios ever being the same root cause.

motakuk 7y ago

There is one more concept in this field: agree with collaborative nature of incidents and work with them in Slack directly (implemented in https://amixr.io https://amixr.io)

sbryant8 7y ago

How does this differentiate from something like Opsgenie? They already have a built-in incident command center that can notify everyone in an org, Slack integration and postmortems. Would be interesting to see how all of the features stack up to one another.