4 ms·
I really like it when people in your position at such a big company post on here, even if it is a brief comment like this. Thank you!
by tentacleuno 5y ago
I really like it when people in your position at such a big company post on here, even if it is a brief comment like this. Thank you!
- jgrahamc 5y agoI would have posted faster but I was too busy doing my own little bit of debugging which consisted of 1. dig @1.1.1.1 jgc.org 2. nc -v 104.22.11.223 80 3. curl -v https://jgc.org/cdn-cgi/trace 4. curl -v https://jgc.org/ Hmm #1 was fast so network is routing OK. Hmm #2 was fast so TCP is OK. Hmm #3 was fast so I know (because I worked on that code) that this code path is good. Hmm #4 is slow so that means component X is slow but still working. Of course, in parallel I'm in a conference call with about 40 other people who have actual access to monitoring and systems and other things who can see exactly where things are. But I was damn close with four commands and gave me confidence in what people were saying. But, I have to say, Cloudflare's internal distributed tracing system is pretty cool because I got sent a trace and you could see right where the slowdown was.
- tailspin2019 5y agoNow you're just showing off :)
- jgrahamc 5y agoAllow an old man the fantasy that he still knows how the whole of the system works.
- jatone 5y agosounds like you do. ;)
- maxgashkov 5y agoThat's a huge problem for companies of almost any scale. Can you shed some light on tools used internally in Cloudflare for tracing?
- stillicidious 5y agoI'm glad commenting on HN takes priority over notifying your enterprise customers.
- jgrahamc 5y agoWe have whole teams of people to get status up and inform people and support them.
- stillicidious 5y agoAnd yet I'm learning about this outage from HN. How did that happen?
- jgrahamc 5y agoSuggest subscribing on https://www.cloudflarestatus.com/ https://www.cloudflarestatus.com/ as that was updated long before I came to HN to say anything (which was only after the incident was resolved).
- tailspin2019 5y agoAviate Navigate Communicate. They did all three of those in the space of a few minutes (if you were checking their status page - which you can subscribe to by the way).
- ithkuil 5y agoThat rule is great for one single person (a pilot). In corporate incident management we can do better, and not blocking communication because you're overwhelmed with keeping the system up; in fact proper incident management workflows designate different people with different roles and delegates communication duties to a dedicated person if possible.
- tailspin2019 5y ago> That rule is great for one single person (a pilot). In corporate incident management we can do better, and not blocking communication because you're overwhelmed with keeping the system up; in fact proper incident management workflows designate different people with different roles and delegates communication duties to a dedicated person if possible. I agree with you, but Cloudflare seem to have done exactly what you're describing in this instance?
- tuananh 5y agoyou often do you code these days?
- jgrahamc 5y agoHardly ever. If I start a project I don’t end up having time to finish or maintain it which isn’t fair to the team. If I write something it’s for my own use. And I like to write things that test Cloudflare. Доверяй, но проверяй.
- maxgashkov 5y agoI see comrades you have on your team have taught you well /jk
- vinay_ys 5y agoWhat was component X? Was it a buggy rollout?
- jgrahamc 5y agoA proxy that serves traffic. No, it was way more complicated than “buggy rollout”.