imbriaco

9 ms·

31.
by imbriaco 12y ago
Exactly this. It would have seemed pretty presumptuous to ask the Debian project to change the support model that they decided worked for them to support us. We were also faced with a very large new infrastructure build and we knew we were
32.
by imbriaco 12y ago
I was part of the Ubuntu decision at GitHub, and it was based on the fact that Ubuntu gave you much longer support after the next release. Debian (at the time) only guaranteed even security updates on previous releases for one year after a
33.
by imbriaco 12y ago
That's incorrect. The way you apply bug fixes to 12.04 LTS is by installing the bug fixes, not by moving to 14.04. The whole point is that 12.04 LTS continues to get updates for 5 years.
34.
by imbriaco 12y ago
http://en.wikipedia.org/wiki/Split-brain_(computing)
35.
by imbriaco 12y ago
I was too. And I had that beer when we finished. ;)
36.
by imbriaco 12y ago
I prefer not to publicly name the vendor. This is about problems on our network, and it's ultimately our responsibility.
37.
by imbriaco 12y ago
Generally speaking I agree with you. However, in this specific case there were a couple of reasons we chose the path we did: 1. We had experienced bugs with the currently running release which we were fairly sure would manifest when we remo
38.
by imbriaco 12y ago
We have very competent network engineers, but it never hurts to have a second opinion. Verifying that there are no subtle interactions that we've missed is simply prudent. At our scale, partnering with our vendors is a necessity. We of
39.
by imbriaco 12y ago
I'll never suggest that teachers are overpaid, but hyperbole doesn't help. The starting salary for the lowliest teacher in the Newark Unified School District is $54,300/year. http://nusd.ca.schoolloop.com/file
40.
by imbriaco 12y ago
She didn't have access to private servers at his old company. Please stop making up things like this that have no basis in reality.
41.
by imbriaco 13y ago
Glad to see the correction, it solves the one problem with an otherwise very well written incident report. Kudos for setting the record straight and taking the criticism so well.
42.
by imbriaco 13y ago
I think the problem I had with it had to do with the way the sentences were constructed. For example: "... this could have been a disaster, very much like the rubygems.org security breach in early 2013" This implies that the iss
43.
by imbriaco 13y ago
It's just the most efficient and visible way for us to do it, it's not the only way. Here's a couple of reasons why we like it: 1. It's scripted so you don't have to think about it at 3am. 2. The rest of the team ca
44.
by imbriaco 13y ago
It's not practical with the sheer number of legitimate IP addresses that access GitHub, unfortunately.
45.
by imbriaco 13y ago
You're totally right. I just adjusted the post to convert the times to UTC.
46.
by imbriaco 13y ago
According to FAR 91.119 you already should be above 500 feet AGL unless you're taking off, landing, or above very sparsely populated areas. I'm assuming fixed wing aircraft, so if you're a helicopter pilot the rules are sligh
47.
by imbriaco 13y ago
I know a bunch of folks in San Francisco compete in a soccer league with some other area tech companies. And there's always our annual dodgeball tournament, so I guess that's a yes! :)
48.
by imbriaco 13y ago
We actually have more like 238 Hubbers. You can see us at https://github.com/humans.txt or https://github.com/about/team .
49.
by imbriaco 13y ago
US/Pacific I believe.
50.
by imbriaco 13y ago
I talked to them at the Open Compute Summit and got the chance to look at it up close. They quoted me 100GB/disc and 10,000 discs in the cabinet for 1PB. They could have been rounding those numbers for simplicity, but that gets you in
51.
by imbriaco 13y ago
Ah, that's interesting. I wasn't aware of this post at all, thanks for pointing it out.
52.
by imbriaco 13y ago
That's not at all what I was getting at. It's not about patting yourself on the back or trying to make the team look like heroes. I was more wondering how the mechanics of their incident response processes were managed and whether
53.
by imbriaco 13y ago
I'm admittedly being nit-picky because I feel very strongly about the importance of outage communication. Good communication both during and after an incident can make a tremendous amount of difference in how you are perceived. They de
54.
by imbriaco 13y ago
That's just it, though. This is the public face of the team that responded to that outage. It absolutely represents them. Now, whether it's a fair depiction or not is definitely a valid question. Having written more than my fair s
55.
by imbriaco 13y ago
This is, sadly, not a great post-mortem. They missed an opportunity for goodwill. I don't feel more confident in their level of understanding or ability to remediate the problems that led to it after having read it. I know they have an
56.
by imbriaco 13y ago
"Quite a few" is actually a very very small percentage these days. If your ISP is disregarding TTL, you're just as bad off with CNAMEs pointing at a third party as you are with the ALIAS-type record.
57.
by imbriaco 13y ago
They came from a datacenter world before they moved to AWS. The problem was that they had a traditional enterprise IT organization with big process and relying on Oracle, SANs, and all the same old enterprise IT solutions. It was built to
58.
by imbriaco 13y ago
Yes, I have in fact, in a DRBD configuration. The bug was esoteric, but it happened and was not the result of user error. DRBD and Gluster both allow faults in the VFS layer to propagate to all replicas.
59.
by imbriaco 13y ago
DRBD and Gluster are not any more resilient to filesystem corruption than a RAID device is. In this kind of case you hope for either real-time replicated storage on a completely separate physical host or very recent backups.
60.
by imbriaco 13y ago
Compare their cost reductions with the decrease in computing costs and you'll very rapidly discover that they don't actually pass on savings to customers. Quite the opposite.

More ›