4 ms·

When I first started at Google I got things done a lot faster because I didn't know all those rules existed and nobody stopped me. My service was still plenty f

by w0mbat 5y ago

When I first started at Google I got things done a lot faster because I didn't know all those rules existed and nobody stopped me. My service was still plenty fast & reliable. Eventually it all got rewritten by other people to do things properly like the video says.

dekhn 5y ago
I managed to deploy a whole system at google that had the ability take down all of google globally by DoS'ing the network, and ran it casually (IE, starting and stopping it when I felt like it, at the capacity I felt like, with the binary versions I wanted) for 3 years. In retrospect, this was absolutely crazy! The actual visible outcomes were: 1 cluster drained due to heat rising so fast the alerting thought there was a fire, 1 page to an engineer in the middle of the night (sorry discovery-service) and a whole bunch of complaints about CPU stealing that weren't my fault. Those were the good old days.
robocat 5y ago
Rachel said something similar “My own ‘solution’ to it after far too much thrashing was just to say ‘we cannot get all N types of quota in the same place so we are at the mercy of whatever happens to be available, and if that dries up, we stop running’. Granted, this was for some internal stuff that was seven or eight levels removed from anything that anyone on the outside might ever see, but still, it was stupid and made me feel so dirty. I'm sure my non-solution probably bit someone later. Sorry, whoever.” — https://rachelbythebay.com/w/2021/10/30/5tb/ https://rachelbythebay.com/w/2021/10/30/5tb/
hiepph 5y ago
Isn't that things are supposed to be? Prototype and then polish later. I think teams should work in layer like this, but I guess with big company they're just too lazy and assume you're smart enough and have time to take care of everything.
- w0mbat 5y ago
  Right. Time was of the essence with this service, and it was still very reliable and scalable in its original simple form, because it leaned on a lot of existing Google infrastructure. Later on it got upgraded to follow the rules better.

by w0mbat 5y ago

dekhn 5y ago

I managed to deploy a whole system at google that had the ability take down all of google globally by DoS'ing the network, and ran it casually (IE, starting and stopping it when I felt like it, at the capacity I felt like, with the binary versions I wanted) for 3 years. In retrospect, this was absolutely crazy! The actual visible outcomes were: 1 cluster drained due to heat rising so fast the alerting thought there was a fire, 1 page to an engineer in the middle of the night (sorry discovery-service) and a whole bunch of complaints about CPU stealing that weren't my fault. Those were the good old days.

robocat 5y ago

Rachel said something similar “My own ‘solution’ to it after far too much thrashing was just to say ‘we cannot get all N types of quota in the same place so we are at the mercy of whatever happens to be available, and if that dries up, we stop running’. Granted, this was for some internal stuff that was seven or eight levels removed from anything that anyone on the outside might ever see, but still, it was stupid and made me feel so dirty. I'm sure my non-solution probably bit someone later. Sorry, whoever.” — https://rachelbythebay.com/w/2021/10/30/5tb/ https://rachelbythebay.com/w/2021/10/30/5tb/

hiepph 5y ago

Isn't that things are supposed to be? Prototype and then polish later. I think teams should work in layer like this, but I guess with big company they're just too lazy and assume you're smart enough and have time to take care of everything.

w0mbat 5y ago
Right. Time was of the essence with this service, and it was still very reliable and scalable in its original simple form, because it leaned on a lot of existing Google infrastructure. Later on it got upgraded to follow the rules better.