3 ms·
I was in Seattle the whole time. It was common on the team I was on to work about 10 hours a day in office and then go home and have meetings with our sister t
by thesuperbigfrog 7y ago
I was in Seattle the whole time.
It was common on the team I was on to work about 10 hours a day in office and then go home and have meetings with our sister team in India and some follow-up work with them. The sister team was working on standing up an automation service to reduce some of our "manual" work and help reduce the oncall burden, but it took nearly a year and a half for that to happen.
>> if you get paged at night people cover for you the next day.
That would have been nice. We initially had to provide 24-7 coverage for a week during our oncall rotation. After the sister team in India stood up, it turned into 12 hours a day and a hand off to an oncall engineer in India who would cover the next 12 hours. Both "tag team" engineers would cover a week together for the oncall rotation.
>> I didn't work on a Q4 peak team but your math is psychotic. 100 hours is 14 hours a day 7 days a week?
Our service directly supported order fulfillment. Therefore it HAD to stay up at all costs, especially during Q4.
Granted 100 hours did not happen often, it was usually closer to 70 and mainly consisted of getting everything ready for Q4 and load testing it all to make sure that it was solid for the tremendous holiday volume. Remember, this was BEFORE the retail website had fully transitioned to elastic scaling. All the supporting layers and configuration were pre-scaled manually based on the previous year's peaks plus the expected increase.