6 ms·

Side note -- I know a lot of early YC startups like to play things fast and loose, but you really should compensate your engineers for after-hours emergencies i

Side note -- I know a lot of early YC startups like to play things fast and loose, but you really should compensate your engineers for after-hours emergencies if they are already working 40 hour weeks. Morally and employee-retention-wise it's the obvious thing to do, but beyond that in certain states and jurisdictions you can easily run afoul of local labor laws if you try to require employees to do things outside of their regular hours without compensation. Another very important angle to consider is you want to incentivize your employees to answer the call to deal with these issues. If there is no incentive structure in place, they might not take it seriously, which in itself can be a security and stability risk that could threaten things like your SOC-2 compliance, if you have it. When I worked as a full time DoD scientist, there was a very set-in-place system for dealing with these situations, and it was normal to pay double overtime for any hours spent by an employee on after hours emergencies. This is the right way to do it. Pre-series B startups largely don't do this, but once you get to series B and C suddenly it becomes a thing because companies realize they either have to legally, or have to to prevent their employees from churning and to protect themselves from people not showing up to put out the fire. Just do it, and do it early. Do it before Series A. That's the advice I give my consulting clients, and the approach I take with my own companies. By compensating your employees for this time you also take what would be a red flag for many would-be employees and turn it into an exciting perk.

bpodgursky 4y ago
Unless you're in a really dysfunctional startup, the employees who voluntarily jump in after-hours to fix OC issues are going to be very quickly compensated with raises and refresher option grants. Far better payout generally than a few bucks of overtime. It becomes really obvious who is holding the company together and who is coasting, relying on the "senior" (ie, anyone who puts in the effort) engineers to keep the ship afloat, and in a highly competitive engineering market, that will definitely be reflected in equity comp.
- SpicyLemonZest 4y ago
  The problem with informal "people will notice" reward schemes, even when administered well (and don't take that for granted, it's easy for them to become popularity contests!), is that they encourage bad work-life balance among more junior employees. If you regularly have people popping up at 10 PM to fix things, and you don't have any formal recognition of the fact that they've gone above and beyond, new hires who want to get ahead will learn that working late into the night is the way to do that.
  - sokoloff 4y ago
    Going above and beyond what your fellow humans are willing to do is a pretty tried-and-true way to get ahead, whether you’re working for yourself or as an employee.
    - zasdffaa 4y ago
      Can do, or your boss can want just a cog in a machine and not care or want anything more.
- hrudham 4y ago
  I respectfully disagree. I did exactly this for a number of years. A colleague (and is now friend since we've both left said company) made it clear he would not work overtime without compensation (and never did), but his day-to-day work was excellent. He knew he had a skill they required, and was good at it (and our skill sets were the same). Our careers both grew at about the same pace for four years. He became known as the go-to guy for green-field projects, while I was the guy you could put onto over-budget tight-deadline projects to rescue them. I worked a lot of weekends for my troubles, and I always envied how he ended up on the "fun" projects. I learnt a lot from him. As a professional, some self respect is required, or you will be abused.
  - toomuchtodo 4y ago
    > As a professional, some self respect is required, or you will be abused. These are hard learned lessons. You also have to structure your life in a way that you don’t need the job and can walk away when needed.
- valeness 4y ago
  Lolol, I've been jumping in after-hours for years. They only take notice when you say "I quit".
  - raverbashing 4y ago
    And some even later than that, when they go to fix something and they realize they don't know how
    - bpodgursky 4y ago
      It's your job to make your work visible.
      - axus 4y ago
        Visible to who? Certainly the request to work outside of their normal schedule, and the report that an issue was resolved are seen by more than just the person responding to the request?
- 725686 4y ago
  I don't know for startups, but for regular companies... not. You must be very young my friend.
  - bpodgursky 4y ago
    https://www.linkedin.com/in/ben-podgursky-5935aa7/ https://www.linkedin.com/in/ben-podgursky-5935aa7/ I'm not exactly anon
sokoloff 4y ago
What I find interesting is that if you give employees a choice of 100 units of currency per year as salary plus 1 unit of currency for each week on call with an expectation of 1 week on call every four weeks versus 115 units of currency for the year with the same on-call expectations, some employees will feel better about the first arrangement and some will feel better about the second arrangement. Some people want to see everything broken out and take a very transactional approach to work; others prefer the simpler approach but with less clear linkage to a specific piece of work.
- DoughnutHole 4y ago
  I'd say the issue for people who are inclined to pick the first arrangement is one of trust. If you don't trust your employer not to abuse the second arrangement then you'd be a fool to accept it. With the first arrangement the employer is incentivised not to exceed the agreed upon expected number of on-call weeks because they are financially penalised for additional on-call time. With the second arrangement the employer can demand as much on-call time as they think they can get away with. It's really a case of if you expect your relationship with your employer to be in some way adversarial - if you trust your employer not to screw you than go ahead, but you can't necessarily reliably make that judgement going into a new job.
  - bornfreddy 4y ago
    I am firmly in the camp that would pick the first arrangement, but not because of (lack of) trust. It is partially due to psychological effect on me (get out of bed, you are compensated for this) and partially because the incentives of the employer are better aligned with mine. They should view 24/7 operation as something that costs money, so that they are more willing to invest in robust solutions.
- sam0x17 4y ago
  The problem is once you consider it part of your standard comp, you take it for granted. If emergencies almost never happen and then one does, you will inevitably feel cheated when you have to wake up at 2 AM on a Sunday. Much better familial optics if you can tell your family "hey we got an extra $500 for that" versues "oh, dealing with these is included in my take-home salary". Even worse, if emergencies happen all the time, your family is going to hate you if you don't have some reward you can show for each one, especially if there are other jobs at your salary range that don't have an on-call clause.
ectopod 4y ago
That's only half the story. You should also compensate employees for being on call. As the article notes, it is a significant chore.
- sam0x17 4y ago
  Yes I completely agree, though I fear that might be out of reach culturally for a lot of current YC companies so I didn't even try to push that one :( Unpaid on-call time is exploitative and illegal in most jurisdictions.
scottlamb 4y ago
Yes, and even more importantly: compensate your engineers who minimize the need for these kinds of heroics. Ones who: * fix the alerts so they reliably page when there's a SLO-worthy problem and only then. * test the restore system so it works smoothly when needed at the necessary scale. * add safety checks to prevent the need to use those backups in the first place. * get to the root cause of yesterday's outage and prioritize the 9–5 engineering work to ensure it won't happen again. It's awful to work for a place where heroics are often necessary and unrewarded. I still don't like working in a place where heroics are often necessary, even if they're celebrated. They're often avoidable.
- morelisp 4y ago
  The trick is to make sure the heroic efforts are rewarded quietly, but the unheroic ones create lots of noise. This is inherently difficult because the heroic efforts tend to come as a result of very noisy incidents - everyone already knows the database cluster was literally on fire and wants to hear how Jane put it out while simultaneously remagnetizing the backup drives. The lead's job is to make the noise ensuring everyone knows Morgan's the one who did the boring rewiring of the backup-restore process to automatically sync and fail over to another AZ when thermal metrics start trending bad. One thing I've noticed as we've been trying to integrate some on-call processes across formerly separate teams is that "our" incident reports were about 25% timeline/responses/impact analysis and 75% RCA and plans for future mitigations. "Theirs" were the opposite. Our incident rate has trended down sharply over the past two years even as our system grew; theirs has scaled up roughly at the same rate as their service count.
- aledalgrande 4y ago
  and document anything and everything and write amazing runbooks
- maerF0x0 4y ago
  > They're often avoidable. Avoidance isn't the goal of mgmt though. Cost minimization is, and by quantizing the compensation portion it can help figure out the cost of a bug. Ultimately the cost of a bug is a key variable in the decision to prevent or solve. The cost can easily vary from $0 (eg at a startup with no customers) or to millions (maybe billions?) when you consider the risk of brand damage, lost ARR (multiply that ARR by the sales multiple!), decremented velocity (lost opportunity value), and eventually even recruiting cost (to replace frustrated employees). I've always wanted to work at a JoelTest[1] company which fixes bugs before writing new code... But the closest i've found is a commitment that the oncall engineer works on bugs for their rotation to reduce the bug backlog ( and is not part of sprint velocity) [1]: https://www.joelonsoftware.com/2000/08/09/the-joel-test-12-steps-to-better-code/ https://www.joelonsoftware.com/2000/08/09/the-joel-test-12-s...
sam0x17 4y ago
Another side note, doing a 4-day work week for engineering is a great and productivity boosting practice, but it shouldn't be seen as a carte blanche to not compensate them for after-hours emergencies. If you have a 4-day work week, then 32 hours a week is their normal full-time number of hours that you are compensating them for. When you ask them to go beyond this, you need to compensate them for their time.

bpodgursky 4y ago

Unless you're in a really dysfunctional startup, the employees who voluntarily jump in after-hours to fix OC issues are going to be very quickly compensated with raises and refresher option grants. Far better payout generally than a few bucks of overtime. It becomes really obvious who is holding the company together and who is coasting, relying on the "senior" (ie, anyone who puts in the effort) engineers to keep the ship afloat, and in a highly competitive engineering market, that will definitely be reflected in equity comp.

SpicyLemonZest 4y ago
The problem with informal "people will notice" reward schemes, even when administered well (and don't take that for granted, it's easy for them to become popularity contests!), is that they encourage bad work-life balance among more junior employees. If you regularly have people popping up at 10 PM to fix things, and you don't have any formal recognition of the fact that they've gone above and beyond, new hires who want to get ahead will learn that working late into the night is the way to do that.
- sokoloff 4y ago
  Going above and beyond what your fellow humans are willing to do is a pretty tried-and-true way to get ahead, whether you’re working for yourself or as an employee.
  - zasdffaa 4y ago
    Can do, or your boss can want just a cog in a machine and not care or want anything more.
hrudham 4y ago
I respectfully disagree. I did exactly this for a number of years. A colleague (and is now friend since we've both left said company) made it clear he would not work overtime without compensation (and never did), but his day-to-day work was excellent. He knew he had a skill they required, and was good at it (and our skill sets were the same). Our careers both grew at about the same pace for four years. He became known as the go-to guy for green-field projects, while I was the guy you could put onto over-budget tight-deadline projects to rescue them. I worked a lot of weekends for my troubles, and I always envied how he ended up on the "fun" projects. I learnt a lot from him. As a professional, some self respect is required, or you will be abused.
- toomuchtodo 4y ago
  > As a professional, some self respect is required, or you will be abused. These are hard learned lessons. You also have to structure your life in a way that you don’t need the job and can walk away when needed.
valeness 4y ago
Lolol, I've been jumping in after-hours for years. They only take notice when you say "I quit".
- raverbashing 4y ago
  And some even later than that, when they go to fix something and they realize they don't know how
  - bpodgursky 4y ago
    It's your job to make your work visible.
    - axus 4y ago
      Visible to who? Certainly the request to work outside of their normal schedule, and the report that an issue was resolved are seen by more than just the person responding to the request?
725686 4y ago
I don't know for startups, but for regular companies... not. You must be very young my friend.
- bpodgursky 4y ago
  https://www.linkedin.com/in/ben-podgursky-5935aa7/ https://www.linkedin.com/in/ben-podgursky-5935aa7/ I'm not exactly anon

sokoloff 4y ago

What I find interesting is that if you give employees a choice of 100 units of currency per year as salary plus 1 unit of currency for each week on call with an expectation of 1 week on call every four weeks versus 115 units of currency for the year with the same on-call expectations, some employees will feel better about the first arrangement and some will feel better about the second arrangement. Some people want to see everything broken out and take a very transactional approach to work; others prefer the simpler approach but with less clear linkage to a specific piece of work.

DoughnutHole 4y ago
I'd say the issue for people who are inclined to pick the first arrangement is one of trust. If you don't trust your employer not to abuse the second arrangement then you'd be a fool to accept it. With the first arrangement the employer is incentivised not to exceed the agreed upon expected number of on-call weeks because they are financially penalised for additional on-call time. With the second arrangement the employer can demand as much on-call time as they think they can get away with. It's really a case of if you expect your relationship with your employer to be in some way adversarial - if you trust your employer not to screw you than go ahead, but you can't necessarily reliably make that judgement going into a new job.
- bornfreddy 4y ago
  I am firmly in the camp that would pick the first arrangement, but not because of (lack of) trust. It is partially due to psychological effect on me (get out of bed, you are compensated for this) and partially because the incentives of the employer are better aligned with mine. They should view 24/7 operation as something that costs money, so that they are more willing to invest in robust solutions.
sam0x17 4y ago
The problem is once you consider it part of your standard comp, you take it for granted. If emergencies almost never happen and then one does, you will inevitably feel cheated when you have to wake up at 2 AM on a Sunday. Much better familial optics if you can tell your family "hey we got an extra $500 for that" versues "oh, dealing with these is included in my take-home salary". Even worse, if emergencies happen all the time, your family is going to hate you if you don't have some reward you can show for each one, especially if there are other jobs at your salary range that don't have an on-call clause.

ectopod 4y ago

That's only half the story. You should also compensate employees for being on call. As the article notes, it is a significant chore.

sam0x17 4y ago
Yes I completely agree, though I fear that might be out of reach culturally for a lot of current YC companies so I didn't even try to push that one :( Unpaid on-call time is exploitative and illegal in most jurisdictions.

scottlamb 4y ago

Yes, and even more importantly: compensate your engineers who minimize the need for these kinds of heroics. Ones who: * fix the alerts so they reliably page when there's a SLO-worthy problem and only then. * test the restore system so it works smoothly when needed at the necessary scale. * add safety checks to prevent the need to use those backups in the first place. * get to the root cause of yesterday's outage and prioritize the 9–5 engineering work to ensure it won't happen again. It's awful to work for a place where heroics are often necessary and unrewarded. I still don't like working in a place where heroics are often necessary, even if they're celebrated. They're often avoidable.

morelisp 4y ago
The trick is to make sure the heroic efforts are rewarded quietly, but the unheroic ones create lots of noise. This is inherently difficult because the heroic efforts tend to come as a result of very noisy incidents - everyone already knows the database cluster was literally on fire and wants to hear how Jane put it out while simultaneously remagnetizing the backup drives. The lead's job is to make the noise ensuring everyone knows Morgan's the one who did the boring rewiring of the backup-restore process to automatically sync and fail over to another AZ when thermal metrics start trending bad. One thing I've noticed as we've been trying to integrate some on-call processes across formerly separate teams is that "our" incident reports were about 25% timeline/responses/impact analysis and 75% RCA and plans for future mitigations. "Theirs" were the opposite. Our incident rate has trended down sharply over the past two years even as our system grew; theirs has scaled up roughly at the same rate as their service count.
aledalgrande 4y ago
and document anything and everything and write amazing runbooks
maerF0x0 4y ago
> They're often avoidable. Avoidance isn't the goal of mgmt though. Cost minimization is, and by quantizing the compensation portion it can help figure out the cost of a bug. Ultimately the cost of a bug is a key variable in the decision to prevent or solve. The cost can easily vary from $0 (eg at a startup with no customers) or to millions (maybe billions?) when you consider the risk of brand damage, lost ARR (multiply that ARR by the sales multiple!), decremented velocity (lost opportunity value), and eventually even recruiting cost (to replace frustrated employees). I've always wanted to work at a JoelTest[1] company which fixes bugs before writing new code... But the closest i've found is a commitment that the oncall engineer works on bugs for their rotation to reduce the bug backlog ( and is not part of sprint velocity) [1]: https://www.joelonsoftware.com/2000/08/09/the-joel-test-12-steps-to-better-code/ https://www.joelonsoftware.com/2000/08/09/the-joel-test-12-s...

sam0x17 4y ago

Another side note, doing a 4-day work week for engineering is a great and productivity boosting practice, but it shouldn't be seen as a carte blanche to not compensate them for after-hours emergencies. If you have a 4-day work week, then 32 hours a week is their normal full-time number of hours that you are compensating them for. When you ask them to go beyond this, you need to compensate them for their time.