8 ms·

How to move fast and not break things as a remote company

156 points by irtefa 4y ago

deleted 4y ago
[deleted]
xwowsersx 4y ago
> Tip #5 – more code comments. I find this to rarely be true in practice. It's not that comments are intrinsically useless (though they do suffer from the intrinsic limitations that are not statically checked or tested), but that most developers tend to write comments for things that benefit from them. Or, when they do need them, they are poorly written or too terse to be useful to anyone else. I would venture to say I've seen maybe or two comments in my 15 years of software engineering that I saw and actually thought "wow, well this helps me understand this a lot better."
- physicles 4y ago
  I’ve been partying on the same code base for about five years now. I’ve written about 70% of it, but it’s large enough (200k lines or so) that I can’t keep all of its subtleties in my head. In the last year alone I’d estimate comments written by my past self have saved me dozens of hours of effort tracking down the reason why code was implemented a certain way. A couple times I’ve undertaken some refactoring that would’ve taken hours, only to find 10 minutes into the process a comment to the effect of “don’t try to refactor in that way, for these reasons” and it was correct. Maybe my memory is just particularly bad. But if the comments are helpful to me, they’d also be helpful to someone else who didn’t write that code. It does take practice to know when and how to write those comments though.
  - xwowsersx 4y ago
    That's great, and some people are better at writing useful comments than others. Not to detract from what you're saying, but > But if the comments are helpful to me, they’d also be helpful to someone else who didn’t write that code. this does not necessarily follow unless you've been extremely thoughtful in how you wrote those comments (which you may very well have been). That is, in fact, the issue most of the time — that comments are written in such a way that they serve to jog the memory of the person who wrote them, but are rarely of general use to anyone else.
  - valenterry 4y ago
    Yeah I agree with this notion. After working in very high level and expressive language (with static typing) I found that I could get rid of about 90% or even more of my comments. But for the rest, it's exactly as you described. There is usually some workaround/shortcut being taken or there is interfacing with an external dependency that does not work as expected. In those cases comments make aware of things that the code doesn't show and can save a looot of time.
mattpallissard 4y ago
> hours of planning can save weeks of engineering. This is so hard to get teams to do. I am almost always the slowest to implement a change simply because I don't conflate design and implementation. At first glance it looks like I get less done, but when you account for the fact that nobody has to clean up after me or shoehorn a work around for an edge case in I become one of the most efficient. Assigning cost to subsequent follow up tasks is difficult as well. Both technically in terms of aggregating/grouping as well as interpersonally because people get defensive or discouraged when you're consistently pointing out shoddy work.
- thedg 4y ago
  > At first glance it looks like I get less done, but when you account for the fact that nobody has to clean up after me or shoehorn a work around for an edge case in I become one of the most efficient. That's so awesome. The Jam dev team recently shipped a major refactor of all the state handling in the app a few weeks ahead of schedule and that's because the team spent so much time upfront documenting and planning the changes. Thanks to Tony Stuck for showing us how important doing upfront planning is on the Jam engineering team!
  - hbrn 4y ago
    > The Jam dev team recently shipped a major refactor of all the state handling in the app a few weeks ahead of schedule To me that doesn't sound like a thing to be proud of. 1. You approved a plan to have a refactor that lasts several weeks. Even for big companies that's usually an anti-pattern. It even contradicts your own advice of "make things better as you go". 2. You tremendously overestimated the required effort (i.e. absolutely failed at planning) 3. You got lucky this time and two wrongs made a right. Also, your example suffers from survivorship bias. Of course everybody has examples where planning paid off. But there are also plenty of examples where planning was a waste of time, or where people were following outdated plan for the sake of following the plan. "More planning" is a bad advice. A good advice helps distinguish areas that benefit from "more planning" from areas that will suffer from it. But that is hard and nuanced.
musk_micropenis 4y ago
All great tips. I would add another bullet point along the lines of, * Make your software observable. Your feature is not finished until you've verified it is being used as expected. Ideally you should have all of the data you need to know if things are going well without having to wait for customers to email you with complaints. Be careful not to develop a culture of developers throwing code "over the fence", either to QA staff or a release team - developers own their code from IDE to production. If you don't develop this culture of ownership you end up with longer feedback loops between customer support, QA and dev.
- tonto 4y ago
  so your users every click is monitored by you? is that what you mean by observability? rather not do that to my users
- thedg 4y ago
  > If you don't develop this culture of ownership you end up with longer feedback loops between customer support, QA and dev. 100%, that's great
- O__________O 4y ago
  Any suggestions for engineering observability?
  - twh270 4y ago
    The most important thing to do IMO is to make it part of your acceptance criteria or definition of done for any story/feature. My observation is that observability gets ignored if it isn't baked into the culture. In terms of implementation, if it isn't on a dashboard or doesn't fire off an alert, it doesn't exist. So put your telemetry and logs on a dashboard, and set up alerts based on your SLA's/SLO's. Also, make it easy for developers to understand, create and edit dashboards/alerts. If it's hard to understand or hard to do, it won't get done.
  - fatnoah 4y ago
    I'll add that incentives can really help as well. Observable software is much more easily operated in production environments, and lets tech ops teams diagnose and fix things with much less involvement from Engineers, which will very much appreciate a much lower escalation rate.
lucgray 4y ago
Just FYI to the author that the embedded link to try out jam.dev in the article doesn't take you to the jam homepage
- thedg 4y ago
  Thank you! Updated!
pfarrell 4y ago
Agree with everything in this blog except the premise Move fast and break things was never about breaking production by being cavalier with your code. It means to don't let inertia accrue, don't fail to challenge assumptions, don't let fear of pushback prevent you from exploring the best course of action. Way easier to achieve in a small organization. edit: Did a bit of digging and I may be wrong here. In an interview in 2014 [0], when Facebook was changing their motto to "move fast with stable infrastructure", Zuckerberg said "As developers, moving quickly was so important, we would even tolerate a few bugs to do it..." I've always interpreted like I originally wrote, but that's not in sync with being ok shipping bugs. 0: https://www.cnet.com/tech/mobile/zuckerberg-move-fast-and-break-things-isnt-how-we-operate-anymore/ https://www.cnet.com/tech/mobile/zuckerberg-move-fast-and-br...
- thedg 4y ago
  Interesting! Well you are right - those are great things to break!
- deltree7 4y ago
  Exactly, the 'break things' part is there to take the fear out of the developer to take more risks. It's complimenting the culture of move fast. Too many orgs don't move fast because of the fear of breaking things.
- hbrn 4y ago
  1. You will never be able to fix all bugs. There's plenty of bugs in your software right that that you are simply not aware of. 2. Bug fixing has diminishing returns and exponential costs. If you accept these two axioms, Zuck's quote makes perfect sense. And certain bugs are simply not worth fixing because cost of fixing it is higher than cost of having it.
  - deleted 4y ago
    [deleted]
  - deleted 4y ago
    [deleted]
hbrn 4y ago
None of these tips are specific to remote. None are insightful either. Write comments, plan better, do things better. Seriously? Just another google spam blog post to drive traffic. Btw, if your team goal is "ship to production" (as per screenshot), you're probably not moving fast enough.
drewcoo 4y ago
The whole point of "move fast and break things" was making an actual choice instead of trying to do everything badly. This puff piece misses the point. It also doesn't tell us what "fast" is at Jam. We only know that it's calendar-driven development with quality gates.
tinglymintyfrsh 4y ago
We move fast and have guardrails such that breaking things permanently is unlikely. All diffs require another engineer to sign-off. There is no directly pushing to production without A Very Very Good Reason(TM). CI/CD and canarying occurs automatically.
revskill 4y ago
OK, i could expect there's tons of hidden tech debts to be discovered as the project goes big. Expect 1 year to migrate to new services. But that's another story, because "If it works in production, don't touch it". Not breaking things is only half of story.
wobbly_bush 4y ago
> Having a daily check in on the project status helps us recognize when we are drifting off track and need to make decisions about whether to change the date or re-prioritize what's in scope Does daily project status seem excessive? Development can be non-linear, so I wonder what decisions are needed to made on a daily basis.
- solatic 4y ago
  Depends on the team. If the team is strong at async communication, writing good status updates whenever such an update exists, then no, daily meetings where everyone would say "well I'm just going to repeat what I wrote elsewhere" are ipso facto excessive and unnecessary. But there are definitely teams where members are poor written communicators, and in such cases, there's nearly always something new that comes up in a daily.
- reb 4y ago
  Daily check-ins don't demand linear progress, do they? I like having frequent checkpoints for brief discussion and visibility, even if they usually pass without a lot of action.
- ketzo 4y ago
  I think daily check-ins are valuable as long as everyone knows that "same as yesterday" is a valid answer, and not problematic in and of itself.
AntiRemoteWork 4y ago
ianlevesque 4y ago
This is just another Waterfall By Another Name process and as a developer all I got out of this was don't work at Jam (whoever that is). > When developers pick up tickets that don't have all the information they need, they need to spend more cycles investigating and waiting for follow up information from the broader team. This could also be interpreted as adding layers of friction between the people experiencing problems and the people fixing those problems. This generally leads to worse outcomes. > After product spec and kick-off, the next step is for engineering to spend a little bit of time writing a technical plan. Oh good, a spec tossed over the fence in yet another meeting, and then another spec before any lines of code are written. No thanks. > Having milestones put on the calendar also gives us the flexibility to move them when needed, but makes it so that we consciously need to decide to move them, rather than letting dates slip by. The dates slip by because software estimation is akin to the Halting Problem. You aren't fixing that, just making your managers feel better. > Having a daily check in on the project status helps us recognize when we are drifting off track and need to make decisions about whether to change the date or re-prioritize what's in scope. Even more meetings, hooray! > All of the tips above are things we started doing as a direct result of something one of us suggested in a retro. Retros are awesome, and are so important. And more meetings about what went wrong in the prior meetings. FAANG doesn't do this. Hire great people, trust them, communicate async frequently, coordinate planning infrequently. This reads like a manifesto for "we hired a bunch of subpar developers and don't give them any ownership or empowerment". Pass.
- kodah 4y ago
  > FAANG doesn't do this. Hire great people, trust them, communicate async frequently, coordinate planning infrequently. This reads like a manifesto for "we hired a bunch of subpar developers and don't give them any ownership or empowerment". Pass. I work at a high performing remote-first company. This is how we do it. Ownership where I work includes an engineer, a TPM, sometimes an exec or manager. The gist is that there's individual ownership over: - Technical outcomes - Business outcomes - Customer outcomes - Integration outcomes Sometimes it can take us a while to agree on direction, but that's okay. For the (potential) slowness with respect to direction, it makes identifying, prioritizing, and fixing bugs much more simple. It also speeds up incident response quite a bit.

156 points by irtefa 4y ago

deleted 4y ago

[deleted]

> Tip #5 – more code comments. I find this to rarely be true in practice. It's not that comments are intrinsically useless (though they do suffer from the intrinsic limitations that are not statically checked or tested), but that most developers tend to write comments for things that benefit from them. Or, when they do need them, they are poorly written or too terse to be useful to anyone else. I would venture to say I've seen maybe or two comments in my 15 years of software engineering that I saw and actually thought "wow, well this helps me understand this a lot better."

physicles 4y ago
I’ve been partying on the same code base for about five years now. I’ve written about 70% of it, but it’s large enough (200k lines or so) that I can’t keep all of its subtleties in my head. In the last year alone I’d estimate comments written by my past self have saved me dozens of hours of effort tracking down the reason why code was implemented a certain way. A couple times I’ve undertaken some refactoring that would’ve taken hours, only to find 10 minutes into the process a comment to the effect of “don’t try to refactor in that way, for these reasons” and it was correct. Maybe my memory is just particularly bad. But if the comments are helpful to me, they’d also be helpful to someone else who didn’t write that code. It does take practice to know when and how to write those comments though.
- xwowsersx 4y ago
  That's great, and some people are better at writing useful comments than others. Not to detract from what you're saying, but > But if the comments are helpful to me, they’d also be helpful to someone else who didn’t write that code. this does not necessarily follow unless you've been extremely thoughtful in how you wrote those comments (which you may very well have been). That is, in fact, the issue most of the time — that comments are written in such a way that they serve to jog the memory of the person who wrote them, but are rarely of general use to anyone else.
- valenterry 4y ago
  Yeah I agree with this notion. After working in very high level and expressive language (with static typing) I found that I could get rid of about 90% or even more of my comments. But for the rest, it's exactly as you described. There is usually some workaround/shortcut being taken or there is interfacing with an external dependency that does not work as expected. In those cases comments make aware of things that the code doesn't show and can save a looot of time.

mattpallissard 4y ago

> hours of planning can save weeks of engineering. This is so hard to get teams to do. I am almost always the slowest to implement a change simply because I don't conflate design and implementation. At first glance it looks like I get less done, but when you account for the fact that nobody has to clean up after me or shoehorn a work around for an edge case in I become one of the most efficient. Assigning cost to subsequent follow up tasks is difficult as well. Both technically in terms of aggregating/grouping as well as interpersonally because people get defensive or discouraged when you're consistently pointing out shoddy work.

thedg 4y ago
> At first glance it looks like I get less done, but when you account for the fact that nobody has to clean up after me or shoehorn a work around for an edge case in I become one of the most efficient. That's so awesome. The Jam dev team recently shipped a major refactor of all the state handling in the app a few weeks ahead of schedule and that's because the team spent so much time upfront documenting and planning the changes. Thanks to Tony Stuck for showing us how important doing upfront planning is on the Jam engineering team!
- hbrn 4y ago
  > The Jam dev team recently shipped a major refactor of all the state handling in the app a few weeks ahead of schedule To me that doesn't sound like a thing to be proud of. 1. You approved a plan to have a refactor that lasts several weeks. Even for big companies that's usually an anti-pattern. It even contradicts your own advice of "make things better as you go". 2. You tremendously overestimated the required effort (i.e. absolutely failed at planning) 3. You got lucky this time and two wrongs made a right. Also, your example suffers from survivorship bias. Of course everybody has examples where planning paid off. But there are also plenty of examples where planning was a waste of time, or where people were following outdated plan for the sake of following the plan. "More planning" is a bad advice. A good advice helps distinguish areas that benefit from "more planning" from areas that will suffer from it. But that is hard and nuanced.

musk_micropenis 4y ago

All great tips. I would add another bullet point along the lines of, * Make your software observable. Your feature is not finished until you've verified it is being used as expected. Ideally you should have all of the data you need to know if things are going well without having to wait for customers to email you with complaints. Be careful not to develop a culture of developers throwing code "over the fence", either to QA staff or a release team - developers own their code from IDE to production. If you don't develop this culture of ownership you end up with longer feedback loops between customer support, QA and dev.

tonto 4y ago
so your users every click is monitored by you? is that what you mean by observability? rather not do that to my users
thedg 4y ago
> If you don't develop this culture of ownership you end up with longer feedback loops between customer support, QA and dev. 100%, that's great
O__________O 4y ago
Any suggestions for engineering observability?
- twh270 4y ago
  The most important thing to do IMO is to make it part of your acceptance criteria or definition of done for any story/feature. My observation is that observability gets ignored if it isn't baked into the culture. In terms of implementation, if it isn't on a dashboard or doesn't fire off an alert, it doesn't exist. So put your telemetry and logs on a dashboard, and set up alerts based on your SLA's/SLO's. Also, make it easy for developers to understand, create and edit dashboards/alerts. If it's hard to understand or hard to do, it won't get done.
- fatnoah 4y ago
  I'll add that incentives can really help as well. Observable software is much more easily operated in production environments, and lets tech ops teams diagnose and fix things with much less involvement from Engineers, which will very much appreciate a much lower escalation rate.

lucgray 4y ago

Just FYI to the author that the embedded link to try out jam.dev in the article doesn't take you to the jam homepage

thedg 4y ago
Thank you! Updated!

pfarrell 4y ago

Agree with everything in this blog except the premise Move fast and break things was never about breaking production by being cavalier with your code. It means to don't let inertia accrue, don't fail to challenge assumptions, don't let fear of pushback prevent you from exploring the best course of action. Way easier to achieve in a small organization. edit: Did a bit of digging and I may be wrong here. In an interview in 2014 [0], when Facebook was changing their motto to "move fast with stable infrastructure", Zuckerberg said "As developers, moving quickly was so important, we would even tolerate a few bugs to do it..." I've always interpreted like I originally wrote, but that's not in sync with being ok shipping bugs. 0: https://www.cnet.com/tech/mobile/zuckerberg-move-fast-and-break-things-isnt-how-we-operate-anymore/ https://www.cnet.com/tech/mobile/zuckerberg-move-fast-and-br...

thedg 4y ago
Interesting! Well you are right - those are great things to break!
deltree7 4y ago
Exactly, the 'break things' part is there to take the fear out of the developer to take more risks. It's complimenting the culture of move fast. Too many orgs don't move fast because of the fear of breaking things.
hbrn 4y ago
1. You will never be able to fix all bugs. There's plenty of bugs in your software right that that you are simply not aware of. 2. Bug fixing has diminishing returns and exponential costs. If you accept these two axioms, Zuck's quote makes perfect sense. And certain bugs are simply not worth fixing because cost of fixing it is higher than cost of having it.
- deleted 4y ago
  [deleted]
- deleted 4y ago
  [deleted]

hbrn 4y ago

None of these tips are specific to remote. None are insightful either. Write comments, plan better, do things better. Seriously? Just another google spam blog post to drive traffic. Btw, if your team goal is "ship to production" (as per screenshot), you're probably not moving fast enough.

drewcoo 4y ago

The whole point of "move fast and break things" was making an actual choice instead of trying to do everything badly. This puff piece misses the point. It also doesn't tell us what "fast" is at Jam. We only know that it's calendar-driven development with quality gates.

tinglymintyfrsh 4y ago

We move fast and have guardrails such that breaking things permanently is unlikely. All diffs require another engineer to sign-off. There is no directly pushing to production without A Very Very Good Reason(TM). CI/CD and canarying occurs automatically.

revskill 4y ago

OK, i could expect there's tons of hidden tech debts to be discovered as the project goes big. Expect 1 year to migrate to new services. But that's another story, because "If it works in production, don't touch it". Not breaking things is only half of story.

wobbly_bush 4y ago

> Having a daily check in on the project status helps us recognize when we are drifting off track and need to make decisions about whether to change the date or re-prioritize what's in scope Does daily project status seem excessive? Development can be non-linear, so I wonder what decisions are needed to made on a daily basis.

solatic 4y ago
Depends on the team. If the team is strong at async communication, writing good status updates whenever such an update exists, then no, daily meetings where everyone would say "well I'm just going to repeat what I wrote elsewhere" are ipso facto excessive and unnecessary. But there are definitely teams where members are poor written communicators, and in such cases, there's nearly always something new that comes up in a daily.
reb 4y ago
Daily check-ins don't demand linear progress, do they? I like having frequent checkpoints for brief discussion and visibility, even if they usually pass without a lot of action.
ketzo 4y ago
I think daily check-ins are valuable as long as everyone knows that "same as yesterday" is a valid answer, and not problematic in and of itself.

AntiRemoteWork 4y ago

ianlevesque 4y ago

This is just another Waterfall By Another Name process and as a developer all I got out of this was don't work at Jam (whoever that is). > When developers pick up tickets that don't have all the information they need, they need to spend more cycles investigating and waiting for follow up information from the broader team. This could also be interpreted as adding layers of friction between the people experiencing problems and the people fixing those problems. This generally leads to worse outcomes. > After product spec and kick-off, the next step is for engineering to spend a little bit of time writing a technical plan. Oh good, a spec tossed over the fence in yet another meeting, and then another spec before any lines of code are written. No thanks. > Having milestones put on the calendar also gives us the flexibility to move them when needed, but makes it so that we consciously need to decide to move them, rather than letting dates slip by. The dates slip by because software estimation is akin to the Halting Problem. You aren't fixing that, just making your managers feel better. > Having a daily check in on the project status helps us recognize when we are drifting off track and need to make decisions about whether to change the date or re-prioritize what's in scope. Even more meetings, hooray! > All of the tips above are things we started doing as a direct result of something one of us suggested in a retro. Retros are awesome, and are so important. And more meetings about what went wrong in the prior meetings. FAANG doesn't do this. Hire great people, trust them, communicate async frequently, coordinate planning infrequently. This reads like a manifesto for "we hired a bunch of subpar developers and don't give them any ownership or empowerment". Pass.

kodah 4y ago
> FAANG doesn't do this. Hire great people, trust them, communicate async frequently, coordinate planning infrequently. This reads like a manifesto for "we hired a bunch of subpar developers and don't give them any ownership or empowerment". Pass. I work at a high performing remote-first company. This is how we do it. Ownership where I work includes an engineer, a TPM, sometimes an exec or manager. The gist is that there's individual ownership over: - Technical outcomes - Business outcomes - Customer outcomes - Integration outcomes Sometimes it can take us a while to agree on direction, but that's okay. For the (potential) slowness with respect to direction, it makes identifying, prioritizing, and fixing bugs much more simple. It also speeds up incident response quite a bit.