6 ms·

John Carmack on lead time to push fixes in startups

72 points by patapong 4y ago

duxup 4y ago
I work on a 4 person team. At one point it is was two of us. I love a quick fix where I tell the customer “try it again” done.
jobs_throwaway 4y ago
>The time to deliver a fix to VR users at Meta was often over six months, disregarding all the problems that were just ignored. I find it astounding that this could be true for someone as senior and respected as Carmack. Is it because he's working in a hardware division? What takes so damn long? Side note - its interesting how Carmack has come out and publicly bashed Meta so much. I guess you can do that when you're in his position.
- ginger2016 4y ago
  Maybe his ego got hurt at Meta. Why would you publicly bash your previous employer. Isn’t staying quiet the right thing to do?
  - tormeh 4y ago
    He's a true VR believer. I think it pained him to watch it go less well than it could.
  - OnlineGladiator 4y ago
    > Why would you publicly bash your previous employer. Maybe because they suck and deserve to be criticized? > Isn’t staying quiet the right thing to do? It's certainly the easier thing to do. It's a sad world we live in where being 'professional' means being dishonest.
- MikeTheRocker 4y ago
  Even software can move slowly because everything has to go through numerous levels of bureaucracy and review. Before you even start working on something, you often need to have meetings with stakeholders from various teams and disciplines to "align" (read: convince them to do the work on their part of the stack). Then on top of that, the Quest OS update shipping cadence is monthly. Source: worked at Meta on Quest system software with Carmack 2018-2021
  - deleted 4y ago
    [deleted]
- Trasmatta 4y ago
  Maybe the regular, 50 people all bikeshedding every single tiny decision rather than just letting the people who know what they're doing get their job done?
- potatolicious 4y ago
  > "Is it because he's working in a hardware division?" Likely? Certainly from all appearances from the outside there are many legitimate bones to pick about Meta's organizational sclerosis, but part of this is structural to the product itself. "I can fix the image quality issues with X relatively easily, I know exactly what needs to be done" elicits (rightfully!) "We operate on an intensely compute-constrained device where every cycle matters. Over-consumption of compute causes battery life to degrade significantly and the device to become uncomfortably warm to wear. How much compute will this solution cost?" Then you have to balance the additional compute with drawing down compute somewhere else. Maybe the tracking camera algorithms can be optimized to consume less compute, which opens room for additional usage elsewhere, but now you've introduced a blocking dependency to delivering this enhancement. Even in a well-functioning organization this prevent shipping as fast as a startup. Part of why startups ship fast isn't just organizational, it's because the products are simpler with fewer technical constraints. Hardware (particularly embedded hardware - which Meta Quests are not precisely, but are pretty close) presents intense levels of constraints that most people who write code on servers (or even PCs) don't often appreciate.
  - tormeh 4y ago
    Meta Quest 2 has 6GB of DDR4 RAM. It has a 64bit CPU. I'd say it has more in common with a server than with an embedded device.
    - __anon-2023__ 4y ago
      > 6GB of DDR4 RAM. It has a 64bit CPU More like a phone from 2020. The Samsung Galaxy S20 base model has 8GB of RAM (the S20FE has 6GB)
- bombcar 4y ago
  Imagine how long it takes even Linus to get a fix into a Linux kernel in a distribution.
  - fragmede 4y ago
    Why would Linus care about the distro kernel when he runs his own?
skellera 4y ago
I’m sure things used to be this way but “we’re like a startup” is so overused in big tech these days. It’s usually an excuse for something wrong rather than the benefits of a startup. I wish we were building for customers instead of building for my or someone in my leadership chain’s promotion. I think most companies start with that focus then lose it after a certain size. Most of the engineers working on projects don’t even care about the product.
- rgblambda 4y ago
  > “we’re like a startup” is so overused in big tech these days. It’s usually an excuse for something wrong rather than the benefits of a startup. I've only ever heard that phrase or something approximating it used to excuse bad practices.
- halvo 4y ago
  My “we’re like a startup” experience was all the uncertainty of a startup along with all the bureaucracy of big tech which resulted in pivoting to a new 12 month roadmap every 6 months…
- allenu 4y ago
  I worked in a "startup-like" team at a big company a few years ago. It was fun in that we moved quickly since we had fewer processes than typical teams at Big Co. Over time though, I realized we were just making stuff to make some higher-up happy, and not really chasing a customer. I also found out we were funded just to light a fire under another team that moved very slowly. Whatever we made would be taken back to that team and we'd have to sell them on using it, and also show them what was possible. Ultimately, we were just a cog in this larger machine. Nowadays I don't really trust anybody at big companies that says "our team is like a startup".
- JohnFen 4y ago
  Yes. "We're like a startup" is a bit of a tipoff that a) the company is no longer like a startup and b) the company has completely forgotten what it's like to be a startup.
- lukefischer 4y ago
simonw 4y ago
Maintaining that velocity inside a larger organization is a really interesting challenge. There are things you can do that can help. The biggest in my experience is robust automated testing combined with robust automated deployments Continuous deployment (any code that passes the tests gets deployed) is incredible if you can get there, but the next best thing is continuous deployment to a staging environment followed by one-click-deploys by trusted release managers to production.
ginger2016 4y ago
Startup probably worked on Carmack’s request because they were afraid he will post a negative review or give negative review about the startup to his friends. A startup can’t afford to piss off a person like John Carmack, they might go out of business, because it might impact their ability to get new customers and raise capital. Meta on other hand could care less about John Carmack’s opinion as it has no impact to their business. In short John Carmack’s opinion was ignored it had no impact for a large company. The mistake John Carmack is making is that he thinks his views are special, they are not.
- bombcar 4y ago
  I don't know that's true, and he may not have even blathered his name out there. I know that I got some really responsive work from startups we purchased from, and it was very nice, and we were just one of their customers. Sadly, once they pivot away from whatever it was then it all goes away.
- alexashka 4y ago
  Is that why Meta never hired John Carmack? Because his opinions have no impact on their business?
  - qbrass 4y ago
    No, it's because he never applied for a job there.
- Sakos 4y ago
  I work at a small company and customer issues or requests can be solved (or worked on) within a day because there's almost zero red tape. And it doesn't need to be Carmack pointing out the issue or delivering the request for it to be done.
- Nimitz14 4y ago
  Downvoted: You clearly have no startup experience.
  - 4y ago
codetrotter 4y ago
> I was pretty sure I knew what most of the problems were, so I got in touch with the developers, and everything got sorted out in a few days (they were already aware of some issues). Yeah but like, I get that startups are better at listening to customers and adjusting quickly. But I can confidently say.. if John Carmack himself legit sent me an email pointing out some problem with something I made. You bet your ass I would respond and listen to what was on his mind. Regardless of the size of my company and the amount of work on my plate :p
- jayparth 4y ago
  But he was doing that daily at Meta, and it still took 6 months for things to happen!
  - bombcar 4y ago
    Something something ancient Chinese parable the only person not impressed by the General/Emperor/Mystic Sage is his wife.
  - deleted 4y ago
    [deleted]
  - bmitc 4y ago
    Do we know that the things he wanted to fix were important enough to fix?
    - nh23423fefe 4y ago
      Yes we know that.
    - lukefischer 4y ago
      Yeah they were. We are writing a blog about it so everyone knows.
- Kranar 4y ago
  But his entire point is that's not what happened at the big company he worked at.
rcme 4y ago
At my last company we had a Discord that our users would use to give use feedback, bug reports, and ideas. It was amazing. The feedback itself was really helpful in building the product, but you instantly acquire a user's loyalty the first time they report an issue and you fix it that day. Users are really grateful for that.
baby 4y ago
Honestly I don’t know what to do with this tweet + HN thread because it leaves little room for actual discussions and a lot of room for misguided guessing. Unless someone from the org comments here, which I doubt considering the animosity HN has at VR or Meta, we’ll never know why oculus didn’t work like that, if it was a real issue there or if Carmack is exaggerating, if it was an issue are tweets like going to help the org rethink itself? The only signal we’re getting, from the outside, are the releases. I had the go and the quest and the gap in technology between the two were amazing. I don’t know about quest to quest 2, or quest 2 to quest pro, and I guess we’ll see what the quest 3 delivers. Maybe other people know?
hideo 4y ago
I have great respect for John Carmack but this feels like a very unnuanced take. Lead time isn't about "startups", it's about code complexity, side effects and blast radius. The time it takes to push code when you have 10 use-cases and 1,000 users, is very different from the time it takes to push code when you have 1,000 use-cases but 10,000,000 users. At that scale, it takes A LOT of effort to keep fix times small. It isn't going to occur naturally. Some times the effort to keep that fix time small may not be worth it, some times it might.
- prowseed 4y ago
  OR you can work out your architecture well so that you don't have to worry about other use-cases :)
deleted 4y ago
[deleted]
bmitc 4y ago
Isn't this him just bragging, which gets extrapolated to some platitude about startups? By the same token of startups supposedly being able to fix things faster, which is debatable, it is more likely that they have more problems by moving faster without that many users. Things change a bit when you have millions, hundreds of millions, and even billions of users.
- kraseng 4y ago
  Nope. Not debatable. Not supposedly. Effective startups move faster … or they are not effective startups and they die. The fact that not all startups die literally shows that not all are creating more problems than they are solving. Large orgs can be pretty nimble too if they are properly set up and have the right people btw
Barrin92 4y ago
You can't and really should not expect this kind of speed from a large company. In this case he used a service where you can pay 170$ to get a custom satellite image from space. My guess is they treat every customer individually and the technical issue reported was obvious. At Meta you have hundreds of millions if not more users and any issue largely isn't so obvious that you can just go and fix five lines of code. When someone comes around and says "hey I'm Bob xyz isn't working" you probably need to figure out first if that's a real issue at all and if fixing it doesn't break ten other things. The potential downside is much, much larger if you overreact to individual complaints at that scale.
- fwlr 4y ago
  Counterpoint: Carmack worked at Meta on the Oculus, he had as much visibility as it’s possible to have on the problems, their fixes, and whether the fixes would break things. He doesn’t spell it out in the tweet but the kinds of issues he talks about the satellite image having sound like the kind of issues you might see in VR hardware and software too, and the fact that he immediately noticed and correctly guessed what was going wrong in the satellite images suggests he had tackled the exact same problem while working on the Oculus. In fact, I’m pretty sure that’s why he made this tweet: he saw a problem he’d seen before, he suggested the same solution he’d suggested before, and he saw the solution implemented in days instead of months.
6451937099 4y ago
[dead]
6451937099 4y ago
[dead]

72 points by patapong 4y ago

I work on a 4 person team. At one point it is was two of us. I love a quick fix where I tell the customer “try it again” done.

jobs_throwaway 4y ago

>The time to deliver a fix to VR users at Meta was often over six months, disregarding all the problems that were just ignored. I find it astounding that this could be true for someone as senior and respected as Carmack. Is it because he's working in a hardware division? What takes so damn long? Side note - its interesting how Carmack has come out and publicly bashed Meta so much. I guess you can do that when you're in his position.

ginger2016 4y ago
Maybe his ego got hurt at Meta. Why would you publicly bash your previous employer. Isn’t staying quiet the right thing to do?
- tormeh 4y ago
  He's a true VR believer. I think it pained him to watch it go less well than it could.
- OnlineGladiator 4y ago
  > Why would you publicly bash your previous employer. Maybe because they suck and deserve to be criticized? > Isn’t staying quiet the right thing to do? It's certainly the easier thing to do. It's a sad world we live in where being 'professional' means being dishonest.
MikeTheRocker 4y ago
Even software can move slowly because everything has to go through numerous levels of bureaucracy and review. Before you even start working on something, you often need to have meetings with stakeholders from various teams and disciplines to "align" (read: convince them to do the work on their part of the stack). Then on top of that, the Quest OS update shipping cadence is monthly. Source: worked at Meta on Quest system software with Carmack 2018-2021
- deleted 4y ago
  [deleted]
Trasmatta 4y ago
Maybe the regular, 50 people all bikeshedding every single tiny decision rather than just letting the people who know what they're doing get their job done?
potatolicious 4y ago
> "Is it because he's working in a hardware division?" Likely? Certainly from all appearances from the outside there are many legitimate bones to pick about Meta's organizational sclerosis, but part of this is structural to the product itself. "I can fix the image quality issues with X relatively easily, I know exactly what needs to be done" elicits (rightfully!) "We operate on an intensely compute-constrained device where every cycle matters. Over-consumption of compute causes battery life to degrade significantly and the device to become uncomfortably warm to wear. How much compute will this solution cost?" Then you have to balance the additional compute with drawing down compute somewhere else. Maybe the tracking camera algorithms can be optimized to consume less compute, which opens room for additional usage elsewhere, but now you've introduced a blocking dependency to delivering this enhancement. Even in a well-functioning organization this prevent shipping as fast as a startup. Part of why startups ship fast isn't just organizational, it's because the products are simpler with fewer technical constraints. Hardware (particularly embedded hardware - which Meta Quests are not precisely, but are pretty close) presents intense levels of constraints that most people who write code on servers (or even PCs) don't often appreciate.
- tormeh 4y ago
  Meta Quest 2 has 6GB of DDR4 RAM. It has a 64bit CPU. I'd say it has more in common with a server than with an embedded device.
  - __anon-2023__ 4y ago
    > 6GB of DDR4 RAM. It has a 64bit CPU More like a phone from 2020. The Samsung Galaxy S20 base model has 8GB of RAM (the S20FE has 6GB)
bombcar 4y ago
Imagine how long it takes even Linus to get a fix into a Linux kernel in a distribution.
- fragmede 4y ago
  Why would Linus care about the distro kernel when he runs his own?

skellera 4y ago

I’m sure things used to be this way but “we’re like a startup” is so overused in big tech these days. It’s usually an excuse for something wrong rather than the benefits of a startup. I wish we were building for customers instead of building for my or someone in my leadership chain’s promotion. I think most companies start with that focus then lose it after a certain size. Most of the engineers working on projects don’t even care about the product.

rgblambda 4y ago
> “we’re like a startup” is so overused in big tech these days. It’s usually an excuse for something wrong rather than the benefits of a startup. I've only ever heard that phrase or something approximating it used to excuse bad practices.
halvo 4y ago
My “we’re like a startup” experience was all the uncertainty of a startup along with all the bureaucracy of big tech which resulted in pivoting to a new 12 month roadmap every 6 months…
allenu 4y ago
I worked in a "startup-like" team at a big company a few years ago. It was fun in that we moved quickly since we had fewer processes than typical teams at Big Co. Over time though, I realized we were just making stuff to make some higher-up happy, and not really chasing a customer. I also found out we were funded just to light a fire under another team that moved very slowly. Whatever we made would be taken back to that team and we'd have to sell them on using it, and also show them what was possible. Ultimately, we were just a cog in this larger machine. Nowadays I don't really trust anybody at big companies that says "our team is like a startup".
JohnFen 4y ago
Yes. "We're like a startup" is a bit of a tipoff that a) the company is no longer like a startup and b) the company has completely forgotten what it's like to be a startup.
lukefischer 4y ago

simonw 4y ago

Maintaining that velocity inside a larger organization is a really interesting challenge. There are things you can do that can help. The biggest in my experience is robust automated testing combined with robust automated deployments Continuous deployment (any code that passes the tests gets deployed) is incredible if you can get there, but the next best thing is continuous deployment to a staging environment followed by one-click-deploys by trusted release managers to production.

ginger2016 4y ago

Startup probably worked on Carmack’s request because they were afraid he will post a negative review or give negative review about the startup to his friends. A startup can’t afford to piss off a person like John Carmack, they might go out of business, because it might impact their ability to get new customers and raise capital. Meta on other hand could care less about John Carmack’s opinion as it has no impact to their business. In short John Carmack’s opinion was ignored it had no impact for a large company. The mistake John Carmack is making is that he thinks his views are special, they are not.

bombcar 4y ago
I don't know that's true, and he may not have even blathered his name out there. I know that I got some really responsive work from startups we purchased from, and it was very nice, and we were just one of their customers. Sadly, once they pivot away from whatever it was then it all goes away.
alexashka 4y ago
Is that why Meta never hired John Carmack? Because his opinions have no impact on their business?
- qbrass 4y ago
  No, it's because he never applied for a job there.
Sakos 4y ago
I work at a small company and customer issues or requests can be solved (or worked on) within a day because there's almost zero red tape. And it doesn't need to be Carmack pointing out the issue or delivering the request for it to be done.
Nimitz14 4y ago
Downvoted: You clearly have no startup experience.
- 4y ago

codetrotter 4y ago

> I was pretty sure I knew what most of the problems were, so I got in touch with the developers, and everything got sorted out in a few days (they were already aware of some issues). Yeah but like, I get that startups are better at listening to customers and adjusting quickly. But I can confidently say.. if John Carmack himself legit sent me an email pointing out some problem with something I made. You bet your ass I would respond and listen to what was on his mind. Regardless of the size of my company and the amount of work on my plate :p

jayparth 4y ago
But he was doing that daily at Meta, and it still took 6 months for things to happen!
- bombcar 4y ago
  Something something ancient Chinese parable the only person not impressed by the General/Emperor/Mystic Sage is his wife.
- deleted 4y ago
  [deleted]
- bmitc 4y ago
  Do we know that the things he wanted to fix were important enough to fix?
  - nh23423fefe 4y ago
    Yes we know that.
  - lukefischer 4y ago
    Yeah they were. We are writing a blog about it so everyone knows.
Kranar 4y ago
But his entire point is that's not what happened at the big company he worked at.

rcme 4y ago

At my last company we had a Discord that our users would use to give use feedback, bug reports, and ideas. It was amazing. The feedback itself was really helpful in building the product, but you instantly acquire a user's loyalty the first time they report an issue and you fix it that day. Users are really grateful for that.

baby 4y ago

Honestly I don’t know what to do with this tweet + HN thread because it leaves little room for actual discussions and a lot of room for misguided guessing. Unless someone from the org comments here, which I doubt considering the animosity HN has at VR or Meta, we’ll never know why oculus didn’t work like that, if it was a real issue there or if Carmack is exaggerating, if it was an issue are tweets like going to help the org rethink itself? The only signal we’re getting, from the outside, are the releases. I had the go and the quest and the gap in technology between the two were amazing. I don’t know about quest to quest 2, or quest 2 to quest pro, and I guess we’ll see what the quest 3 delivers. Maybe other people know?

hideo 4y ago

I have great respect for John Carmack but this feels like a very unnuanced take. Lead time isn't about "startups", it's about code complexity, side effects and blast radius. The time it takes to push code when you have 10 use-cases and 1,000 users, is very different from the time it takes to push code when you have 1,000 use-cases but 10,000,000 users. At that scale, it takes A LOT of effort to keep fix times small. It isn't going to occur naturally. Some times the effort to keep that fix time small may not be worth it, some times it might.

prowseed 4y ago
OR you can work out your architecture well so that you don't have to worry about other use-cases :)

deleted 4y ago

[deleted]

bmitc 4y ago

Isn't this him just bragging, which gets extrapolated to some platitude about startups? By the same token of startups supposedly being able to fix things faster, which is debatable, it is more likely that they have more problems by moving faster without that many users. Things change a bit when you have millions, hundreds of millions, and even billions of users.

kraseng 4y ago
Nope. Not debatable. Not supposedly. Effective startups move faster … or they are not effective startups and they die. The fact that not all startups die literally shows that not all are creating more problems than they are solving. Large orgs can be pretty nimble too if they are properly set up and have the right people btw

Barrin92 4y ago

You can't and really should not expect this kind of speed from a large company. In this case he used a service where you can pay 170$ to get a custom satellite image from space. My guess is they treat every customer individually and the technical issue reported was obvious. At Meta you have hundreds of millions if not more users and any issue largely isn't so obvious that you can just go and fix five lines of code. When someone comes around and says "hey I'm Bob xyz isn't working" you probably need to figure out first if that's a real issue at all and if fixing it doesn't break ten other things. The potential downside is much, much larger if you overreact to individual complaints at that scale.

fwlr 4y ago
Counterpoint: Carmack worked at Meta on the Oculus, he had as much visibility as it’s possible to have on the problems, their fixes, and whether the fixes would break things. He doesn’t spell it out in the tweet but the kinds of issues he talks about the satellite image having sound like the kind of issues you might see in VR hardware and software too, and the fact that he immediately noticed and correctly guessed what was going wrong in the satellite images suggests he had tackled the exact same problem while working on the Oculus. In fact, I’m pretty sure that’s why he made this tweet: he saw a problem he’d seen before, he suggested the same solution he’d suggested before, and he saw the solution implemented in days instead of months.

6451937099 4y ago

[dead]

6451937099 4y ago

[dead]