9 ms·

Tell me you never ran a large code base in production, without telling me. Past a certain point there are no "simple code changes" anymore. Automation and func

Tell me you never ran a large code base in production, without telling me. Past a certain point there are no "simple code changes" anymore. Automation and functional QA needs to ensure zero regressions, internal and external docs need to be considered, it all needs to be bundled in with many other code changes (feature and fixes), etc. There is no shortcut. Even FB had to dial back their "break shit in prod" mantra. Of course startup devs hate this, real men fuck around in production, which is great. Running and maintaining a large code base is very grown up sport, more like running a country than building a house. For those on the other side, who inherit startup code and have to fold it in ... the hatred is mutual.

dllthomas 5y ago
> very grown up sport, more like running a country than building a house We need to stop calling things "grown up" when what we mean is that they take a different set of skills. It's childish, and I think it's been causing harm in our industry.
- dools 5y ago
  So it’s okay to refer to something as childish but not as grown up?
  - roflulz 5y ago
    so much irony
  - dllthomas 5y ago
    It's okay to refer to capabilities that should be expected of everyone as "grown up", and falling short of those standards as "childish". It is harmful to extend that to specialized skills, because it leads people to forget that they are specialized skills that aren't going to be learned through mere maturation (and that their lack doesn't necessarily imply immaturity).
- doctor_eval 5y ago
  I agree with this and it’s how I felt when I read the parent. A recent startup I was involved with hired a senior manager from the “grown up” end of town and he is in the process of comprehensively destroying the business because the rules and processes are more important to him than shipping product. He literally has no idea what he is doing, and it’s entirely because he doesn’t have the skill set necessary to bootstrap a company. There is a stark difference between the mindset and skills required to start from zero versus those required to defend billions in revenue, and it has nothing to do with being “grown up”. “Tell me you never bootstrapped a startup, without telling me”.
timdaub 5y ago
I dislike your condescending tone and I also think your speech doesn't generalize. There are large code bases in production that are productive: Everything that is open source - and still the process isn't as shit as op describes. So this has to do with corporate hierarchies and not the quality of the code.
- danielmarkbruce 5y ago
  The original post is pretty condescending, it's just worded slightly less so. Now, that doesn't mean the very best response is to be condescending, but the invitation is right there. Sometimes a reasonable response to ~ "can you believe these idiots!" is "actually, look in the mirror, and squint a little".
- bastijn 5y ago
  > There are large code bases in production that are productive: Everything that is open source Yes there are but these are not without rules. Try to add a new feature to a large open source project as a new contributor. That has probably as much chance of succeeding as doing this in an enterprise. There is a reason larger OSS projects have the "good first contribution" section. These are small bug fixes where the work is already spelled out typically. You do these to get familiar with the codebase and learn how to make a valuable contribution the size of a feature.
zepolen 5y ago
> Tell me you never ran a large code base in production, without telling me. Your post does a good job of that.
throwaway2331 5y ago
Man, unvarnished, lacking agenda---besides the stroking of one's ego---and blunt communication: I missed you so. I agree with the sentiment. Being an "adult" is about restrictions, limits, boundaries, and confines; you can't just willy-nilly do whatever you feel like. You must now realize and accept that we all bend the knee to something. We are not immortal gods, free from all consequence, but merely ants allowed to live by the systems we inhabit. Perhaps the most authentic post I've seen on this entire site in a long time.
TameAntelope 5y ago
You're presenting one way of running, "a large code base in production" as if it's the only way. It's not. The other way is to deploy frequently, accept some amount of bugginess, and build your systems to deal with it appropriately. Running and maintaining a large code base is not the huge technical challenge it once was, but unfortunately the relics of the past remain, and force everyone to do things their way, lest their hard-won knowledge becomes obsolete.
- LilBytes 5y ago
  Maybe they are frequently deploying to prod, but those pipelines are being used by other processes and commits. There's a big differece between committing small changes to an existing service or infrastructure base frequently, and conversely a PR or feature request from left field from a recent acquire.
- nightowl_games 5y ago
  The code bases we have today are vastly larger than at any point in history and grow larger every day. The past is not so black and white.
- deleted 5y ago
  [deleted]
carabiner 5y ago
You guys complain about things taking quarters to complete. In other industries like aerospace, small changes take years.
matwood 5y ago
There is a large gradient between 'fucking around in production' and crippling big corporate process. Maybe the big tech companies all have the perfect amount of process, but much of the process and ceremony at other big companies I've seen is simply in place to keep the lowest common developer employed. Heck, people complain about excessive process on this very site all the time. On other nitpick is that startups live and die with automation by their lean nature. They have fewer resources and must automate everything they can. It's built into startups mindset.
- mike_d 5y ago
  Try to think about the other side of this. Do you want to be standing in line at the grocery store and have your debit card just not work at all because Visa aquired this dudes startup and he didn't understand that feature X wasn't implemented because it would swamp the 10 megabit VPN link between Visa and your credit union? It isn't about keeping the lowest common developer employed, it is about keeping the lowest common developer from bricking the entire system. Unless of course you can tell me where to find enough developers of your skill level to write all the code required to run a modern society.
  - lmeyerov 5y ago
    consider $bigco making $10B year that's $20K/minute chances are, $shiny_acquisition isn't making $20K/minute, and woe is the helpful IT responder who kills the golden goose. it'd be crazy if anyone can do that, so many layers of bureaucracy & compliance have evolved to prevent that. The solution is generally (a) do work not involving production (b) work where production isn't like that, which is generally smaller companies or a surprisingly small number of ~tech companies. Digital Transformation (cloud, PaaS, ...) often involves trying to enable self-serve at enterprise-scale, but I generally still see roadblocks when GPUs, TLS, DNS, SaaS, etc, get involved.
  - megablast 5y ago
    Instead of sending the credit card ID number across the phone line, we will take a picture of the credit card for extra security, and send a full RAW PNG each time.
satyrnein 5y ago
For those on the other side, who inherit startup code and have to fold it in ... the hatred is mutual. Yeah, I was wondering what this might sound like to the other side, perhaps something like: "Yeah, hi, I just got here last week, I'm here to inject innovation into your life. So your battle tested deployment pipelines that deploy all the code that pays both of our paychecks (because my stuff is pre-revenue, naturally), they don't support the idiosyncratic workflow I literally made up myself. Can you drop everything and jump on this instead?"
kkjjkgjjgg 5y ago
But Microservices?
TheGigaChad 5y ago

dllthomas 5y ago

> very grown up sport, more like running a country than building a house We need to stop calling things "grown up" when what we mean is that they take a different set of skills. It's childish, and I think it's been causing harm in our industry.

dools 5y ago
So it’s okay to refer to something as childish but not as grown up?
- roflulz 5y ago
  so much irony
- dllthomas 5y ago
  It's okay to refer to capabilities that should be expected of everyone as "grown up", and falling short of those standards as "childish". It is harmful to extend that to specialized skills, because it leads people to forget that they are specialized skills that aren't going to be learned through mere maturation (and that their lack doesn't necessarily imply immaturity).
doctor_eval 5y ago
I agree with this and it’s how I felt when I read the parent. A recent startup I was involved with hired a senior manager from the “grown up” end of town and he is in the process of comprehensively destroying the business because the rules and processes are more important to him than shipping product. He literally has no idea what he is doing, and it’s entirely because he doesn’t have the skill set necessary to bootstrap a company. There is a stark difference between the mindset and skills required to start from zero versus those required to defend billions in revenue, and it has nothing to do with being “grown up”. “Tell me you never bootstrapped a startup, without telling me”.

timdaub 5y ago

I dislike your condescending tone and I also think your speech doesn't generalize. There are large code bases in production that are productive: Everything that is open source - and still the process isn't as shit as op describes. So this has to do with corporate hierarchies and not the quality of the code.

danielmarkbruce 5y ago
The original post is pretty condescending, it's just worded slightly less so. Now, that doesn't mean the very best response is to be condescending, but the invitation is right there. Sometimes a reasonable response to ~ "can you believe these idiots!" is "actually, look in the mirror, and squint a little".
bastijn 5y ago
> There are large code bases in production that are productive: Everything that is open source Yes there are but these are not without rules. Try to add a new feature to a large open source project as a new contributor. That has probably as much chance of succeeding as doing this in an enterprise. There is a reason larger OSS projects have the "good first contribution" section. These are small bug fixes where the work is already spelled out typically. You do these to get familiar with the codebase and learn how to make a valuable contribution the size of a feature.

zepolen 5y ago

> Tell me you never ran a large code base in production, without telling me. Your post does a good job of that.

throwaway2331 5y ago

Man, unvarnished, lacking agenda---besides the stroking of one's ego---and blunt communication: I missed you so. I agree with the sentiment. Being an "adult" is about restrictions, limits, boundaries, and confines; you can't just willy-nilly do whatever you feel like. You must now realize and accept that we all bend the knee to something. We are not immortal gods, free from all consequence, but merely ants allowed to live by the systems we inhabit. Perhaps the most authentic post I've seen on this entire site in a long time.

TameAntelope 5y ago

You're presenting one way of running, "a large code base in production" as if it's the only way. It's not. The other way is to deploy frequently, accept some amount of bugginess, and build your systems to deal with it appropriately. Running and maintaining a large code base is not the huge technical challenge it once was, but unfortunately the relics of the past remain, and force everyone to do things their way, lest their hard-won knowledge becomes obsolete.

LilBytes 5y ago
Maybe they are frequently deploying to prod, but those pipelines are being used by other processes and commits. There's a big differece between committing small changes to an existing service or infrastructure base frequently, and conversely a PR or feature request from left field from a recent acquire.
nightowl_games 5y ago
The code bases we have today are vastly larger than at any point in history and grow larger every day. The past is not so black and white.
deleted 5y ago
[deleted]

carabiner 5y ago

You guys complain about things taking quarters to complete. In other industries like aerospace, small changes take years.

matwood 5y ago

There is a large gradient between 'fucking around in production' and crippling big corporate process. Maybe the big tech companies all have the perfect amount of process, but much of the process and ceremony at other big companies I've seen is simply in place to keep the lowest common developer employed. Heck, people complain about excessive process on this very site all the time. On other nitpick is that startups live and die with automation by their lean nature. They have fewer resources and must automate everything they can. It's built into startups mindset.

mike_d 5y ago
Try to think about the other side of this. Do you want to be standing in line at the grocery store and have your debit card just not work at all because Visa aquired this dudes startup and he didn't understand that feature X wasn't implemented because it would swamp the 10 megabit VPN link between Visa and your credit union? It isn't about keeping the lowest common developer employed, it is about keeping the lowest common developer from bricking the entire system. Unless of course you can tell me where to find enough developers of your skill level to write all the code required to run a modern society.
- lmeyerov 5y ago
  consider $bigco making $10B year that's $20K/minute chances are, $shiny_acquisition isn't making $20K/minute, and woe is the helpful IT responder who kills the golden goose. it'd be crazy if anyone can do that, so many layers of bureaucracy & compliance have evolved to prevent that. The solution is generally (a) do work not involving production (b) work where production isn't like that, which is generally smaller companies or a surprisingly small number of ~tech companies. Digital Transformation (cloud, PaaS, ...) often involves trying to enable self-serve at enterprise-scale, but I generally still see roadblocks when GPUs, TLS, DNS, SaaS, etc, get involved.
- megablast 5y ago
  Instead of sending the credit card ID number across the phone line, we will take a picture of the credit card for extra security, and send a full RAW PNG each time.

satyrnein 5y ago

For those on the other side, who inherit startup code and have to fold it in ... the hatred is mutual. Yeah, I was wondering what this might sound like to the other side, perhaps something like: "Yeah, hi, I just got here last week, I'm here to inject innovation into your life. So your battle tested deployment pipelines that deploy all the code that pays both of our paychecks (because my stuff is pre-revenue, naturally), they don't support the idiosyncratic workflow I literally made up myself. Can you drop everything and jump on this instead?"

kkjjkgjjgg 5y ago

But Microservices?

TheGigaChad 5y ago