4 ms·

I once said to the CTO of the company I worked for "do we back up our source code"? He said, "no, it's on github". I said no more.

Dylan16807 1y ago
If nobody has the repo checked out, what are the odds it's important?
- bryant 1y ago
  > If nobody has the repo checked out, what are the odds it's important? Oh boy. Tons of apps in maintenance mode run critical infrastructure and see few commits in a year.
  - Dylan16807 1y ago
    And the people using it multiple times a year delete it afterwards?
    - RealStickman_ 1y ago
      relying on random local copies as a backup strategy is not a strategy.
      - bufferoverflow 1y ago
        [dead]
    - bryant 1y ago
      > And the people using it multiple times a year delete it afterwards? The people wouldn't, but in the environments I'm thinking of, security policies might. What you're leaning into is a high-risk backup strategy that would rely mostly on luck to get something remotely close to the current version back online. It's pretty reckless.
      - darkwater 1y ago
        > The people wouldn't, but in the environments I'm thinking of, security policies might. In environments that go so far (deleting local checkouts of code out of security concerns), I bet they do have a mirror/copy of the version controlled code.
    - Lammy 1y ago
      More like “none of the people who worked on it are at the company any more”
    - shakna 1y ago
      They often only have a binary that you would have to reverse engineer. Source code gets lost. To step outside just utility programs, the reason why Command & Conquer didn't have a remaster was: > I'm not going to get into this conversation, but I feel this needs to be answered. During this project of getting the games on Steam, no source code from any legacy games showed up in the archives.
- NewJazz 1y ago
  Devs clean up their workstation sometimes. You can get fancy about deleting build artifacts or just remove the whole directory. Devs move to new machines sometimes and don't always transfer everything. Devs leave. Software still runs, and if you don't have the source then you'll only have the binary or other build artifact.
- OutOfHere 1y ago
  You clearly haven't worked much with code over many years. When laptops change, not all existing projects get checked out. In fact, in VSCode, one can use a project without cloning and checking it out at all.
  - Dylan16807 1y ago
    Honestly I'm just really wondering what the odds are. In particular for code that made it onto git.
    - OutOfHere 1y ago
      Over the long term, the odds reach 100% that it won't be checked out. That's because people mostly only work on newer projects. As for mature older projects, even if they're running in production, cease to see many/any updates, and so they don't get cloned on to newer laptops. This doesn't mean that the older projects are now less important, because if they ever need to be re-deployed to production, only the source code will allow it.
    - CRConrad 1y ago
      > In particular for code that made it onto git. By “onto git”, do you mean “onto GitHub”? I really wish people would stop conflating the two. In fact, I can't tell whether this confusion is just another symptom of, or a (major?) part of the reason for, why we're in the mess we're in.
      - Dylan16807 1y ago
        I do not mean that, and I was not conflating anything. Just making and committing to a repo at all is a step that implies a certain level of caretaking.
- burnt-resistor 1y ago
  Popularity != importance. There is plenty of absolutely critical FOSS code that receives very little maintenance and attention, yet is mission critical to society functioning efficiently. And the same happens in organizations too, with say their bootloader for firmware of hardware products.
simondotau 1y ago
If the repo is on GitHub and two or more developers keep reasonably up-to-date checkouts on their local computers, the “3-2-1” principle of backups is satisfied. Additionally to that, if any of those developers have a backup strategy for their local computer, those also count as a backup of that source code.
- wewewedxfgdf 1y ago
  CTO explaining that to the CEO when your source code is completely gone: CTO: "I know our entire github repo is deleted and all our source code is gone and we never took backups, but I'm hoping the developers might have it all on their machines." CEO: "Hoping developers had it locally was your strategy for protecting all our source code?" CTO: "It's a sound approach and ticks all the boxes." CEO: "You're fired." Board Directors to CEO: "You're fired."
- koonsolo 1y ago
  The benefit of DVCS. Losing the source code from github when it's all on local computers is the least of problems.
- Tohsig 1y ago
  Technically true, but only if we consider dev checkouts as "backups". In the majority of cases they probably are, but that's not guaranteed. The local repo copy could be in a wildly different state than the primary origin, a shallow clone, etc... While the odds of that mattering are very low, they're not zero. I personally prefer to have a dedicated mirror as a failsafe.
OutOfHere 1y ago
I don't understand how the VC financiers got so rich while being so stupid as to hire such stupid people at the executive level, e.g. your CTO.
- pintxo 1y ago
  If only 10 out of 100 of your investments make it, does it matter if one of the 90 failed because of lacking backups? Their risk strategy is diversification of investments. Not making each investment itself bulletproof.
  - OutOfHere 1y ago
    Yes, it is in effect a gamble. The issue is that this strategy doesn't really prove profitable for the majority of VCs. Less than 30% of VCs get to a unicorn or IPO deal. 46% of VCs don't profit at all. This is as per the recent post "(Only) half of senior VCs make at least one successful deal.". I am even ignoring the ones who drop out and don't contribute to the active statistics. The strategy is about as silly as having ten babies and expecting that one of them will make it. It is what you would expect out of the worst poverty-ridden parts of Africa. An alternative is to select and nurture your investments really well, so the rate of success is much higher. I'd like to see the script be flipped, whereby 90% of investments go on to becoming profit making, secondarily with their stable cash income being preferred to big exits.
gerdesj 1y ago
I would suggest your CTO needs some gentle reminders about risk management and how the cloud really works. If your boss is that daft, its probably a sign to bail out. Remember to do your own personal due diligence. Due dill is not something that you just do for someone else: do your own! Do your own personal risk assessment. If code was lost, who would be found accountable? You or them? EDIT: PS - I am a CTO ...
- wewewedxfgdf 1y ago
  But it's github ...... it can't be lost. Unless github closes the account, or a hacker gets access, or a rogue employee gets mad and deletes all, or some development accident results in repo deletion, or etc etc.
  - weikju 1y ago
    Gotta add AI agents to that list
  - SanjayMehta 1y ago
    Or if the one employee who created the account and was paying for it on his personal credit card and then got laid off. And no one else in the company knew what GitHub was.
    - robotnikman 1y ago
      Is there a story behind this oddly specific comment?
      - speeder 1y ago
        Not op but I saw this happen kinda maliciously... Company was founded by "visionary" CEO that is an sculptor and doesn't understand tech. Most crucial employee is young guy hired right out of college. Company now has 50+ developers, but the critical software is still made by that guy several years later. Critical guy still get recent graduate salary, notices he is actually the most important person in the company, and asks for a raise. Company denies his raise. He quits. Company hire a bunch of developers from cheap third world countries to replace him... Company learns: 1. All software guy made a bit before he asked for a raise was on his personal, not company github. 2. The software is in other programming languages, not the one the company uses normally. 3. Everything the guy wrote since he joined the company, is extremely difficult to maintain, guy is a genius and all his code is correct, clean and well made, but his thought patterns and how they end in the code are just too different, and all people that worked with him before also were geniuses and didn't care the code was "crazy". Note: the "other geniuses" mentioned above, also quit when other companies made them great offer and the stingy employer refused them a raise too.
klysm 1y ago
I mean it’s also on everybody’s laptop. Recovering from GitHub going away would be trivial for me
burnt-resistor 1y ago
LMAO. Must be one of those MBA CTOs. At least mirror the crown jewels to bitbucket, Tarsnap, or somewhat else that has 2 weeks - 3 months worth of independent copies made daily. If not MBA, the problem may also stem from the gradual atrophy and disrespect shown towards the sysadmin profession.
bufferoverflow 1y ago
[dead]

Dylan16807 1y ago

If nobody has the repo checked out, what are the odds it's important?

bryant 1y ago
> If nobody has the repo checked out, what are the odds it's important? Oh boy. Tons of apps in maintenance mode run critical infrastructure and see few commits in a year.
- Dylan16807 1y ago
  And the people using it multiple times a year delete it afterwards?
  - RealStickman_ 1y ago
    relying on random local copies as a backup strategy is not a strategy.
    - bufferoverflow 1y ago
      [dead]
  - bryant 1y ago
    > And the people using it multiple times a year delete it afterwards? The people wouldn't, but in the environments I'm thinking of, security policies might. What you're leaning into is a high-risk backup strategy that would rely mostly on luck to get something remotely close to the current version back online. It's pretty reckless.
    - darkwater 1y ago
      > The people wouldn't, but in the environments I'm thinking of, security policies might. In environments that go so far (deleting local checkouts of code out of security concerns), I bet they do have a mirror/copy of the version controlled code.
  - Lammy 1y ago
    More like “none of the people who worked on it are at the company any more”
  - shakna 1y ago
    They often only have a binary that you would have to reverse engineer. Source code gets lost. To step outside just utility programs, the reason why Command & Conquer didn't have a remaster was: > I'm not going to get into this conversation, but I feel this needs to be answered. During this project of getting the games on Steam, no source code from any legacy games showed up in the archives.
NewJazz 1y ago
Devs clean up their workstation sometimes. You can get fancy about deleting build artifacts or just remove the whole directory. Devs move to new machines sometimes and don't always transfer everything. Devs leave. Software still runs, and if you don't have the source then you'll only have the binary or other build artifact.
OutOfHere 1y ago
You clearly haven't worked much with code over many years. When laptops change, not all existing projects get checked out. In fact, in VSCode, one can use a project without cloning and checking it out at all.
- Dylan16807 1y ago
  Honestly I'm just really wondering what the odds are. In particular for code that made it onto git.
  - OutOfHere 1y ago
    Over the long term, the odds reach 100% that it won't be checked out. That's because people mostly only work on newer projects. As for mature older projects, even if they're running in production, cease to see many/any updates, and so they don't get cloned on to newer laptops. This doesn't mean that the older projects are now less important, because if they ever need to be re-deployed to production, only the source code will allow it.
  - CRConrad 1y ago
    > In particular for code that made it onto git. By “onto git”, do you mean “onto GitHub”? I really wish people would stop conflating the two. In fact, I can't tell whether this confusion is just another symptom of, or a (major?) part of the reason for, why we're in the mess we're in.
    - Dylan16807 1y ago
      I do not mean that, and I was not conflating anything. Just making and committing to a repo at all is a step that implies a certain level of caretaking.
burnt-resistor 1y ago
Popularity != importance. There is plenty of absolutely critical FOSS code that receives very little maintenance and attention, yet is mission critical to society functioning efficiently. And the same happens in organizations too, with say their bootloader for firmware of hardware products.

simondotau 1y ago

If the repo is on GitHub and two or more developers keep reasonably up-to-date checkouts on their local computers, the “3-2-1” principle of backups is satisfied. Additionally to that, if any of those developers have a backup strategy for their local computer, those also count as a backup of that source code.

wewewedxfgdf 1y ago
CTO explaining that to the CEO when your source code is completely gone: CTO: "I know our entire github repo is deleted and all our source code is gone and we never took backups, but I'm hoping the developers might have it all on their machines." CEO: "Hoping developers had it locally was your strategy for protecting all our source code?" CTO: "It's a sound approach and ticks all the boxes." CEO: "You're fired." Board Directors to CEO: "You're fired."
koonsolo 1y ago
The benefit of DVCS. Losing the source code from github when it's all on local computers is the least of problems.
Tohsig 1y ago
Technically true, but only if we consider dev checkouts as "backups". In the majority of cases they probably are, but that's not guaranteed. The local repo copy could be in a wildly different state than the primary origin, a shallow clone, etc... While the odds of that mattering are very low, they're not zero. I personally prefer to have a dedicated mirror as a failsafe.

OutOfHere 1y ago

I don't understand how the VC financiers got so rich while being so stupid as to hire such stupid people at the executive level, e.g. your CTO.

pintxo 1y ago
If only 10 out of 100 of your investments make it, does it matter if one of the 90 failed because of lacking backups? Their risk strategy is diversification of investments. Not making each investment itself bulletproof.
- OutOfHere 1y ago
  Yes, it is in effect a gamble. The issue is that this strategy doesn't really prove profitable for the majority of VCs. Less than 30% of VCs get to a unicorn or IPO deal. 46% of VCs don't profit at all. This is as per the recent post "(Only) half of senior VCs make at least one successful deal.". I am even ignoring the ones who drop out and don't contribute to the active statistics. The strategy is about as silly as having ten babies and expecting that one of them will make it. It is what you would expect out of the worst poverty-ridden parts of Africa. An alternative is to select and nurture your investments really well, so the rate of success is much higher. I'd like to see the script be flipped, whereby 90% of investments go on to becoming profit making, secondarily with their stable cash income being preferred to big exits.

gerdesj 1y ago

I would suggest your CTO needs some gentle reminders about risk management and how the cloud really works. If your boss is that daft, its probably a sign to bail out. Remember to do your own personal due diligence. Due dill is not something that you just do for someone else: do your own! Do your own personal risk assessment. If code was lost, who would be found accountable? You or them? EDIT: PS - I am a CTO ...

wewewedxfgdf 1y ago
But it's github ...... it can't be lost. Unless github closes the account, or a hacker gets access, or a rogue employee gets mad and deletes all, or some development accident results in repo deletion, or etc etc.
- weikju 1y ago
  Gotta add AI agents to that list
- SanjayMehta 1y ago
  Or if the one employee who created the account and was paying for it on his personal credit card and then got laid off. And no one else in the company knew what GitHub was.
  - robotnikman 1y ago
    Is there a story behind this oddly specific comment?
    - speeder 1y ago
      Not op but I saw this happen kinda maliciously... Company was founded by "visionary" CEO that is an sculptor and doesn't understand tech. Most crucial employee is young guy hired right out of college. Company now has 50+ developers, but the critical software is still made by that guy several years later. Critical guy still get recent graduate salary, notices he is actually the most important person in the company, and asks for a raise. Company denies his raise. He quits. Company hire a bunch of developers from cheap third world countries to replace him... Company learns: 1. All software guy made a bit before he asked for a raise was on his personal, not company github. 2. The software is in other programming languages, not the one the company uses normally. 3. Everything the guy wrote since he joined the company, is extremely difficult to maintain, guy is a genius and all his code is correct, clean and well made, but his thought patterns and how they end in the code are just too different, and all people that worked with him before also were geniuses and didn't care the code was "crazy". Note: the "other geniuses" mentioned above, also quit when other companies made them great offer and the stingy employer refused them a raise too.

klysm 1y ago

I mean it’s also on everybody’s laptop. Recovering from GitHub going away would be trivial for me

burnt-resistor 1y ago

LMAO. Must be one of those MBA CTOs. At least mirror the crown jewels to bitbucket, Tarsnap, or somewhat else that has 2 weeks - 3 months worth of independent copies made daily. If not MBA, the problem may also stem from the gradual atrophy and disrespect shown towards the sysadmin profession.

bufferoverflow 1y ago

[dead]