6 ms·

> I'm amused by the tone. It's like the author doesn't realize that 99% of software development and deployment is done like this, or much much worse. Welcome to

> I'm amused by the tone. It's like the author doesn't realize that 99% of software development and deployment is done like this, or much much worse. Welcome to the real world. Agree with this, a lot of developers are in a filter bubble where they stick to communities that advocate modern practices like automated testing, continuous integration, containers, gitflow, staging environments etc. As a contractor, I get to see the internals of lots of different companies - forget practices even as basic as doing code reviews, I've seen companies not using source control with no staging environments and no local development environments where all changes are being made directly on the production server via SFTP on a basic VPS. A lot of the time there's no internal experts there that are even aware there's better ways to do things instead of it being the case they're lacking resources to make improvements.

foldor 8y ago
I wish I hadn't experienced this exact scenario as well. I actually worked at a publishing company where I discovered the previous team building their major education platform didn't know there was such thing as source control. Their method for sharing multiple team members changes was to email the full project to one person every Friday who would manually open a merging tool and merge all changes themselves. They would then send out the updated code base to all people again. Because of this method, they were afraid to delete any stale code, and just prefixed old versions of functions with XX. As you can imagine, inheriting that code base was a nightmare to deal with.
- WrtCdEvrydy 8y ago
  Someone created a jenkins pipeline that would deploy code in a zip file into prod.
  - mattmanser 8y ago
    Am I missing something? Isn't this exactly how deployment servers actually work? I'm not into Java development, but this sounds fine on the face of it, without you giving the context of how this pipeline is triggered.
    - WrtCdEvrydy 8y ago
      Developers had a copy of code in a google drive... they'd modify it, zip it and overwrite the one in google drive, copy it to the network folder which would deploy it and delete it from the network folder... in 2016.
    - Bartweiss 8y ago
      In theory it sounds alright. It's not great, because Jenkins is usually layered with some existing deploy framework that makes "deploy a zip file" pretty suspect. A healthier setup would look more like "build a Tomcat war file Maven, upload and deploy that with Jenkins". But in context, it sounds like the horror is that people were making and transferring a zip from local code rather than building from the tip of source control.
  - debaserab2 8y ago
    That’s a pretty standard CI process - zip files are often your deploy artifact (using, for example, git archive)
    - WrtCdEvrydy 8y ago
      Yeah, except the zip file was... the input. You pulled the code from google drive, modified it, pushed it to PROD, checked it and moved it back to google drive... and asked the other developers to update.
- yibg 8y ago
  I remember having this debate many years ago. Would it be better to introduce a team with no source control knowledge to something like svn first or straight to git? Svn is easier to understand and use, but then you’d have to break some existing habits to get to git. But going straight to git might be a big step and cause reversion back to whatever system was already there.
  - srean 8y ago
    Here's a vote for going straight to Mercurial (a lot easier to grok) and link to Joel Spolky's excellent tutorial https://web.archive.org/web/20180903164646/http://hginit.com/ https://web.archive.org/web/20180903164646/http://hginit.com...
  - ehnto 8y ago
    Straight to git but have a simple and clear branching strategy. As in a full written procedure for the three main events of git. "Get code, commit code, push code". Then disallow direct commits to master to make people work in feature branches and make merge requests through the platform (github /lab/ bitbucket). I find merging and branching locally is where people normally trip up. Git GUI tools always make git seem way more complicated than it is so depending on the teams platform I would recommend cli from the start.
    - jliptzin 8y ago
      I would not start people new to git on cli. That's how you get someone's entire Camera Uploads directory committed to master (I've seen it before). I recommend Git Tower. I use it for most tasks actually even though I am comfortable in the CLI too. It tends to stop me from doing stupid things before I do them.
      - seanwilson 8y ago
        You can probably assume a lot of people who aren't using version control aren't using the terminal either. GUIs are generally much better at CLIs at giving you an overview of what's happening and what actions are available too.
carlmr 8y ago
>I've seen companies not using source control Well I've seen companies that have their own idea of source control. Which is lots of copies on the network drive, and an Excel registry with what is in which file. It is source control. Just bad source control.
- blowski 8y ago
  I worked on a system with “octopus locking”. We a had toy octopus, and you could only change the code (obviously in production) when you had the octopus.
- yellowapple 8y ago
  My day job involves multiple third-party systems that enforce their own proprietary version control systems for any custom scripting/programming within those systems. Unsurprisingly, these proprietary version control systems are complete garbage 90% of the time, especially if you're trying to collaborate with someone else.
Reedx 8y ago
> I've seen companies not using source control > ...all changes are being made directly on the production servers via SFTP I know this used to be common, but recently? Curious how often this is still the case.
- cr0sh 8y ago
  Hey, at least they're using SFTP - you know, for the security! /seriously, though - I...hope this isn't being done any longer - but I bet it is. Sigh...
- Bartweiss 8y ago
  At my first (very small, research-group) employer, I was the one to introduce source control. Another company I saw ~4 years ago had Git, but no one knew how to use it adequately so "lost code" was still a regular event. I haven't seen it since, though; lots of people frightened by and badly misusing Git, but they mostly manage to keep code histories intact. Having no testing/staging environments remains pretty common, along with its cousin "production work happens on staging". Partnering with not-primarily-software companies and asking about staging infrastructure, you har that regularly. And yeah, SFTP/SCP/SSH is a standard push-and-deploy approach in places where that happens.
- frosted-flakes 8y ago
  I have seen it recently. I did my best to change the practice before I left the company, but was mostly unsuccessful. Given that they were still running some of the spaghetti-code PHP scripts that were written in 1999 and still used PHP4 in new development they were stuck in the stone ages. To give a little perspective, support for PHP4 ended in 2008, so they had almost a decade to update, but didn't.
  - HenryBemis 8y ago
    "If it ain't broken, don't fix it". And then one day the server goes boom, the backup was incomplete, and everyone is trying to find the usb flash disk with Spinrite in it. Meanwhile the CEO who was rejecting the €¥$£ in yh budget since 2000 is angry at everyone! Oh the times I have seen this!!!
tetha 8y ago
This comment is resonating very well with that I've been experiencing at my current job, and with our new head company. It's always nice if you can take a new application, throw it into a clean container build chain with an orchestration - or some AWS stack, and there's a cheap, low-maintenance, HA production system. However, there's also the skill set of taking such a ... let's call it well-aged development team and approach and modernizing and/or professionalizing it. And yes, sometimes this means to build some entirely lovecraftian deployment mechanism on mutable VMs because of how the application behaves. But hey, automated processes are better than manual processes, which beat undocumented processes. Baby steps.
dba7dba 8y ago
> I've seen companies not using source control with no staging environments and no local development environments where all changes are being made directly on the production server via SFTP on a basic VPS. Omg. And here I am feeling ashamed to tell others about my small small personal website with separate dev, qa, production environments on same server (via VirtualHost), code checked into github, deployed via Jenkins self hosted on another VPS, which was initially spun up with Ansible and shell scripts. All done by me for self training purpose. All because I thought businesses would have something more sophisticated with bells and whistles. And then I hear there are businesses that make changes directly on live production servers... But I'm not surprised by such stories as I have seen some bad workflow in real businesses that deal with tens of millions of dollars a year. Years ago, I worked in the NOC of a company that's top in the small niche. They have dozens of employees, and been around for years. Part of the job responsibility was rolling out patches to production servers. The kicker was the production servers were all Windows servers, running various versions, covering practically all Windows versions ever released by Microsoft. You can see where this is headed. Rolling out a patch was all done manually, 1 Windows server at a time. Everything was manually done. The instruction for deploying a change was easily multiple lines long, each with different style of writing/instruction. Often in plain text file format. We would print them out so we could check them off as we went down the list. The CTO is still there, but everyone IT person under him has left or been let go. Working in the IT there is a struggle because lack of automation and old old stuff, but the CTO just blames bad employees and keeps churning them in/out. When the real issue is the decade or 2 worth of old legacy stuff that need to be cleaned up and/or thrown out, which can only be done by the direction of the CTO. But he knows he won't get that kind of budget from higher-ups so he will just keep hiring/firing employees and/or bring in some H1B workers who's basically trapped once they join. And of the few H1B workers I met there, they were truly completely non-technical. One did not want to learn how to use keyboard shortcuts to do common tasks... Good guy though.

foldor 8y ago

I wish I hadn't experienced this exact scenario as well. I actually worked at a publishing company where I discovered the previous team building their major education platform didn't know there was such thing as source control. Their method for sharing multiple team members changes was to email the full project to one person every Friday who would manually open a merging tool and merge all changes themselves. They would then send out the updated code base to all people again. Because of this method, they were afraid to delete any stale code, and just prefixed old versions of functions with XX. As you can imagine, inheriting that code base was a nightmare to deal with.

WrtCdEvrydy 8y ago
Someone created a jenkins pipeline that would deploy code in a zip file into prod.
- mattmanser 8y ago
  Am I missing something? Isn't this exactly how deployment servers actually work? I'm not into Java development, but this sounds fine on the face of it, without you giving the context of how this pipeline is triggered.
  - WrtCdEvrydy 8y ago
    Developers had a copy of code in a google drive... they'd modify it, zip it and overwrite the one in google drive, copy it to the network folder which would deploy it and delete it from the network folder... in 2016.
  - Bartweiss 8y ago
    In theory it sounds alright. It's not great, because Jenkins is usually layered with some existing deploy framework that makes "deploy a zip file" pretty suspect. A healthier setup would look more like "build a Tomcat war file Maven, upload and deploy that with Jenkins". But in context, it sounds like the horror is that people were making and transferring a zip from local code rather than building from the tip of source control.
- debaserab2 8y ago
  That’s a pretty standard CI process - zip files are often your deploy artifact (using, for example, git archive)
  - WrtCdEvrydy 8y ago
    Yeah, except the zip file was... the input. You pulled the code from google drive, modified it, pushed it to PROD, checked it and moved it back to google drive... and asked the other developers to update.
yibg 8y ago
I remember having this debate many years ago. Would it be better to introduce a team with no source control knowledge to something like svn first or straight to git? Svn is easier to understand and use, but then you’d have to break some existing habits to get to git. But going straight to git might be a big step and cause reversion back to whatever system was already there.
- srean 8y ago
  Here's a vote for going straight to Mercurial (a lot easier to grok) and link to Joel Spolky's excellent tutorial https://web.archive.org/web/20180903164646/http://hginit.com/ https://web.archive.org/web/20180903164646/http://hginit.com...
- ehnto 8y ago
  Straight to git but have a simple and clear branching strategy. As in a full written procedure for the three main events of git. "Get code, commit code, push code". Then disallow direct commits to master to make people work in feature branches and make merge requests through the platform (github /lab/ bitbucket). I find merging and branching locally is where people normally trip up. Git GUI tools always make git seem way more complicated than it is so depending on the teams platform I would recommend cli from the start.
  - jliptzin 8y ago
    I would not start people new to git on cli. That's how you get someone's entire Camera Uploads directory committed to master (I've seen it before). I recommend Git Tower. I use it for most tasks actually even though I am comfortable in the CLI too. It tends to stop me from doing stupid things before I do them.
    - seanwilson 8y ago
      You can probably assume a lot of people who aren't using version control aren't using the terminal either. GUIs are generally much better at CLIs at giving you an overview of what's happening and what actions are available too.

carlmr 8y ago

>I've seen companies not using source control Well I've seen companies that have their own idea of source control. Which is lots of copies on the network drive, and an Excel registry with what is in which file. It is source control. Just bad source control.

blowski 8y ago
I worked on a system with “octopus locking”. We a had toy octopus, and you could only change the code (obviously in production) when you had the octopus.
yellowapple 8y ago
My day job involves multiple third-party systems that enforce their own proprietary version control systems for any custom scripting/programming within those systems. Unsurprisingly, these proprietary version control systems are complete garbage 90% of the time, especially if you're trying to collaborate with someone else.

Reedx 8y ago

> I've seen companies not using source control > ...all changes are being made directly on the production servers via SFTP I know this used to be common, but recently? Curious how often this is still the case.

cr0sh 8y ago
Hey, at least they're using SFTP - you know, for the security! /seriously, though - I...hope this isn't being done any longer - but I bet it is. Sigh...
Bartweiss 8y ago
At my first (very small, research-group) employer, I was the one to introduce source control. Another company I saw ~4 years ago had Git, but no one knew how to use it adequately so "lost code" was still a regular event. I haven't seen it since, though; lots of people frightened by and badly misusing Git, but they mostly manage to keep code histories intact. Having no testing/staging environments remains pretty common, along with its cousin "production work happens on staging". Partnering with not-primarily-software companies and asking about staging infrastructure, you har that regularly. And yeah, SFTP/SCP/SSH is a standard push-and-deploy approach in places where that happens.
frosted-flakes 8y ago
I have seen it recently. I did my best to change the practice before I left the company, but was mostly unsuccessful. Given that they were still running some of the spaghetti-code PHP scripts that were written in 1999 and still used PHP4 in new development they were stuck in the stone ages. To give a little perspective, support for PHP4 ended in 2008, so they had almost a decade to update, but didn't.
- HenryBemis 8y ago
  "If it ain't broken, don't fix it". And then one day the server goes boom, the backup was incomplete, and everyone is trying to find the usb flash disk with Spinrite in it. Meanwhile the CEO who was rejecting the €¥$£ in yh budget since 2000 is angry at everyone! Oh the times I have seen this!!!

tetha 8y ago

This comment is resonating very well with that I've been experiencing at my current job, and with our new head company. It's always nice if you can take a new application, throw it into a clean container build chain with an orchestration - or some AWS stack, and there's a cheap, low-maintenance, HA production system. However, there's also the skill set of taking such a ... let's call it well-aged development team and approach and modernizing and/or professionalizing it. And yes, sometimes this means to build some entirely lovecraftian deployment mechanism on mutable VMs because of how the application behaves. But hey, automated processes are better than manual processes, which beat undocumented processes. Baby steps.

dba7dba 8y ago

> I've seen companies not using source control with no staging environments and no local development environments where all changes are being made directly on the production server via SFTP on a basic VPS. Omg. And here I am feeling ashamed to tell others about my small small personal website with separate dev, qa, production environments on same server (via VirtualHost), code checked into github, deployed via Jenkins self hosted on another VPS, which was initially spun up with Ansible and shell scripts. All done by me for self training purpose. All because I thought businesses would have something more sophisticated with bells and whistles. And then I hear there are businesses that make changes directly on live production servers... But I'm not surprised by such stories as I have seen some bad workflow in real businesses that deal with tens of millions of dollars a year. Years ago, I worked in the NOC of a company that's top in the small niche. They have dozens of employees, and been around for years. Part of the job responsibility was rolling out patches to production servers. The kicker was the production servers were all Windows servers, running various versions, covering practically all Windows versions ever released by Microsoft. You can see where this is headed. Rolling out a patch was all done manually, 1 Windows server at a time. Everything was manually done. The instruction for deploying a change was easily multiple lines long, each with different style of writing/instruction. Often in plain text file format. We would print them out so we could check them off as we went down the list. The CTO is still there, but everyone IT person under him has left or been let go. Working in the IT there is a struggle because lack of automation and old old stuff, but the CTO just blames bad employees and keeps churning them in/out. When the real issue is the decade or 2 worth of old legacy stuff that need to be cleaned up and/or thrown out, which can only be done by the direction of the CTO. But he knows he won't get that kind of budget from higher-ups so he will just keep hiring/firing employees and/or bring in some H1B workers who's basically trapped once they join. And of the few H1B workers I met there, they were truly completely non-technical. One did not want to learn how to use keyboard shortcuts to do common tasks... Good guy though.