5 ms·

Kek. Auth: nobody knows IP address of our server anyway, don't bother with that. And for extra security we have secret port number. Monitoring? Well, we have

Kek. Auth: nobody knows IP address of our server anyway, don't bother with that. And for extra security we have secret port number. Monitoring? Well, we have our clients for that. They'll call us if something happens. Deploy? Copy .war file and restart tomcat. Rollback? What's rollback? Outdated packages? We call those stable packages. I wish I would work in a miracle world you describe.

stepbeek 5y ago
This sounds dreadful. I don’t think microservices are with talking about with these basics figured out.
paul_f 5y ago
I cannot tell if this is satire or not. Entering Poe's Law territory
- LaGrange 5y ago
  The answer is "both." This is very funny, very dumb, and how it's done, at the same time.
KronisLV 5y ago
It's easy to be snarky and dismissive about these things, but i've stood in the offices of a governmental org and have looked at people who are unable to receive their healthcare services queueing up because some similarly neglected system refused to work. One that also had a lot of the basics left out. Figuring out what was wrong with it was hard, because the logging was inconsistent, all over the place in where it logged and also insufficient. The deployments and environments were inconsistent, the sysadmins on clients's side manually changed stuff in the .war archives, all the way up to library versions, which was horrendous from a reproducibility perspective. The project was also severely out of date, not only package wise, but also because the new versions that had been developed actually weren't in prod. After ripping out the system's guts and replacing it with something that worked, i became way more careful when running into amateur hour projects like that and about managing the risks and eventual breakdown of them. I suggest that you do the same. Don't put yourself at risk, especially if some sort of a liability about the state of the system could land on you. What are you going to do when the system gets breached and a whole bunch of personal data gets leaked?
- madsbuch 5y ago
  Not so much snarky as a commentary on the diligence. Yes, if there is mission critical software that affects millions of people, do all that stuff. If your pdf -> stardust service is more a convenience then don't spend hundreds of thousands on infrastructure.
  - KronisLV 5y ago
    > ...then don't spend hundreds of thousands on infrastructure. I find it curious how we went from doing the basics of software development that would minimize risks and be helpful to almost any project out there to this. To clarify, i agree with the point that you'll need to prioritize different components based on what matters the most, but i don't think that you can't have a common standard set of tools and practices for all of them. Let me address all of the points with examples. > Auth: nobody knows IP address of our server anyway, don't bother with that. And for extra security we have secret port number. Port scanning means that none of your ports are secret. JWT is trivial to implement in most languages. Even basic auth is better than nothing with HTTPS, you don't always need mTLS or the more complicated solutions, but you need something. This should take a few days to a week to implement. Edit: probably a day or less if you have an easily reusable library for this. > Monitoring? Well, we have our clients for that. They'll call us if something happens. This is not viable if you have SLAs or just enjoy sleeping and not getting paged. There are free monitoring solutions out there, such as Zabbix, Nagios, Prometheus & Grafana and others. Preconfigured OS templates also mean that you just need the monitoring appliance and an agent on the node you want to monitor in most cases. This should take close to a week to implement. Edit: probably an hour or less if you already have a server up and just need to add a node. > Deploy? Copy .war file and restart tomcat. Rollback? What's rollback? This is an error prone way to do things, as the experience of Knight Capital showed: https://dougseven.com/2014/04/17/knightmare-a-devops-cautionary-tale https://dougseven.com/2014/04/17/knightmare-a-devops-caution.... In addition, manual configuration changes lead to configuration drift and after a few years you'll have little idea about who changed what and when. In contrast, setting up Ansible and versioning your config, as well as using containers for the actual software releases alongside fully automated CI cycles addresses all of those problems. In regards to rollbacks, if you have automated DB migrations, you might have to spend some time writing reverse migrations for all of the DDL changes. This should take between one to two weeks to implement. Edit: probably a day or less once per project with small fixes here and there. > Outdated packages? We call those stable packages. Log4j might be stable, but it also leads to RCEs. This is not a good argument, at least for as long as the software packages that we use are beyond our ability to control or comprehend. This should be a regular automated process, that alerts you about outdated and/or insecure packages, at least use npm audit or something, or proactive scanning like OpenVAS. This should take close to a week to implement. All of the mentioned software can easily run on a single node with 2 CPU cores and about 8 GB of RAM. I know this, because i did all of the above in a project of mine (mentioned technologies might have been changed, though). Of course, doing that over N projects will probably increase the total time, especially if they have been badly written. In my eyes that's a worthwhile investment, since when you finally have that across all of your apps, you can develop with better certainty that you won't have to stay late and deliver 2 or 3 hotfixes after your release, which goes hand in hand with test coverage and automated testing. That is hardly hundreds of thousands of dollars/euros/whatever on infrastructure. And if the personnel costs are like that, i'd like to work over there, then.
Ma8ee 5y ago
I really can’t tell how serious you are. I’m too aware that what you describe often is exactly how it works in practice. It’s just that very few admit it in public. :)

stepbeek 5y ago

This sounds dreadful. I don’t think microservices are with talking about with these basics figured out.

paul_f 5y ago

I cannot tell if this is satire or not. Entering Poe's Law territory

LaGrange 5y ago
The answer is "both." This is very funny, very dumb, and how it's done, at the same time.

KronisLV 5y ago

It's easy to be snarky and dismissive about these things, but i've stood in the offices of a governmental org and have looked at people who are unable to receive their healthcare services queueing up because some similarly neglected system refused to work. One that also had a lot of the basics left out. Figuring out what was wrong with it was hard, because the logging was inconsistent, all over the place in where it logged and also insufficient. The deployments and environments were inconsistent, the sysadmins on clients's side manually changed stuff in the .war archives, all the way up to library versions, which was horrendous from a reproducibility perspective. The project was also severely out of date, not only package wise, but also because the new versions that had been developed actually weren't in prod. After ripping out the system's guts and replacing it with something that worked, i became way more careful when running into amateur hour projects like that and about managing the risks and eventual breakdown of them. I suggest that you do the same. Don't put yourself at risk, especially if some sort of a liability about the state of the system could land on you. What are you going to do when the system gets breached and a whole bunch of personal data gets leaked?

madsbuch 5y ago
Not so much snarky as a commentary on the diligence. Yes, if there is mission critical software that affects millions of people, do all that stuff. If your pdf -> stardust service is more a convenience then don't spend hundreds of thousands on infrastructure.
- KronisLV 5y ago
  > ...then don't spend hundreds of thousands on infrastructure. I find it curious how we went from doing the basics of software development that would minimize risks and be helpful to almost any project out there to this. To clarify, i agree with the point that you'll need to prioritize different components based on what matters the most, but i don't think that you can't have a common standard set of tools and practices for all of them. Let me address all of the points with examples. > Auth: nobody knows IP address of our server anyway, don't bother with that. And for extra security we have secret port number. Port scanning means that none of your ports are secret. JWT is trivial to implement in most languages. Even basic auth is better than nothing with HTTPS, you don't always need mTLS or the more complicated solutions, but you need something. This should take a few days to a week to implement. Edit: probably a day or less if you have an easily reusable library for this. > Monitoring? Well, we have our clients for that. They'll call us if something happens. This is not viable if you have SLAs or just enjoy sleeping and not getting paged. There are free monitoring solutions out there, such as Zabbix, Nagios, Prometheus & Grafana and others. Preconfigured OS templates also mean that you just need the monitoring appliance and an agent on the node you want to monitor in most cases. This should take close to a week to implement. Edit: probably an hour or less if you already have a server up and just need to add a node. > Deploy? Copy .war file and restart tomcat. Rollback? What's rollback? This is an error prone way to do things, as the experience of Knight Capital showed: https://dougseven.com/2014/04/17/knightmare-a-devops-cautionary-tale https://dougseven.com/2014/04/17/knightmare-a-devops-caution.... In addition, manual configuration changes lead to configuration drift and after a few years you'll have little idea about who changed what and when. In contrast, setting up Ansible and versioning your config, as well as using containers for the actual software releases alongside fully automated CI cycles addresses all of those problems. In regards to rollbacks, if you have automated DB migrations, you might have to spend some time writing reverse migrations for all of the DDL changes. This should take between one to two weeks to implement. Edit: probably a day or less once per project with small fixes here and there. > Outdated packages? We call those stable packages. Log4j might be stable, but it also leads to RCEs. This is not a good argument, at least for as long as the software packages that we use are beyond our ability to control or comprehend. This should be a regular automated process, that alerts you about outdated and/or insecure packages, at least use npm audit or something, or proactive scanning like OpenVAS. This should take close to a week to implement. All of the mentioned software can easily run on a single node with 2 CPU cores and about 8 GB of RAM. I know this, because i did all of the above in a project of mine (mentioned technologies might have been changed, though). Of course, doing that over N projects will probably increase the total time, especially if they have been badly written. In my eyes that's a worthwhile investment, since when you finally have that across all of your apps, you can develop with better certainty that you won't have to stay late and deliver 2 or 3 hotfixes after your release, which goes hand in hand with test coverage and automated testing. That is hardly hundreds of thousands of dollars/euros/whatever on infrastructure. And if the personnel costs are like that, i'd like to work over there, then.

Ma8ee 5y ago

I really can’t tell how serious you are. I’m too aware that what you describe often is exactly how it works in practice. It’s just that very few admit it in public. :)