9 ms·

I am a lead on a small startup team and one of the biggest pain points is dealing with infrastructure. We have no dedicated devops person and much of that work

by vvpan 3y ago

I am a lead on a small startup team and one of the biggest pain points is dealing with infrastructure. We have no dedicated devops person and much of that work falls on me and other people who would be better writing code. I think the cloud paradigm is experiencing a shift. Few of us want to deal with cloud infrastructure (whether clicking around or via Terraform or equivalent) to execute a function every X minutes - its unnecessary developer hours. The functionality is so common that it should just work. Perhaps stemming from that there seem to be two recent trends: 1. Services like this cron with Dyno Deploy or Vercel where the cloud things are abstracted away for you. 2. I have no data to back it up but it seems like JVM is experience a bit of a comeback with a few companies adopting Kotlin for backend and, also subjectively, talk about Elixir has increased here on Hacker News (we've all seen that table that shows how the Erlang VM is cron/background jobs/logging service/KV store all rolled into one). And if those are not the trends I feel like they should be. Both of those are very appealing to me as a team lead. Honestly, I have never been a fan of managing services in AWS/GCP/etc - the setup overhead rarely seemed to outweigh the pros, at least until some heavy amount of data start moving around. I might be stating the obvious - less work is definitely better, but pretty much every company I have consulted for in the last few years had a (crappy) bespoke cloud setup that was a massive time sink.

mattgreenrocks 3y ago
For sure. As someone who has stayed away from this part of the biz, infrastructure always looked like makework to me. Necessary, mind you, but still makework that was indicative of poor/insufficient computing primitives for services. > it seems like JVM is experience a bit of a comeback with a few companies adopting Kotlin for backend JVM 21 adding virtual threads, and Spring Boot 3.2 using them with one line of config is huge. We can now write simple code that looks blocking and let the runtime handle it instead of writing async function/await everywhere. Personally I'm loving Spring Boot 3.2 with Kotlin, especially for the fact that I can bundle scheduled jobs, API, and frontend all in one place for my indie projects. Plus the JVM world of devs seems to have a somewhat decent appreciation of how to make web services that aren't rife with unnecessary coupling. The bad? There is a legendary amount of cargo-culting blog posts and questionable advice around Spring Boot, often for older versions.
mike_d 3y ago
It is really disappointing to me when I hear takes like this. As an industry we are splitting the roles of programmers and engineers so that programmers can just throw spaghetti at the wall and it is the engineers problem to happily run it. We need more well rounded people that also fundamentally understand how their code is executing on the backend so performance and cost can be optimized.
- marcosdumay 3y ago
  We need decently designed environments where running stuff is trivial. It's not practical to expect the same person to be able to deal with both all the bullshit that comes from making the spaghetti and all the bullshit that comes from making it adhere to the wall.
  - everforward 3y ago
    It is practical, a lot of us do it. Besides, the amount of bullshit it takes to make the spaghetti stick to the wall is dictated by how it was made. Make nice spaghetti and it sticks to the wall effortlessly. Make bad spaghetti and it takes 4 people and a dozen rolls of duct tape to get it up there.
    - marcosdumay 3y ago
      > the amount of bullshit it takes to make the spaghetti stick to the wall is dictated by how it was made I really don't know where the bullshit coming from. Often it looks like every single person is fighting for their lives against it, and yet it keeps growing. But it's mostly dictated by overarching decisions that constrain both how it runs and how it's made (AKA, outside somewhere). Of course, there exist people on both sides that create problems all by themselves, but those are always easy to solve.
      - everforward 3y ago
        > But it's mostly dictated by overarching decisions that constrain both how it runs and how it's made (AKA, outside somewhere). While annoying, there's often a good reason behind those if you can drill down to the engineers that asked for that rule. Just as an example, it's easy to provide a single HA solution at the infrastructure tier... provided the infra people can make a few assumptions about your app (stateless, can handle multiple copies running at once, etc). It's virtually impossible to build HA that makes no assumptions, and it's incredibly burdensome to support 85 different HA implementations with varying sets of assumptions. It makes business sense to have infra do the HA instead of each developer team, and that means restrictions on how apps are built and run. What you're experiencing is the platform-ization of a bunch of aspects of the app. Making nice spaghetti usually consists of finding the rules, finding the people who made them, and then grabbing them and asking "what does it take for this app to be in prod 15 minutes after we push the last commit?". It sounds like it will be a lot of people because there's a lot of rules, but there's usually like a half-dozen teams that make all the rules. It will likely force you to build differently than you would have, but it will also mean that you're more likely to have 0 friction and you're more likely to get help with the friction you do encounter.
dyeje 3y ago
There’s a new crop of PaaS products that aim to provide a layer of sanity on top of cloud providers (e.g. Aptible). Might be a good fit for you.
klabb3 3y ago
> 1. Services like this cron with Dyno Deploy or Vercel where the cloud things are abstracted away for you. I agree with the sentiment, but to me it’s a shame this even qualifies as abstraction in the first place. Periodic execution of a function is extremely basic and already exists in almost every programming environment. If this needs to be a service (instead of a library call), there is something deeply wrong elsewhere, imo. We all laughed about leftpad, but at least that was native code and not “as-a-service”, dependent on networking and proprietary code on someone else’s machine. I have nothing against Deno in particular, and I’m also not saying this is easy from an implementation perspective (due to complexity elsewhere). I’m just surprised that trivial features – from an end-user perspective – justify such a ruckus.
- notpachet 3y ago
  I wonder why more people don't just set up a cheap VM and run this sort of thing via crontab.
  - Swizec 3y ago
    Does crontab on a self-managed VM guarantee at least once delivery? For many, if not most, use-cases that guarantee is critical.
    - notpachet 3y ago
      No. But I don't think there's any environment out there that is going to 100% guarantee that your cron succeeds. Even if it is reenqueued on failure, it could just keep hitting the same problem and crashlooping. You still need some kind of failure reporting, and a human to jump in to fix whatever went wrong.
  - fesc 3y ago
    The same reason why most people stopped manually editing some random files via FTP to do deployments: to get a proper reproducible, automated and monitored production environment.
conradfr 3y ago
For cron jobs in Elixir you can use: - [0] Quantum, quite simple and supports cron syntax. - [1] Oban, which is a job runner, requires a DB, but also supports periodic jobs with a cron syntax. (probably others but that's the ones I've used) [0] https://github.com/fanduel/quantum-elixir https://github.com/fanduel/quantum-elixir [1] https://github.com/sorentwo/oban https://github.com/sorentwo/oban
- lucasyvas 3y ago
  There's a good argument to make that you should never use your application code for running tasks better handled by infrastructure. Eventually you will want an operations team of some kind and they will absolutely hate you for doing this. It doesn't scale, organizationally speaking. Logic? write it in app code sure. How it's triggered or runs? Probably don't go near that. Use your job scheduler (CRON) or cloud infra to do it. Your operations team can port it, scale it, and cost optimize it without breaking a sweat. If more sysadmin / DevOps are hired to help they are going to roast your development team if they can't actually help you. Elixir may be able to glide by some of this given how flexible its process model is - but it's not good general advice.
  - auggierose 3y ago
    Doesn't make any kind of sense to me. It might be best practices, but if you can automate things, you should be doing it. And the best automation happens behind a great abstraction.
seqizz 3y ago
> one of the biggest pain points is dealing with infrastructure > We have no dedicated devops person I understand that not every startup can afford an extra person for infra-heavy tasks, but this should not be surprising. Any tool which looks like an abstraction to you might get complex by time, and steal your time. The cloud was supposed to fix some of these, look where we are now.
RadiozRadioz 3y ago
Or you could just use an actual host running actual cron. No clicking, no terraform, just crontab -e. The last thing small short-staffed startups with no infra people should be doing is messing around setting up cloud infra to run something on a schedule.
- ketzo 3y ago
  Okay, but who's taking care of the actual host? I know this is HN and we're all server admin wizards, but there is some effort and maintenance required. This is theoretically even easier than that. Yay for a spectrum of options!
  - jokethrowaway 3y ago
    That's an overblown problem, especially for a small business which is not going to have the infra problem of someone with scale problems. We've just been sold lies that pushed us to buy expensive clouds - and are not that much simpler than good old tools
    - ketzo 3y ago
      I agree that "it's way too much effort to ever consider running your code on your own servers / EC2 instances" is overblown, and has very much favored cloud providers. But for the specific problem of "I wanna execute some code on a regular schedule, but I want it to be production-ready", Deno Cron does seem much quicker and lighter than provisioning a host, setting up health checks, setting up remote access controls for new devs, etc. There are obviously many people who would find it very easy to do all the "server-work" I'm describing. I am not one of them! This service looks awesome to me.
- vvpan 3y ago
  It's not an option if you have more than one machine and you want the machines to be uniform. If they are not uniform means it's back to square one - bespoke setups. And then I want reproducibility across environments and environments that are easy to set up (for testing for example). Basically I think it's not an option.
MuffinFlavored 3y ago
> Honestly, I have never been a fan of managing services in AWS/GCP/etc - the setup overhead rarely seemed to outweigh the pros I just `ssh` into prod and `git pull` and either `docker compose up` or `kubectl apply` or `terraform apply` or `helm update` or `argocd sync`
- ketzo 3y ago
  Okay, this is either really good satire, or you are wildly overestimating the average startup engineer's knowledge of each of those tools and their configurations
bauble 3y ago
I don't understand the reluctance to hire a system administrator. If you have two or more technical folks, at least one of them should be a sysadmin (or "devops" if you prefer). Pretending that it's an unskilled role that anyone should pick up reflects a poor understanding of the industry.
pech0rin 3y ago
The "crappy bespoke cloud setup" is usually the result of people whose jobs are usually just regular app devs, but had been thrown into the infrastructure side head first. I think the key is to actually have someone who knows what they are talking about. Unlike other people I don't think you need to hire someone directly. I do devops freelancing so you could hire someone like me. Or look online for various people that do it. I mean if its a toy app without many customers then who cares, but once you start scaling infrastructure gets to me more and more important.
no_wizard 3y ago
I think this is where modern PHP (8.0+) has its strengths. Its well known how to deploy. Its easy to deploy. The footguns are vigorously and thoroughly documented. The frameworks (Symfony, Laravel) are good. They all have a solution for X (be it queues, async work, cron jobs etc) and they have turn key deploy solutions favored by their respective communities. Its honestly one of the easiest languages to deploy with nowadays, in my estimation. Rivaled only by C# / F# or platforms like Deno Deploy or Cloudflare workers. Sometimes the most boring / mature thing is the best thing.
jcusch 3y ago
There is a collection of new tools trying to address this exact problem such as Nitric, Winglang and Ampt for example. (disclaimer, I work on Nitric)
rstuart4133 3y ago
> We have no dedicated devops person and much of that work falls on me and other people who would be better writing code. I think the cloud paradigm is experiencing a shift. Part of that shift is that prior to the cloud, the development environment and production environments were very similar. By that I mean if a programmer could set up his debian / redhat laptop from the command line, then it wasn't a big jump to set up a server running debian / redhat in a production environment. But no one in their right mind runs k8s on their laptop just to support VSCode and a compiler. Worse they could not run AWS/GCP because they are proprietary, insanely large, complex and forever changing. I'm pretty sure becoming an expert engineer in those environments is a full time job. The end result is a new class of engineers have arisen that look after the production side, and the developers just throw their code over the wall once it's tested. I'm sure the cloud providers love this, as they have created a cohort of engineers that have invested years in learning their product, now have their wages dependent on that product being used within their organisation, and a total monopoly over that product because it can never be reproduced by anyone else. But for the industry as a whole, it looks to be a backward step. Once the problem expands beyond what a single human mind can cope with, you need teams. The communication overhead of a team means they don't have a hope of being as productive as a single person. The share amount of complexity introduced by these proprietary cloud frameworks looks to be unnecessary. Most of it smells like technical debit created by organic growth together with an insistence on backward compatibility (don't want to give those locked in customers an excuse to move). And desktop OS's are (Debian / Windows / ...) are insecure by design, at least when compared to their phone brethren. And, their phone brethren look somewhat like cloud designs now, Phones have isolated apps with private data areas. The apps communicate via channels provided by the OS, and the availability of those channels is controlled by permissions assigned by the user. In the cloud we isolated things for performance as they could run on distance machines, whereas in a personal device we did it for security. But the end result looks similar. So maybe one day we will end up where we started, with a developer environment looking like the cloud we deploy too. I fear I'm too old to experience if / when it does happen, but it does seem like something we should aspire to.
brundolf 3y ago
+1, as someone whose company has a crappy bespoke cloud setup that's a massive time sink
alberth 3y ago
PaaS Sounds like you’d like to use a PaaS provider. Heroku/Render/Fly/etc.

by vvpan 3y ago

mattgreenrocks 3y ago

For sure. As someone who has stayed away from this part of the biz, infrastructure always looked like makework to me. Necessary, mind you, but still makework that was indicative of poor/insufficient computing primitives for services. > it seems like JVM is experience a bit of a comeback with a few companies adopting Kotlin for backend JVM 21 adding virtual threads, and Spring Boot 3.2 using them with one line of config is huge. We can now write simple code that looks blocking and let the runtime handle it instead of writing async function/await everywhere. Personally I'm loving Spring Boot 3.2 with Kotlin, especially for the fact that I can bundle scheduled jobs, API, and frontend all in one place for my indie projects. Plus the JVM world of devs seems to have a somewhat decent appreciation of how to make web services that aren't rife with unnecessary coupling. The bad? There is a legendary amount of cargo-culting blog posts and questionable advice around Spring Boot, often for older versions.

mike_d 3y ago

It is really disappointing to me when I hear takes like this. As an industry we are splitting the roles of programmers and engineers so that programmers can just throw spaghetti at the wall and it is the engineers problem to happily run it. We need more well rounded people that also fundamentally understand how their code is executing on the backend so performance and cost can be optimized.

marcosdumay 3y ago
We need decently designed environments where running stuff is trivial. It's not practical to expect the same person to be able to deal with both all the bullshit that comes from making the spaghetti and all the bullshit that comes from making it adhere to the wall.
- everforward 3y ago
  It is practical, a lot of us do it. Besides, the amount of bullshit it takes to make the spaghetti stick to the wall is dictated by how it was made. Make nice spaghetti and it sticks to the wall effortlessly. Make bad spaghetti and it takes 4 people and a dozen rolls of duct tape to get it up there.
  - marcosdumay 3y ago
    > the amount of bullshit it takes to make the spaghetti stick to the wall is dictated by how it was made I really don't know where the bullshit coming from. Often it looks like every single person is fighting for their lives against it, and yet it keeps growing. But it's mostly dictated by overarching decisions that constrain both how it runs and how it's made (AKA, outside somewhere). Of course, there exist people on both sides that create problems all by themselves, but those are always easy to solve.
    - everforward 3y ago
      > But it's mostly dictated by overarching decisions that constrain both how it runs and how it's made (AKA, outside somewhere). While annoying, there's often a good reason behind those if you can drill down to the engineers that asked for that rule. Just as an example, it's easy to provide a single HA solution at the infrastructure tier... provided the infra people can make a few assumptions about your app (stateless, can handle multiple copies running at once, etc). It's virtually impossible to build HA that makes no assumptions, and it's incredibly burdensome to support 85 different HA implementations with varying sets of assumptions. It makes business sense to have infra do the HA instead of each developer team, and that means restrictions on how apps are built and run. What you're experiencing is the platform-ization of a bunch of aspects of the app. Making nice spaghetti usually consists of finding the rules, finding the people who made them, and then grabbing them and asking "what does it take for this app to be in prod 15 minutes after we push the last commit?". It sounds like it will be a lot of people because there's a lot of rules, but there's usually like a half-dozen teams that make all the rules. It will likely force you to build differently than you would have, but it will also mean that you're more likely to have 0 friction and you're more likely to get help with the friction you do encounter.

dyeje 3y ago

There’s a new crop of PaaS products that aim to provide a layer of sanity on top of cloud providers (e.g. Aptible). Might be a good fit for you.

klabb3 3y ago

> 1. Services like this cron with Dyno Deploy or Vercel where the cloud things are abstracted away for you. I agree with the sentiment, but to me it’s a shame this even qualifies as abstraction in the first place. Periodic execution of a function is extremely basic and already exists in almost every programming environment. If this needs to be a service (instead of a library call), there is something deeply wrong elsewhere, imo. We all laughed about leftpad, but at least that was native code and not “as-a-service”, dependent on networking and proprietary code on someone else’s machine. I have nothing against Deno in particular, and I’m also not saying this is easy from an implementation perspective (due to complexity elsewhere). I’m just surprised that trivial features – from an end-user perspective – justify such a ruckus.

notpachet 3y ago
I wonder why more people don't just set up a cheap VM and run this sort of thing via crontab.
- Swizec 3y ago
  Does crontab on a self-managed VM guarantee at least once delivery? For many, if not most, use-cases that guarantee is critical.
  - notpachet 3y ago
    No. But I don't think there's any environment out there that is going to 100% guarantee that your cron succeeds. Even if it is reenqueued on failure, it could just keep hitting the same problem and crashlooping. You still need some kind of failure reporting, and a human to jump in to fix whatever went wrong.
- fesc 3y ago
  The same reason why most people stopped manually editing some random files via FTP to do deployments: to get a proper reproducible, automated and monitored production environment.

conradfr 3y ago

For cron jobs in Elixir you can use: - [0] Quantum, quite simple and supports cron syntax. - [1] Oban, which is a job runner, requires a DB, but also supports periodic jobs with a cron syntax. (probably others but that's the ones I've used) [0] https://github.com/fanduel/quantum-elixir https://github.com/fanduel/quantum-elixir [1] https://github.com/sorentwo/oban https://github.com/sorentwo/oban

lucasyvas 3y ago
There's a good argument to make that you should never use your application code for running tasks better handled by infrastructure. Eventually you will want an operations team of some kind and they will absolutely hate you for doing this. It doesn't scale, organizationally speaking. Logic? write it in app code sure. How it's triggered or runs? Probably don't go near that. Use your job scheduler (CRON) or cloud infra to do it. Your operations team can port it, scale it, and cost optimize it without breaking a sweat. If more sysadmin / DevOps are hired to help they are going to roast your development team if they can't actually help you. Elixir may be able to glide by some of this given how flexible its process model is - but it's not good general advice.
- auggierose 3y ago
  Doesn't make any kind of sense to me. It might be best practices, but if you can automate things, you should be doing it. And the best automation happens behind a great abstraction.

seqizz 3y ago

> one of the biggest pain points is dealing with infrastructure > We have no dedicated devops person I understand that not every startup can afford an extra person for infra-heavy tasks, but this should not be surprising. Any tool which looks like an abstraction to you might get complex by time, and steal your time. The cloud was supposed to fix some of these, look where we are now.

RadiozRadioz 3y ago

Or you could just use an actual host running actual cron. No clicking, no terraform, just crontab -e. The last thing small short-staffed startups with no infra people should be doing is messing around setting up cloud infra to run something on a schedule.

ketzo 3y ago
Okay, but who's taking care of the actual host? I know this is HN and we're all server admin wizards, but there is some effort and maintenance required. This is theoretically even easier than that. Yay for a spectrum of options!
- jokethrowaway 3y ago
  That's an overblown problem, especially for a small business which is not going to have the infra problem of someone with scale problems. We've just been sold lies that pushed us to buy expensive clouds - and are not that much simpler than good old tools
  - ketzo 3y ago
    I agree that "it's way too much effort to ever consider running your code on your own servers / EC2 instances" is overblown, and has very much favored cloud providers. But for the specific problem of "I wanna execute some code on a regular schedule, but I want it to be production-ready", Deno Cron does seem much quicker and lighter than provisioning a host, setting up health checks, setting up remote access controls for new devs, etc. There are obviously many people who would find it very easy to do all the "server-work" I'm describing. I am not one of them! This service looks awesome to me.
vvpan 3y ago
It's not an option if you have more than one machine and you want the machines to be uniform. If they are not uniform means it's back to square one - bespoke setups. And then I want reproducibility across environments and environments that are easy to set up (for testing for example). Basically I think it's not an option.

MuffinFlavored 3y ago

> Honestly, I have never been a fan of managing services in AWS/GCP/etc - the setup overhead rarely seemed to outweigh the pros I just `ssh` into prod and `git pull` and either `docker compose up` or `kubectl apply` or `terraform apply` or `helm update` or `argocd sync`

ketzo 3y ago
Okay, this is either really good satire, or you are wildly overestimating the average startup engineer's knowledge of each of those tools and their configurations

bauble 3y ago

I don't understand the reluctance to hire a system administrator. If you have two or more technical folks, at least one of them should be a sysadmin (or "devops" if you prefer). Pretending that it's an unskilled role that anyone should pick up reflects a poor understanding of the industry.

pech0rin 3y ago

The "crappy bespoke cloud setup" is usually the result of people whose jobs are usually just regular app devs, but had been thrown into the infrastructure side head first. I think the key is to actually have someone who knows what they are talking about. Unlike other people I don't think you need to hire someone directly. I do devops freelancing so you could hire someone like me. Or look online for various people that do it. I mean if its a toy app without many customers then who cares, but once you start scaling infrastructure gets to me more and more important.

no_wizard 3y ago

I think this is where modern PHP (8.0+) has its strengths. Its well known how to deploy. Its easy to deploy. The footguns are vigorously and thoroughly documented. The frameworks (Symfony, Laravel) are good. They all have a solution for X (be it queues, async work, cron jobs etc) and they have turn key deploy solutions favored by their respective communities. Its honestly one of the easiest languages to deploy with nowadays, in my estimation. Rivaled only by C# / F# or platforms like Deno Deploy or Cloudflare workers. Sometimes the most boring / mature thing is the best thing.

jcusch 3y ago

There is a collection of new tools trying to address this exact problem such as Nitric, Winglang and Ampt for example. (disclaimer, I work on Nitric)

rstuart4133 3y ago

> We have no dedicated devops person and much of that work falls on me and other people who would be better writing code. I think the cloud paradigm is experiencing a shift. Part of that shift is that prior to the cloud, the development environment and production environments were very similar. By that I mean if a programmer could set up his debian / redhat laptop from the command line, then it wasn't a big jump to set up a server running debian / redhat in a production environment. But no one in their right mind runs k8s on their laptop just to support VSCode and a compiler. Worse they could not run AWS/GCP because they are proprietary, insanely large, complex and forever changing. I'm pretty sure becoming an expert engineer in those environments is a full time job. The end result is a new class of engineers have arisen that look after the production side, and the developers just throw their code over the wall once it's tested. I'm sure the cloud providers love this, as they have created a cohort of engineers that have invested years in learning their product, now have their wages dependent on that product being used within their organisation, and a total monopoly over that product because it can never be reproduced by anyone else. But for the industry as a whole, it looks to be a backward step. Once the problem expands beyond what a single human mind can cope with, you need teams. The communication overhead of a team means they don't have a hope of being as productive as a single person. The share amount of complexity introduced by these proprietary cloud frameworks looks to be unnecessary. Most of it smells like technical debit created by organic growth together with an insistence on backward compatibility (don't want to give those locked in customers an excuse to move). And desktop OS's are (Debian / Windows / ...) are insecure by design, at least when compared to their phone brethren. And, their phone brethren look somewhat like cloud designs now, Phones have isolated apps with private data areas. The apps communicate via channels provided by the OS, and the availability of those channels is controlled by permissions assigned by the user. In the cloud we isolated things for performance as they could run on distance machines, whereas in a personal device we did it for security. But the end result looks similar. So maybe one day we will end up where we started, with a developer environment looking like the cloud we deploy too. I fear I'm too old to experience if / when it does happen, but it does seem like something we should aspire to.

brundolf 3y ago

+1, as someone whose company has a crappy bespoke cloud setup that's a massive time sink

alberth 3y ago

PaaS Sounds like you’d like to use a PaaS provider. Heroku/Render/Fly/etc.