22 ms·

Ask HN: Solo-preneurs, how do you DevOps to save time?

• popotamonga 5y ago
  I do all like this: GiHub actions-> docker build->docker push to github package repo then on vm a 2 line script to docker pull && docker-compose up -d Database all by hand. Extremely easy i just push, wait build and restart
• oliv__ 5y ago
  Call me nuts but I run my own startup's DevOps in an extremely minimal, low-tooling fashion: - Linode VPS - Nginx - SSH to deploy code - Google Cloud hosted (& backed up) DB Why would I need K8 and some crazy architecture when I'm just trying to get my product to my customers in the fastest / most efficient way possible? I'm not FB or Amazon and if I do get that size, I'll have plenty of time to adapt on the way there Finally I was going to quote the Pieter levels case but someone else beat me to it
• BilalBudhani 5y ago
  For the longest time I have used capistrano to deploy my projects on VPS from Cloud providers. It served me well but wasn't as convenient. At the start of this year I joined hands with Chris on https://hatchbox.io https://hatchbox.io to work on cost effective deployment service. We are putting together all the deployment best practices we've learned over the years into this product in a cost effective way.
• twobrainy 5y ago
  I pay money so I can sleep easier at night! Envoyer and Forge for PHP stuff. Render/Heroku for node.js stuff. Netlify for static stuff. For the more complex bits I have, I use Pulumi to manage everything through IaC. It's still complex but at least it's robust and I can sense-chech changes to infra through a PR to myself.
• soneca 5y ago
  I only have side-projects, very different from a full-time, revenue earning solo-preneur, but I only use serverless. My preferred stack now is NextJS, hosted on Vercel with FaunaDB. The main reason is to “outsource” devops to them. I am not sure how it would work on a larger scale, but for my use is perfect.
  • aristofun 5y ago
    How much money do you spend on this?
    • soneca 5y ago
      Right now, zero, as I am using only free tiers. But my understanding is that it is expensive when you scale. Vercel first paid plan is $20/month and FaunaDB $25/month, so my guess is a business with some traction would have to pay at least these $45/month.
• vhodges 5y ago
  https://fly.io https://fly.io - Easy Postgres clusters (but not managed) - Heroku like deploys (but less magic) - Easily geodistributed (deploy near your users) - reasonable pricing to get started (and not too bad beyond that).
• tofukid 5y ago
  A single VPS server with a GitHub action that builds and syncs code to the server using Git, with a bash one-liner to restart the server. Properly configured this can serve millions of requests. A cron job which saves a copy of the DB to S3. That’s it. If you’re focused on anything more with no employees you’re probably wasting your time.
• Longwelwind 5y ago
  I'm not a solo-preneurs, but I have some open-source projects (mainly a live one, and one that I'm currently developping). Here are the list of DevOpss flow I've done: * One project is a website. I have a pipeline, triggered when something is merged to master, that builds Docker images, pushes them to a Dokku instance and deploy them. * One project is a JS library. I have a pipeline, triggered when a git tag is created, that packages the library and uploads it to npm. I use Netlify to host the documentation website of it. * For this same project, I'm planning to have a pipeline triggered for each PR to run the linter and tests. I work as a DevOps/MLOps, and I'm quite comfortable with CI tools in general. I hate doing things manually and I love automating, so it comes quite naturally for me to do it. Once it's set up, it rarely needs to be changed.
• woudsma 5y ago
  Not a solo-preneur, but I’ve had great experiences so far with Dokku, an open-source Heroku clone. I’m running ~40 apps on my single VPS since a couple of years. I use it in combination with Github Actions, Bitbucket Pipelines and Gitlab CI. I still feel very flexible working with Dockerfiles and different cloud services. I’ve only had to update the dokku-letsencrypt plugin a couple of times in the past 5 years. Note that I haven’t had to protect important databases with PII for example.
• NomDePlum 5y ago
  I'm starting to consider this myself and have been looking at https://serverless-stack.com/#guide https://serverless-stack.com/#guide as a way to prototype and build an MVP. The guide has quite a bit in it that I believe can be repurposed to that end although it doesn't cover backups etc, more integrations with AWS services.
  • jayair 5y ago
    Yup it's meant to be a starting point for projects of all sizes. We built our own stuff using it as well. Feel free to reach out if you have any questions. Or join us on Slack: https://launchpass.com/serverless-stack https://launchpass.com/serverless-stack Good luck!
• xrd 5y ago
  GitLab and Dokku. You can run almost ANYTHING inside dokku. I use it to run a private/public docker registry, minio (amazon S3 clone), analytics via matomo, and a ton of mini apps that share a bunch of databases. It's incredible. With dokku, deploying your app is just "git push dokku main" GitLab plus CI is amazing. And, other than hosting, I don't pay a penny for it. If you self-host, you can put it behind a wireguard VPN. It's incredible. You can run as many runners as you want on your personal laptops, or on the cloud, and bring them up and down when needed.
• swlkr 5y ago
  My solo ops/data stack is: - docker - dokku - sqlite - litestream
• Inhibit 5y ago
  First you worry about meeting your paying customers need. Then you spin up just enough infrastructure to handle effectively meeting the need to continue getting paid. You can do that with some shell scripts on an instance and an e-mail service and manually copy the whole mess for backup if it makes sense. The how of implementation is very case specific. Thinking more on it I'd sum up as "provide a service that doesn't require that you be Google". If you start by specifying an impossible amount of infrastructure maybe there's an issue with management :).
• adamcharnock 5y ago
  I think this is going to be a pretty contrary comment, let's see how this goes. I don't know what everyone is doing with kubernetes that takes up so much time and effort, especially in a SME environment. I'm currently operating three separate kubernetes clusters for three different clients. And this is just me, there are no other tech people involved day-to-day. These clusters require minimal maintenance, just the occasional upgrade. These clusters are small (1-10 nodes), and are on either GKE or bare metal. The biggest is around 100-200 running pods. I have a CI/CD pipeline setup in GitHub actions for each client. Development & CI uses docker-compose. I used to do the old-hat 'scp your code to the server' kind of deployment, and this was a lot more pain. Kubernetes is big, but it solves so many problems. Ensuring everything is actually running, networking, SSL certs, persistence, configuration, etc etc. I've been running this setup for 3+ years now. If I cast my mind back, I think I spent a weekend reading the k8s docs concepts section, then pretty quickly deployed a cluster on GKE. Given it was a team of one, I ditched RBAC which removed some complexity. Getting the CI/CD pipeline setup took some time (maybe a few more days), and I built more tooling as a went. I really think kubernetes can be really useful for tiny teams, but there is a lot in there that is aimed at big teams. Use the bare minimum possible at first, and grow from there. --- Now here is a bonus unsolicited contrary comment. Deploying k8s used to really require a cloud provider, particularly for persistent volumes and load balancing. But I think that has changed over the last few years. OpenEBS and MetalLB look like they are really addressing this. Also, has anyone noticed that dedicated servers are cheap as chips now? I think the equivalent dedicated server would cost around 10x as much as an AWS instance. (I'm looking at Hetzner in particular, but OVH isn't far behind) Given what is currently available from the k8s ecosystem, there seems to be a really strong case for deploying on bare metal these days. For example, at a ballpark you can get a 400GB+/40core cluster for around $300/month (which also includes another 80GB/10cores of ancillary gubbins). The cost for that easily comes in at around $3,000 on AWS (ex any storage). OpenEBS gives you replicated storage, MetalLB gives you your load balancing, and each instance has a couple of terabytes of NVMe too. And, if you're worried about reliability of physical servers, at that price you could replicate your setup across datacenters at minimal cost. --- I should add a caveat here. I've been working freelance in tech for 15 years and I am a massive generalist. I do everything from business consulting to design, devops to development, and I also run an ISP so know my way around networking/BGP etc. That being said, setting up a quick cluster on GKE doesn't require many of those skills. Having written this ambling comment I guess I'll also say that I am am available if anyone wants a cluster setting up. I'm also about to start providing managed k8s services.
• BiteCode_dev 5y ago
  Talking about myself, as a python dev: - cheap VPS for hosting, postgres for the db, nginx as a proxy, redis for everything else, including caching. - deploy python projects by packing them as zipapp with shiv, then use fabric to ssh and performs any migrations necessary. No, not even ansible. - build, lint, format and test, like all automatizable stuff, are made using pydoit. If you are solo, you don't need a CI service. Your laptop is the CI machine. Scaling is always the same story: 1 - start with the cheapest VPS possible. This will force you to code with some constraints. Not crazy, just enough to balance it out. 2 - once you start seing your load average rising, just move the db on a second cheap VPS. You just tripled the load you can sustain. Not doubled, tripled. Most solo projects will stop here actually. 3 - wait again, collect some perfs data when you see the site starting to be slow. Check for slow queries, code hot paths, etc. Optimize that, add some cache, setup tasks queues. Use cron to purge / regenerate cache if you need to smooth out the curve. Maybe slap some varnish for extra peps, or just cloudflare. 4 - now you got what looks like a real load, and a more realistic code base, so when you peak again, request more perfs for your server, or migrate to another bigger one. You can scale vertically to crazy highs nowadays. Really, really crazy. You can get terabytes of RAM, 64 cores servers, etc. At your current scale, your service should generate enough money to pay 10 times for it anyway. But you probably won't need to. Even cheap servers are beast, look at the current leaseweb offer: https://www.leaseweb.com/dedicated-servers#NL https://www.leaseweb.com/dedicated-servers#NL For €320.09 per month, you get: 2x 16 cores 2.30GHz 28GB DDR4 4x960GB SSD 30 TB traffic To put it in context, myspace used to serve all its users with 2 servers only, until they reached 500,000 accounts. Only then it was too much. This was with hardware (and price!) in the years 200x. 5 - you will probably never reach this point. This is the point where kubs, load balancers, sharing, etc. start to be interesting. How to preserve your data: - Raid - Dump the db with a _randomized_ cron - rsync the dump and all assets. You can do it to another server, or just your laptop at the beginning. You can get fancy with database replication if you want, or use backups that stream in real time. But there is one trick: not all data are equal. Identify some data in your db that you can't afford to lose, and make sure this one is saved separately and very regularly while being given priority. There are plenty of data, if you have a hole in it, most people don't care. E.G: if 1 tweet out of million from 10 years ago is missing, do you think it affects the service? Monitoring: - Sentry That's all. It's free (or cheap), and for a small service, you don't need real time. It's ok to be down a few hours once a month for most services at first. I have a service with 700k unique users a day, it still goes down sometimes. It's a blip. Sometimes, log into the servers, run htop and checks what's up. You can install open telemetry later if you really need to, but for now, even HN hug is not going to kill you. Summary: - Modern software and hardware are amazing. Max them out. Horizontal scaling is hard, and expensive. - You don't need a perfect service. Unless you are handling patient cancer data, that is. Don't worry about perfect uptime, 0 data loss, etc. If you are a solo dev, the cost for that is huge. Just do 97% right⋅ - what worked 20 years ago still work today, and will likely be there tomorrow. And you can move from that to the cloud later. The reverse is not that nice.
• austinpena 5y ago
  For my side project I try to keep it as simple as possible. After I run tests locally, I've got a makefile that deploys to either Google Cloud Run or fly.io. Database is Cloud SQL which works well enough for me.
• piyiotisk 5y ago
  I've used this in the past https://github.com/piyiotisk/fieldbot-server/blob/master/.gitlab-ci.yml https://github.com/piyiotisk/fieldbot-server/blob/master/.gi... Basically, building a docker image and server it using Docker compose. Of course you'll need NGINx as a reverse proxy running on the server. If I do it again I'll use a service like render. Not worth managing this myself.
• hubarb 5y ago
  Lots of easy to use CI tools about. Bitbucket, Github both have easy to use config based pipelines/runners. Very little to manage there. Build containers and get somebody else to run them for you. There are lots of options to do that too. There should be very little overhead in managing that.
• jon-wood 5y ago
  I have no idea why there's nothing here saying "don't". Go and get a Heroku account, hook it up to your Github repository's master branch for deploys, use Github Actions for CI, and then get on with life. Yes, you'll pay more for a Heroku Postgres instance than you would for a VPS on Digital Ocean with Postgres running on it, likewise you'll pay more for Heroku Dynos than another VPS to run your application server. On the other side though, your backup, rollback, deploy, and scaling strategies can all be "Heroku does that", and you can focus on the much more valuable job of building a business.
  • darkr 5y ago
    +1. For pre-A series/pre-revenue/pre-1k paying customers and other early stage businesses you can do a lot worse than starting out on Heroku, especially if you don't have sysadmin/cloud/devops expertise (and even if you do, that time could/should be spent on much more important things at this stage).
  • strzibny 5y ago
    Until Heroku has a huge downtime (like recently) and you are f*ked. If you have your own VMs, you can quickly move them to a different provider, but it's hard to start figuring out servers when Heroku is down and your business stopped. That said, I would also recommend Heroku to many.
    • jon-wood 5y ago
      I can almost guarantee that over a period of years you'll see significantly less downtime that can be attributed to Heroku than you would on VMs that you manage. Even if that isn't the case, if Heroku are responsible for it then you can mostly just kick back and let them fix it, rather than spending 24 hours without sleep migrating VMs between providers. The context here is also that of a single entrepreneur building a business. At that point in the lifecycle of a product downtime really isn't that big of a deal, customers tend to be pretty understanding so long as you recover.
      • ohashi 5y ago
• jamesmishra 5y ago
  I wrote https://provose.com/ https://provose.com/ so I could quickly set up databases, containers, and filesystems on AWS. I wrote Provose because I needed to deploy a ton of infrastructure to AWS as a solo developer, but I could not afford to use a dramatically more expensive PaaS like Heroku. Provose is an open source project written in pure Terraform, so all of the high level abstractions are computed locally, and in general do not raise your AWS bill.
• imedadel 5y ago
  I use Dokku (https://dokku.com/ https://dokku.com/). It works with GitHub Actions and it's enough for most use cases.
• wjd2030 5y ago
  Devops is a culture, not a verb. Success generally being measured by 4 key metrics: Lead time to change Deployment Frequency Mean time to restore (MTTR) Change Failure Rate. Though it often is, I do not believe it should be used interchangeably to refer to a suite of tools designed to enable CI/CD and SDLC Lifecycles.
• q-base 5y ago
  Sounds like you are overthinking and/or over engineering it. Unless CI or deploy/rollback functionality IS your product then it does not matter. Until you have got traction for your actual product then deployments can have 100 manual steps that you can write down and follow - it does not matter. I have no idea what you are building or planning it to be but for my projects I have a single VPS with Hetzner per project running linux and hosting the applications. Deployment is done by a powershell script that copies files from my laptop to the server - that's it. If I were to think about clusters, cloud instances, load balancers and what not then I would be scared away before having delivered anything of value. Cross that bridge when you get there.
• bhouston 5y ago
  Vercel, Porter, Heruko is sort of the simplest way to avoid DevOps. Use their integrated services for data and storage until you get large enough to afford a DevOps team.
• tnolet 5y ago
  Simple. Don’t use K8S. It’s absolutely 100% the wrong way to go for solo devs with no product market fit. Use Heroku (or any other PaaS) and throw in some serverless stuff here and there. Again DO NOT geek out on the tech. Geek out on getting customers.
• Layvier 5y ago
  I've been using a simple circle CI setup and AWS beanstalk on my side, and for the databases I only look for fully managed. I've had a few issues with beanstalk but it still saved me a lot of time, and it's pretty easy to find AWS credits. It's pretty good also to avoid wasting time on SSL setup as everything is integrated (if you use a load balancer) with Route 53 and ACM. Beanstalk uses application version so it's very easy to rollback to a previous version. I never liked DevOps much so this setup works pretty well for me, and seems quite scalable as well. If you're not very comfortable with k8s yet I really wouldn't recommend you to setup a cluster for a small project.
• lbriner 5y ago
  Like most others said, if its not a painpoint now, just deploy manually and keep it simple. You will save time and money and more importantly, you can concentrate on what is important for early-stage companies: Nailing product-market fit so you can get the sales moving in. It isn't that hard, for someone with decent experience, to setup TeamCity/Jenkins/Octopus/Whatever but why spend the time? If you change something, you then have to go and spend time changing the automatic deployment. As someone else said, Devops is about velocity with quality, if velocity is OK and quality won't improve with tooling, don't bother.
• moltar 5y ago
  AWS CDK, and use the library constructs and solutions they provide. It’s like LEGO blocks.
• ArtWomb 5y ago
  Choose a platform like App Engine. It's auto-scaling. Supports bigtable. And opentelemetry. Manageable from anywhere via Cloud Shell. You can run multiple instances and partition load between them. Even includes a free tier ;) https://github.com/GoogleCloudPlatform/golang-samples https://github.com/GoogleCloudPlatform/golang-samples
  • BiteCode_dev 5y ago
    You did that, as a solo dev with little money, and made a profitable business out of it?
    • ArtWomb 5y ago
      not quite yet ;) ps some of the google cloud next announcements today around appsheet, appscript, looker, ai vertex notebooks, and "no-code" solutions are starting to look really sophisticated!
• dsgrillo 5y ago
  I have currently 2 projects with some traffic (5k req/day) running in production. Both have a similar setup: - 1 droplet with docker pre-installed from DigitalOcean - clone directly the repo from github - together with the code, I have a folder with a bunch of docker images (caddy, mysql, php, redis, etc.) that I can easily spin up. - for any release, I manually ssh, git pull and run the migrations (if any) and manually rebuild any docker image if needed - I have daily jobs that dumps and zip the entire DB to S3 - if I have some deployment that I know will break any functionality during deployment, I warn the users before and accept that downtime. - never had to handle a "hard" rollback till now. I've planned to change this setup for while, but until now, didn't find any reason that justifies the effort. I spend 20$ (10$ per droplet) + few cents on S3 per month with them.
• adventured 5y ago
  I've been a solo Web/Internet entrepreneur for most of 20+ years, non-stop. The DevOps side of things is trivial in the beginning and takes very little time compared to actual development. Ubuntu server, Nginx, Mysql or Postgres, PHP, Go, Redis. Configured reasonably, it's a ridiculously reliable stack where things very rarely go wrong. I prefer DigitalOcean these days. Takes <15 minutes to configure a base new setup - a template - from scratch, make a few adjustments and double check everything. From there I can pop up a lot of servers as needed. I usually tweak things based on the project, although that doesn't take a ton of time. And that's it. Back-ups on eg a database focused droplet are automated by DigitalOcean. Occasionally I have specialized back-up needs, and I'll do a bit of custom work for that. Most of this could be offloaded by using DigitalOcean's database service, I just prefer to limit cost by not. Under no circumstances would I use Kubernetes for a smaller to medium size service.
• TekMol 5y ago
  I scaled to millions of users as a solo founder and still run the whole show myself. AMA if you like. Here is what works for me: CI: From the terminal, I run my tests and commit to git. Deployments: rsync Rollbacks: Never did one. If something breaks, I fix it and rsync the fix to production. DB: MariaDB k8s: I don't use it. Computers are very fast these days. A cheap single VPS will get you a long way. Nightmare: Not really. I spend about 30 minutes per week on DevOps.
  • CodinM 5y ago
    What is the domain of your venture? (assuming it's not something you want to advertise specifically)
    • TekMol 5y ago
      I like to keep it off my HN profile. I feel I am more open to share details this way.
      • lambdaba 5y ago
        Do you mind putting in some contact details (even temporarily) on your profile? I'd like to have a chat with you some time, I was looking through this whole thread for a comment like yours and I resonated a lot with you. (my details are in my profile) Not immediately, but in the following weeks. Thanks again for sharing.
  • aristofun 5y ago
    Best comment so far. Thank you for being specific and to the point.
  • deleted 5y ago
    [deleted]
  • launchiterate 5y ago
    What are the most important things you learned and what would be your core advice for someone who wants to run a solo operation? How do you schedule your time to work on your project and how has it changed over time? Thanks in advance!
• CodinM 5y ago
  I'm using Hetzner, Terraform, GitHub Actions, Docker to automate the build and release of an app I'm planning on launching (..eventually). I use Terraform to provision the Hetzner VMs with Docker running on them, in there I have a container with the GitHub Runner that pulls Flutter and creates a nice build environment as well as release env (using Fastlane), then it uses the latest code, compiles, releases to Play store. Works like a charm, initially it was just a Docker container on a Pi, but the main reason I went that route was being able to move it whenever/wherever.
• Aulig 5y ago
  I'm a noob when it comes to devops, so you might not want to do it my way. I keep it super simple. I have a git repo where I directly commit into master. When I want to take the changes live I ssh onto my VPS, I git pull the changes and restart the website service. I use an SQLite DB that I automatically copy (backup) to my local machine daily via the Windows equivalent of a cron job. Once it's on my local machine it's also backed up to other places as part of my local backup process. I run my tests manually when I think I've made significant changes. Otherwise I just react to errors ASAP (I get an email notification from sentry when a new error occurs) A lot of this doesnt work with a bigger team of course. It's a balance between risk and time investment. The simpler your dev ops stuff, the higher your risk of introducing bugs into production. But it saves you a lot of time not dealing with that stuff.
• brightball 5y ago
  You avoid it as much as possible. When your time is that valuable, lean on PaaS options like Heroku, Fly.io, Gigalixir, etc. If it’s just you, you can afford to use a simple shell script to run tests locally before deploying or use a Travis/Circle/Cloudbees CI option too. You have enough to focus on with the application itself.
• p0nce 5y ago
  I have no DB, static website, deploying in production on a VPS with git (manually) and rsync + Cloudflare. I just don't know what kubernetes is, nor do I encounter any of the problems you mention. > Just properly setting up and maintaining a small k8s cluster looks almost like a fulltime job. ok.
• eatonphil 5y ago
  I'm building an open-core data IDE that runs as a desktop application or server. Since it's open source, all tests run on Github Actions for free. This includes basic e2e testing using Selenium on Windows, macOS, and Linux (e.g. [0]); and unit/integration tests. If it were a private repo I'd still shell out for Github Actions or Circle CI most likely. I'd also consider buying a chunky-enough minipc for ~$500 and an older mac mini and set up runners on them. For the moment private runners isn't a problem. But soon I'll need to start integration-testing proprietary code paths like querying Oracle or MS SQL Server. In that case I probably need to set up a dedicated box with all the right licenses so I can run CI jobs on it. [0] https://github.com/multiprocessio/datastation/blob/master/.github/workflows/pull_requests.yml#L51 https://github.com/multiprocessio/datastation/blob/master/.g...
• vishnumohandas 5y ago
  > CI GitHub Actions > deployments/rollbacks Docker. Scaleway offers a container registry that's ridiculously cheap[1]. Deployments are infrequent and executed manually. > DBs Again, Scaleway's managed RDS[2]. Outside these, we have setup Grafana + Loki cloud[3] for monitoring and alerting. They have a generous free plan. For product analytics that can be derived from our database we've a self hosted instance of Metabase[4]. [1]: https://www.scaleway.com/en/container-registry/ https://www.scaleway.com/en/container-registry/ [2]: https://www.scaleway.com/en/database/ https://www.scaleway.com/en/database/ [3]: https://grafana.com/ https://grafana.com/ [4]: https://www.metabase.com/ https://www.metabase.com/ P.S. We were a 1 person company when most of this was setup. We're 3 now, works just as well.
  • aristofun 5y ago
    Thanks for being specific.
• caffeine 5y ago
  Use Linux, pretend you’re in the 90s/2000s. If you can’t afford a team, you’re too small to need more. - Use NFS (EFS), put builds in a /releases folder - Bash scripts for build, deploy, release - Bash scripts to wrap apps (write pidfile, start, stop, etc) - Cron those scripts - Check crontabs into a repo along with all other config - Cron a bash script to pull from that repo and update crontabs on every box - Have a staging environment, deploy stuff there (esp DB migrations) and do smoke tests. - Have some kind of monitoring (I have a slackbot that sends me messages when things break) I general I follow the principle of proportional response, and n=2 (or 3 depending on how trivial the task is to automate). Proportional response means you invest in automation / tooling proportional to the pain you have suffered. Burning a day to build a bulletproof script only makes sense to solve a big problem. For a smaller issue, maybe adding a debug line or writing a line somewhere that you can copy-paste later is good enough. N=2 means you don’t solve stuff until the second time you need it. This stops you from burning hours building clueless complex garbage to solve unnecessary problems - at least by the time you are automating, you have solved it once or twice by hand and you know where you are going. Elon’s 5 principles have been very useful to me as a solo dev: 1. Make requirements less stupid 2. Delete the part/requirement 3. Simplify/Improve the design 4. Accelerate cycle time 5. Automate So in particular you might try 1-4 to see if you can magic problems away before you invest in building devops automation you will then need to maintain.
• aristofun 5y ago
  I wonder why no comments about using docker containers directly or with Swarm
• mibollma 5y ago
  Having 90% of the system in a single Cloudformation template worked pretty well for me. On every stage (dev, test, prod) I can deploy it automatically up to two times (blue and green) by running a simple shell script. So when I update the application I just deploy it another time on prod, test it one last time by adding the IP of the load balancer in my local hosts file and if I'm satisfied with the result switching the DNS entry to the new version using a weighted DNS record with the ability to switch back until I shut down the old version. Doing anything continuous doesn't feel worth it in such a small setup with one update every 1-3 months. What I like most about the approach is that I'm free to change any aspect in the main template without any risk to break something in the live application. Only changes to the elements shared between versions need to be handled carefully. The main template includes VPC, network, fargate service, loadbalancer, firewall, KMS, DNS records, access rights, database tables, queues, monitoring metrics, email alerts It excludes everything that is shared from update to update which are defined separately and just referenced from the main template such as some database tables, persistent storage container registry, user groups (Only the groups, not the rights assigned to them)
• SquishyPanda23 5y ago
  I personally like k8s a lot, but my general approach is to fake it until you make it and only add things once it's clear why I need them. I have a few old boxes that run VMs. Some of these VMs make up a virtual k8s cluster, but almost everything I need runs on bare VMs rather than on k8s. When I find something particularly annoying to administer, I move it off a bare VM and onto k8s. The main thing this gives me is a migration path off of "a bunch of custom shell scripts" toward something I believe will be useful in the long run. A bunch of shell scripts and rsync will work for a long time until they don't. And when they stop working you'll be in for some pain.
• vishnugupta 5y ago
  Not solo but I head a tiny tech team (4 engineers) so we don't have bandwidth for DevOps stuff. Here's our current state. We are all in on AWS. 0. Tech stack: NodeJS, React, RDS Aurora, Redis, S3, Route 53, SES, Lambda and a few more. 1. 3 EC2 machines behind an ELB. Deployment is scp + cd <deployment-directory> + ./deploy.sh prod 2. Aurora takes care of everything around DB for us. Backup, read/write replica etc., etc., 3. No CI; our development cycle isn't rapid enough to warrant investing in CI. 4. We recently moved some of long running tasks to Lambda. There's definitely a big value here so we'll invest further on it. 5. Also, recently, we experimented with an internal service on "ALB + AWS Lambda"; it's a sweet combination so we'll invest further. 6. Frontend is hosted on AWS Amplify. It's an underrated nifty service; highly recommend it. 7. Datadog is for monitoring. It's quite good but a bit expensive. I can tell you from my experience that CI, auto-rollback etc., are overrated for smallish teams given their development velocity. Even if you have to do once or twice a deployments per week you can do it manually, don't need to wade into the CI territory. And unless you have prior experience steer clear of k8s.
• a_square_peg 5y ago
  I run https://oikolab.com https://oikolab.com - it's a weather data service with global, hourly historical weather data from 1950 to 16-day forecast. It launched about a year ago and I think it's probably the largest data offering of its kind. I don't have dev ops experience (I still don't know much) so a lot of what I did was to exhaust the limits of the setup before implementing any new infrastructure, sort of learning as I go along, with many 'oh, so that's why they do this' moments. I've read about K8 but still can't see when I would need that though. A lot of what saved effort for me was to use off-the-shelf component whenever I can. I was quoted $10~20k to build a Django website for user & API management to connect to my back-end but it was much easier to use Azure API Management service (~$400/month), which also came with simple, reasonable looking web front-end that I was able to launch within a week. Before moving things to cloud (Digital Ocean), I've also exhausted what I was able to do with simple NAS servers (overall I processed about 200TB of raw data with ~1TB/day on-going using what I call a 'poor man's HPC' setup) - luckily I'm based out of Hong Kong so had access to fiber-optic internet at home.
  • terrortrain 5y ago
    Nice, I bookmarked your site. Weather data is surprisingly hard to navigate. Did you source it from NOAA.gov or somewhere else? I tried to slap together a simple weekend app based on historical weather data by zip code, and never even got to the coding. Just spent most the weekend trying to decipher those API's.
    • a_square_peg 5y ago
      The primary forecast data is from NOAA while the historical data is from ECMWF (sort of the European equivalent of NOAA). I’ve been told that my API is easy to use - feel free to let me know your thoughts if you get a chance.
• nkotov 5y ago
  I had a chance to setup several businesses in the past and to setup the foundation - I ended up writing a lot of templates for myself. Nowadays, if I want to setup a SaaS company, I have 90% of the foundational groundwork covered (AWS, Terraform, CI/CD, Github, etc.) We're building [1] something similar to solve this problem but just on the infrastructure end. I really believe in the power of templates. In our future release, we're planning on launching a "infrastructure" catalog of commonly used setups so that you can just get the foundation in place. Why re-invent the wheel every time when you can just spin up a boilerplate that gets you 90% of the way there? [1] https://atomizedhq.com https://atomizedhq.com
• marvinblum 5y ago
  I found the HashiStack to be manageable by a single person or very small teams. It's easy to set up and hasn't let us down so far. We get away with using shell scripts to test and build our containers for now. The trickiest part is scaling as soon as it becomes necessary. Our current three-node cluster handles a few hundred requests a second without issues, but I migrated the database three times already, and I'm in the progress of migrating it a forth time to a much bigger machine. You can find more about our setup and ops here: https://pirsch.io/blog/techstack/ https://pirsch.io/blog/techstack/
• jbrun 5y ago
  We use an external firm. Works great.
• JanSt 5y ago
  Digital Ocean App platform is doing most of my work and getting better by the day
• DethNinja 5y ago
  I just use Ansible, Docker, Jenkins, Gitea, NewRelic, rclone, and some custom build services with GoLang for DB backups and maintenance, that’s it. I’m fairly sure I can maintain up to 50 servers as solo developer with the above mentioned tools.
• ensemblehq 5y ago
  As a solo founder, your time is the most precious resource so all your DevOps effort should be minimized to the extent that you can prevent and debug issues. In practice, for me, this meant using tools that I'm already comfortable with (preferably in the same language as my codebase to minimize context switching) and resisting making things overly complicated with too many tools in the chain. For automated deployment, I'm using Fabric (for Python) that allows me to remotely run commands. DB backups are handled by cloud snapshots and rollbacks are ad-hoc based on Git. This was also how things began at a previous startup I worked at until we grew and we could hire more resources to fix things.
  • chrisgoman 5y ago
    using fabric for deployments (fabric2 kind of a pain but almost got most things ported over) even if our main app is in PHP
• sergiotapia 5y ago
  I recommend not doing any devops at all if you're a solo founder and trying to get it off the ground. You want to spend time on your product market fit, not fuck around with linux. New services like render.com make it really easy to get robust, cheap hosting up. Much cheaper than Heroku.
• vmt-man 5y ago
  I prefer Gitlab CI for solo projects, it's super convenient. Just create .gitlab-ci.yml file, attach runners on my pc, laptops and go. Deploy with scp/sftp. Also it's very useful to have DEV.md file for each repo. It helps to remember how to build and deploy your project. Documentation for future me :)
• scottydelta 5y ago
  Since all the people in this thread do some kind of personal hosting/DevOps, I am posting one of my recent comments on how to make the security of your infrastructure rock solid: > I self-host three VPNs to protect my infrastructure heavy startup https://quantale.io https://quantale.io I am a big fan of Pritunl which is opensource and provides network security with a lot of ease. I am in no way affiliated with them, I am just a big fan of Pritunl. I use Pritunl to limit access to servers and web applications for my different teams. For each user, you can generate a profile and assign the servers and port they have access to on the server. For eg: - Only dev team can access ssh port(22) on stage server and not open to internet. - Any one in the team can access stage version(port 443) for testing purpose.(Not open to internet) - Only I can access all ports on all Prod servers(only 443 open to public) What hackers can't see, they can't attack. Especially the port 22 on your servers should only be accessible to you and not the internet. I self-host one instance each of OpenVPN and Wireguard with Pi-Hole which is then used to access my Pritunl Server adding extra layer of security. Each of these 3 servers can be hosted on Hetzner $2/month instance. With a mere $6, you can add an extra layer of security to your infrastructure. Pritunl itself also provides subscription so that is also an option. https://news.ycombinator.com/item?id=28671753 https://news.ycombinator.com/item?id=28671753 If you want to discuss more about this or security in general, feel free to reach out to me on my email on profile.
• deleted 5y ago
  [deleted]
• deleted 5y ago
  [deleted]
  • deleted 5y ago
    [deleted]
• sealthedeal 5y ago
  Just use Heroku until its time to scale
• deleted 5y ago
  [deleted]
• throwaway2016a 5y ago
  Don't: - Manage your own DBs for production. I used to manage my own but now I use IaaS for it. It's worth the extra cost even if your budget is low since it lowers your risk and lets you move faster. AWS Aurora is awesome but expensive. There's a SaaS/IaaS out there for most DBs. And they handle the backups! - Use Kubernetes. k8s is awesome, I love it, but it is a beast and probably overkill for your solo ops team that is also multitasking. - Manually SSH / rsync / scp directly onto prod. Do: - Version control (Git) everything - Use containers where it makes sense (I use Docker when I'm not using Serverless). It keeps your dev and prod in sync and helps with immutable deployments (see later points) - Use Infrastructure as Code - Use Github actions to do your deploys - Use immutable deployments. More time to setup but makes rollbacks easier and if you never hot patch production you can't create a unique state that can't be restored. Finally, (and possibly controversially): use serverless where you can (I personally use AWS Lambda, S3, and Cloudfront). My personal flow: - All my infrastructure is in a parameterized CloudFormation template. - The template is checked into Git with the service code. - After CI runs the tests, CD builds my serverless functions and uploads the zip. - Then it runs the cloudformation template. The same flow also works for Docker except "uploads the zip" becomes "pushes the container" Using this setup I can tear down and build up my setup in minutes, making it easy (and cheaper) to separate dev and production.
  • dsgrillo 5y ago
    > Don't: manage your own DBs for production. [...] Why, though? If you're really on a low budget, the extra $$ for a managed DB doesn't really pay for itself. I'd rather prefer spinning up a container running whatever DB I need and run daily backups to an external vendor. In my case, the website is free, living on ads revenue, so every penny counts.
    • throwaway2016a 5y ago
      Good question and I have an answer for you. It depends on your risk tolerance and resiliency. If you compare it to a single machine with multiple tenants and no replication, it doesn't make sense. But that comes with risks and given the OPs question in the context of a k8s cluster (which for HA is a minimum of 5 servers), I figured they are prioritizing time savings over money. Losing customer data, downtime, and data breaches can kill some companies before they even take off. If your particular business is not sensitive to that than you can run your own database. Or better yet, if you're a content business you may not even need a database at all. But even so, if you run your own DB you need to worry about backups and such so it may even still be worth the money. And if you are going for High Availability (HA), running a properly replicated cluster can be time consuming. The cost of a t3.medium RDS instance on AWS for instance is $0.068 / hour compared to a t3.medium instance on EC2 is $0.0416 per hour. That is an extra $19 per month. For my use case, paying an extra $19 a month is well worth the savings in time and the piece of mind I have knowing my backups are running and if the server crashes I can recover quickly. But you must consider use cases and your's might be different.
• tbronchain 5y ago
  I have worked mainly on ops/infra professionally the past few years and will strongly with the idea to avoid any complicated tool - i.e Kubernetes nowadays. I create my small/bootstrap projects the following way: 1- create a free tier AWS account - can work with any VPS/server really, but with AWS you can get it 100% free :) 2- create some Ansible provision/setup/deploy scripts 3- create some bash scripts to wrap this all Create the Ansible scripts: 1- Provision script, consisting of 1 EC2 instance, 1 RDS and the security groups. Store the IPs/adresses of the newly created instances. 2- Setup script, basic instance config, packages, app/project repo, you name it 3- Deploy script, to run every time I want to update the app, mainly just doing a git pull on the instance and some database backup/migrations (although on RDS backups aren't always necessary) I get these Ansible scripts wrapped into basic bash scripts, provide my AWS credentials via the AWS cli to keep it safe, few extra creds with Ansible vaults, SSH keys, publish in a Github [private] repo and I'm all set. It took me a couple of days to get fully operational and I was learning Ansible at the same time so it can really be done with basic features. Now it's done, I can reuse the same skeleton for every new project in a couple of hours! I find this solution extremely resilient. That's basically free hosting for personal projects. Every year, just need to run the script again when free-tier expire, and can change hosting provider anytime or upgrade the instance type if I decide to get a project publicly released. A small extra Ansible task to write for migrating the data and that's it!
• kukabynd 5y ago
  I use Dokku and Caprover, Git hooks and two branches (staging, master) which I think is more than enough for a solopreneur. Dockerized apps are easy to reason about and fix in case of fires.
• no_circuit 5y ago
  I decided to take a few years off work to just build on what I'd like. Perhaps in a startup studio model, so I have a bias for having something that is easily reusable, and that uses tech someone else can pick up and run with easily. I'll probably be in the business of dev/infra tooling. Currently going with a container image as the minimal deployable unit that gets put on top of a clean up to date OS. For me that's created with a Dockerfile using Alpine image variants. In a way I could see someone's rsync as an ok equivalent, but I'd do versioned symlinked directories so I can easily roll back if necessary if I went with this method. Something like update-alternatives or UIUC Encap/Epk: https://www.ks.uiuc.edu/Development/Computers/docs/sysadmin/Build/encap.html https://www.ks.uiuc.edu/Development/Computers/docs/sysadmin/.... Anyone remember that? I guess the modern version of Epkg with dependencies these days is https://docs.brew.sh/Homebrew-on-Linux https://docs.brew.sh/Homebrew-on-Linux. :-) Or maybe Nixpkgs: https://github.com/NixOS/nixpkgs https://github.com/NixOS/nixpkgs? Deployment-wise I've already done the Bash script writing thing to help a friend automate his deployment to EC2 instance. For myself I was going to start using boto3, but just went ahead and learned Terraform instead. So now my scripts are just simple wrappers for Docker/Terraform that build, push, or deploy that work with AWS ECS Fargate or DigitalOcean Kubernetes. No CI/CD yet. DBs/backups I'll tackle next as I want to make sure I can install or failover to a new datacenter without much difficulty.
• sirodoht 5y ago
  My web app is hosted on a server on Hetzner Cloud. I don't use Docker. For: * Database: PostgreSQL installed through apt in the same server: https://github.com/sirodoht/mataroa/blob/master/docs/server-playbook.md#postgresql https://github.com/sirodoht/mataroa/blob/master/docs/server-... * Backups: MinIO-upload to an S3-compatible object storage: https://github.com/sirodoht/mataroa/blob/master/backup-database.sh https://github.com/sirodoht/mataroa/blob/master/backup-datab... * CI: Github Actions + sr.ht builds: https://github.com/sirodoht/mataroa/blob/master/.github/workflows/django-build.yml https://github.com/sirodoht/mataroa/blob/master/.github/work... + https://github.com/sirodoht/mataroa/blob/master/.build.yml https://github.com/sirodoht/mataroa/blob/master/.build.yml * CD: (not exactly CD but...) ssh + git pull + uWSGI reload: https://github.com/sirodoht/mataroa/blob/master/deploy.sh https://github.com/sirodoht/mataroa/blob/master/deploy.sh * Rollbacks: git revert HEAD + ./deploy.sh * Architecture pattern: stick to the monolith; avoid to deploy another service at all costs; eg. we need to send multiple emails? not celery, that would mean hosting a redis/rabbitmq. We already have a database so let's use that. We can also use Django management commands and cron: https://github.com/sirodoht/mataroa/blob/5bb46e05524d99c346ccf1e5080cb51cb93aa9cc/main/models.py#L309 https://github.com/sirodoht/mataroa/blob/5bb46e05524d99c346c... + https://github.com/sirodoht/mataroa/blob/master/main/management/commands/process_notifications.py https://github.com/sirodoht/mataroa/blob/master/main/managem...
• davidkuennen 5y ago
  For the backend of my app[1] I'm mainly using Google Cloud: * Database: CloudSQL/PostgresQL (with Query Plan for analysis) * Services: Kubernetes (Services in Go) * CI: Cloud Build It's pretty easy to maintain. [1] https://stockevents.app https://stockevents.app
• terrortrain 5y ago
  Some if the comments here are way over the top IMO. Like a full time DevOps person went solo and spent a few weeks setting up their perfect deployment process. If we're talking a plain saas type deal, I'd keep it simple, elastic bean stalk, or use a heroku or render.com like setup until you grow to the point of hiring a team. If it's just a basic saas, I don't see how a 1 man team could really out grow this setup. I've seen 100 person teams using heroku. K8s is just way too much work. Even cloud formation is to much for my tiny show. Use the automated backups setup by your host for your db. If you need to roll back, just redeploy your previous git hash. I typically use GitHub actions to deploy, so rolling back is just a matter of force pushing the prod branch to the sha I want to deploy Skip micro services, they are too much work for a small time thing, and don't really provide much benefit.
• cblconfederate 5y ago
  So none of you edits the code directly in the production machine?
• jermaustin1 5y ago
  So I am a solo-preneur and have launched lots of web-apps in my career. And the DevOps hasn't really changed for me is incredibly simple: commit source control and copy the files to the server. I've done this since writing my first CGI app in 2003. And it hasn't changed other than the commands I call. Back then it was manual through GUIs (visual source safe and FTP) and now it is git and scp, but 80% of the process hasn't changed in almost 2 decades. If it breaks (which it shouldn't, but inevitably does), I just git checkout the previous commit and scp again, and try and fix the issue. If I have a "high availability" product (which I hate doing), I will release onto a clone of the production server, then test it, swap to that clone, deploy to the others, swap back and kill the clone. It is a much more involved process that I really don't like doing, so I've stopped making high availability applications. But that is web, so it's super simple. Also to avoid issues regarding data schemas, I always try to make sure my changes are additive. And on startup, my web server checks the database version then runs any upgrade scripts if needed. If the deploy fails, the db changes stay, and the code just gets updated. This is sloppy, but it's simple. When a real schema change needs to happen, I abandon the project (just kidding). I usually do it in multiple updates. New table with the schema changes, upgrade script transfers the data, and code to point to new table. Manually verify that all of the data copied. Backup the old table and drop it. Then another update to rename the new table to the old name (if needed) and it's done with minimal disruption in service.
• sentinel 5y ago
  My rule of thumb when working on side projects is to keep it simple: - use tech-stacks that I know - even simpler – use node.js / express (i.e. use only one programming language) - Standard AWS EC2, S3, Postgres instances - scale vertically before scaling horizontally - use GitHub Actions as much as possible - deploys and rollbacks are done manually - I keep a list of useful CLI commands in Notion If you're a small team you shouldn't need to think about kubernetes tbh.
• goodpoint 5y ago
  For 1-person projects you just need VPSes, git, and dpkg-buildpackage. A single makefile command is enough to build .deb files, rsync and install them. Rollbacks are trivial. Put configuration files as well in git and in the .deb git tags give you a history of releases and also deployments.
• m0llusk 5y ago
  Very carefully: Start with a very specific set of steps for security nodes and storing credentials. Then move on to backup and restore. All development and deployment details should have some kind of test to determine if the code or configuration are basically functional.
• todd3834 5y ago
  I think the key to getting a good answer here is the “beyond heroku” part. It is hard to make suggestions without knowing why a platform as a service no longer works. There are so many different options available but they very for different needs. Regardless, I would assume that you are going to have to pay more for a managed solution that managing everything yourself. I know the founders at https://releasehub.com/ https://releasehub.com/ and it sounds like it might hit that sweet spot for “outgrowing heroku” but “don’t want to do full time devops”.
• joshghent 5y ago
  As others have said, try to avoid any kind of DevOps if you can. But here's my strategy: Monitoring - Loads of cron jobs pinging slack (monitoring disk, CPU, Network I/O etc), Healthchecks.io, TurboAPI, Nginx Amplify. Mostly built up over time DB - Done entirely via migrations. Most of the time they are small migrations so I can do it by running the command directly on the instance Deployments - GitHub actions to build a docker image (there was a prebuilt template). Watchtower then runs on my server to pick up the new image and that's then deployed automatically. There is no rollback system as I've not needed that so far (and 99% of the time you won't). Backups - DB backup is done via a cron pushing to AWS S3 Glacier. The app code is stored on GitHub already so no need to back that up. Happy to share the script if you like :)
• Fiahil 5y ago
  Because everyone here is hell bent on "spooky arcane oldhat sysadmin deploy techniques", I'll share my setup using k8s. - CI on github actions - Project management on post-its (short) and READMEs (long term) - deployment is done by a github action trigger on the `main` branch - Hosting is using AKS, GKE, or on-prem k3s on a raspberry pi. Want to restart a service ? just kill the pods. - Devops took about 2 days of work initially and now shared by every project for less than 30 minutes by project. - Deloying a test cluster (or a test k3s node) is reproducible on whatever hardware / cloud provider you get, sometimes I often create a full-blown dummy cluster for running test scenario - Certificate renewal is automatic (cert-manager) - DNS record creation to services/ingress is automatic (ExternalDNS) - Authentication using OAuth/OIDC is also set up automatically on whatever identity provider you have - Database backup is automatic - Load Balancing is built-in - Job scheduling is built-in (and -fuck- logs are much more accessible on a failed job container than in a custom cron + sh solution) - Service discovery is not needed - Monitoring and alerting is not, but cloud providers often have something for you Note: this is highly effective because I already had significant K8s experience, so, if you're still learning about what Ingress-Controller to choose for your OIDC proxy, then don't go that route.
  • caffeine 5y ago
    Hey, this is pretty cool (I am doing the oldhat stuff, so interesting to hear about something different) >> Job scheduling is built-in (and -fuck- logs are much more accessible on a failed job container than in a custom cron + sh solution) Can you expand on this? How do you handle logs?
    • Fiahil 5y ago
      I don't handle anything, I just output to stdout and I can access any directly from its container. Job or Pod, it's the same. If I had a larger log volume, I would have some kind of log aggregator to send them to ES or cloud provider monitoring stack
  • techn00 5y ago
    How did you set up the DNS part?
• piettes 5y ago
  I might be biased here, as I started as a side project and wanted to learn k8s in the process. It turned out quite funny to set up and I am happy with the result, except for the struggle I had to add SSL. I have 1 frontend, 4 micro-services and 2 postgres. One cluster for test, one for production. I use Gitlab for CI, commits to master create a new docker image in the Gitlab registry. I manually trigger deployment on k8s with `kubectl rollout restart XXX`
• decidertm 5y ago
  I'm a technical co-founder at Northflank https://northflank.com https://northflank.com. A platform to deploy microservices, cron jobs and stateful databases from development to production. I started working on Northflank to make it more simple for developers either solo or in a team to manage and automate DevOps complexity away. We support CI for GitHub, Bitbucket & Gitlab (SaaS and self-hosted) with either Dockerfiles or Buildpacks. We have a robust and feature rich platform via UI, API and CLI. Out of the box you get end-to-end DevOps: CI/CD/CD, Horizontal and vertical scaling, persistent workloads with managed (MongoDB, Redis, Postgres, MySQL, Minio), backups, restores, real-time observability and metrics, DNS, TLS, mTLS, Teams & RBAC and more… Let me know what you think of our offering + site! Platform: https://northflank.com https://northflank.com Application Documentation: https://northflank.com/docs/v1/application/overview https://northflank.com/docs/v1/application/overview API Documentation: https://northflank.com/docs/v1/api/introduction https://northflank.com/docs/v1/api/introduction
  • aristofun 5y ago
    From the first glance it's not clear how is it different from heroku and what exact pain points are you trying to solve
• c0nrad 5y ago
  (Profitable Solo founder) Thankfully lots of tools do most of the heavy lifting. I use k8s, GKE does most of the work for me. It's very nice to have autoscaling for traffic spikes. Same with database (MongoDB Altas), dead simple autoscaling. I would never run my own k8s nor database. I wrote more details about some of the Ops stuff I do here in a previous similar question: https://news.ycombinator.com/item?id=26204402 https://news.ycombinator.com/item?id=26204402 Coincidentally I also wrote some architecture notes about a new product last night: https://blog.c0nrad.io/posts/slack-latex/ https://blog.c0nrad.io/posts/slack-latex/ I think everyone's milage will very, but as general principles, staging is nice, reading docs saves time overall, tests help you sleep at night and make it easier to make changes 6 months in the future, simple health checks (or anything on a critical path) help you catch the real issues that need immediate attention. Good luck!
• leros 5y ago
  I could run things myself on k8s or other self-managed systems but I have better things to spend my time on and it's not my expertise. I host my apps and databases in Heroku. Yes it's more expensive than alternatives, but it's easy and I trust their database backups. For everything else, I find managed services. Basically, I'm a developer and I let other companies handle most of my DevOps for me.
• bullen 5y ago
  I made my own HTTP app. server and JSON database on top of that. The server accepts .jars over HTTP with code (and files) so I can hotdeploy while developing on live on the entire cluster in real time. My turnaround is about 1 second. The JSON database allows for schema-less simplicity, and it has all the features you need like indexes, relations, multi crossref and security (and then some, like global realtime distributed while still being performant) in 2000 lines of code. I have zero pain developing the most scalable (and energy efficient) backend in the world, yet very few seem to care or use it: https://github.com/tinspin/rupy https://github.com/tinspin/rupy It has been proven on a real project with 5 years uptime and 350.000 users: https://store.steampowered.com/app/486310/Meadow/ https://store.steampowered.com/app/486310/Meadow/
  • programd 5y ago
    Does "Book of Travels" run on the same infrastructure? https://store.steampowered.com/app/1152340/Book_of_Travels/ https://store.steampowered.com/app/1152340/Book_of_Travels/ I know it had some server issues on launch - too popular it seems.
    • bullen 5y ago
      No, nor I or fuse had any involvement with BoT. I wouldn't post here if I was... BoT did not fail because it had too many players, you can clearly see how many are playing, maybe it has a complex backend... simplicity is your friend! Meadow for reference launched with ~150 (on T2 Micro at 30%) concurrent players per machine and peaked at 450 (on T2 Medium at 25%). That's before you could buy more credits so we had to upgrade to M4 Large after a while. And Meadow is an action game, not a point and click! The rupy chain is solid and proven and has been running Meadow for 5 years without any write downtime except last week when one instance got nuked by AWS or somebody at M&D. That's a pretty good track record of 100% read uptime! The Meadow launch probably was the only MMO to have a flawless launch with high pressure per server so far, and it did so over HTTP with a JSON file database! :D It's also the only MMO with one database spanning 3 continents in real-time.
• throw03172019 5y ago
  I use Convox. It’s great and has been stable the last 5 years for us at many different scale levels.
• strzibny 5y ago
  I strip it down to a bare minimum. Some people probably don't realize that you can have a full-featured git-push deployment including a PostgreSQL database and Redis on few hundred lines of simple Bash code. The only exception to keeping it bare is security (proper SSL, SELinux, etc.). - 1 cheap VM per project (I prefer Digital Ocean at this time) + snapshots - No CI (run tests locally), no staging until later - Little bit of Bash to configure everything, no IaC - Simple systemd services (+ maybe systemd socket activation) - git-push deploy... bad release? repush previous version - Automatic system updates and log rotation - Few auxiliary scripts for Rails console (./railsc.sh), backups (./backup.sh), etc. - External error and performance monitoring, but keeping raw logs on server (rarely need them) - Stable CentOS/Rocky Linux with long time support, rootless access I teach all of that in my book https://deploymentfromscratch.com/ https://deploymentfromscratch.com/ and I basically run my oldest side-project on the book demo which is similar to this. Some people might not like Bash, but it's surprisingly refreshing, and I keep my script flexible and idempotent: Set everything up after providing IP address and domain name in settings.sh: $ ./setup.sh Change the database configuration after changing the config file: $ ./setup.sh -u postgresql Deploy a new version: $ git push production master:master Later on I would separate the database and have staging, but it's overkill for my projects right now. The funny thing is that apart from having CI and staging we could have the same setup at 2 early stage startups I worked on...and it would probably last 3-5 years with maybe an in-place server spec upgrade, no kidding. People really like to overdo operations and you know where overdoing it leads to? Mistakes. You maintain K8s and then forget to do something basic security-wise that almost kill the company (seen).
• dangerard 5y ago
  I replicated Anthony Simon's architecture [0] in a few days and it has worked beautifully so far with a toy project [1]. Probably overkill for OP, but the combination of GH Actions & Flux [2] means that starting (and killing) namespaced projects – with all the cluster benefits of automatic TLS certification, DNS record management, rate limiting, pod restarts, logging, autoscaling, CD, etc. – is a single push to a git repository. Really nice if you're taking a shotgun approach and trying different ideas quickly. [0] https://anthonynsimon.com/blog/one-man-saas-architecture/ https://anthonynsimon.com/blog/one-man-saas-architecture/ [1] https://www.wyzetickets.com/ https://www.wyzetickets.com/ [2] https://fluxcd.io/ https://fluxcd.io/
• waspight 5y ago
  I dont think that a hosted k8s with a hosted db on, say DO, would require a full time job? I can imagine it seems that way if one is coming from a large 1k+ employee company though. I always wonder how those companies can utilize so many people.
• alberth 5y ago
  I'll get downvoted for this but use Dreamhost (if you're just starting out). Both their Shared and Dedicated servers are fully managed. You can run PHP, NodeJS, etc on it. They also fully manage MySQL too. All you have to do is drop your code onto their servers and it just runs. It's also hard to beat their pricing too.
• trhr 5y ago
  I made the decision early on that a best practices CI/CD setup wasn't a time waster, but an integral part of building robust software.
• trhr 5y ago
  I made the decision before starting that a best practices CI/CD setup wasn't a time waster, but an integral part of building robust software. Deployed into two off-lease dell servers; one for staging and one for production; via docker-compose.
• 0xbadcafebee 5y ago
  Hi OP and everyone else, DevOps is not a synonym for "operations work" or "server work" or "devs doing operations work". DevOps literally means "dev teams and ops teams working together". That's all. You don't "do DevOps" unless you are "doing the work of collaborating between two different teams of people".
• jeffbarg 5y ago
  We use https://porter.run https://porter.run. Pretty great.
• wenbin 5y ago
  Solo founder here. (Typed on my phone) I’ve been running my small company for 4 years so far. It’s a podcast search engine & api. Product & infra & Devop & all kinds of manual processes all evolve over the past 4 years. Things were added / improved on demand. 4 years ago: 3 digital ocean instances + ssh to manually deploy code. No users. So no devops. Now: - ~20 EC2 instances, provisioned via ansible - to deploy code: run a bash script on my MacBook, which is actually to run ansible. The script can also rollback to any specific git sha - use rollbar + Datadog + PagerDuty for monitoring & alerting - 3 Postgres database EC2 instances (1 master, 2 slaves). Daily cron job to automatically upload db dump to aws s3 and google cloud storage - no CI. No docker / kubernatte - a few interesting devop events are sent to slack, so I’m aware of what’s going on for the entire system on my phone. Eg, when a db dump was successfully uploaded to s3, when a Django request is slower than 5 seconds, when an important cron job fails… Not very sophisticated, right? But in practice, we rarely have outages and spend very little time to do devops. The most annoying thing was to upgrade Postgres to a new version. We had to prepare for an entire week and minimize downtime (in practice, disable db write for < 1 minute or so). I’ve got a couple old blog posts on engineering for solo founder: - https://www.listennotes.com/blog/the-boring-technology-behind-a-one-person-23/ https://www.listennotes.com/blog/the-boring-technology-behin... - https://www.listennotes.com/blog/good-enough-engineering-to-start-an-internet-27/ https://www.listennotes.com/blog/good-enough-engineering-to-...
  • vladfr 5y ago
    Is Datadog worth it? I thought it was too expensive, although they do have a lot of features.
• bdcravens 5y ago
  ECS gives you pretty much everything K8S gives you, in a fully managed package. Obviously you can't (simply) run ECS in development, but sticking with Docker Compose keeps the semantics close enough, as well as giving you advantage of clean separation from any other code on your machine. CI/Deployment: hosted CI, merged PRs to main branch auto-deploy (I've used both Codeship and Github Actions with ECS) DB: managed RDS
  • frankwiles 5y ago
    Keep in mind you can run EKS in aws to get K8S in a fully managed package as well.
    • bdcravens 5y ago
      My understanding is that with EKS, you still are responsible for parts of the process that are managed under ECS (for example, firewall and security rules as opposed to leaning on ALB and Target Groups in ECS)
• nullbytesmatter 5y ago
  I run a solo SaaS product with 15,000+ users. I use the "arcane/wizardry" techniques still. I simply run "make deploy" and a new production build is made and rsync'd to production. Then the script executes a "deploy.sh" on the production server which reloads the system process to pick up the new binary. Poof.
  • all_usernames 5y ago
    And if your singular production server catches on fire?
    • nullbytesmatter 5y ago
      Has been five years and never happened. I don't optimize for problems I don't have. I do daily backups and have the ability to deploy a "new" server if need be. It's a little bit of manual work but would be trivial.
• nickspicer1993 5y ago
  My entire stack is in a docker-compose.yml that looks like this: services: certbot haproxy db api client And gets deployed by a GitLab config file that looks like this: build: stage: build script: - docker-compose -f docker-compose.yml -f docker-compose.release.yml up -d --remove-orphans --build
• deleted 5y ago
  [deleted]
• Amasuriel 5y ago
  I recently started a small project on Azure for something new, but most of my experience is small companies on AWS which is similar. If you aren’t already comfortable with K8 I wouldn’t use them, there is a learning curve there for sure and not necessary. I wouldn’t even worry about containers at all unless you are planning on using a managed container service like elasticbeanstalk. My CI pipeline is almost nothing, just run tests on main and dev commits. I have no deploy pipeline, I choose to actively deploy. My deployment workflow is a powershell script since I’m on Azure, if it was AWS and I was using plain EC2 instead of a container service I would probably use ansible to avoid having to repeat commands 2-3 times when I deploy something behind a load balancer, but it would be super minimal. DB backups are super easy to schedule with RDS or similar offerings. For rollback with database changes I would schedule downtime, let your customers know, replace your page with a maintenance banner, do upgrade, test upgrade then repoint site to your running instance. Overall I echo what many have said, use familiar tools as much as you can, don’t worry about whether what you are doing will scale to even 2-3 people if it costs significant time. Use managed services where possible to save you time, usually the cost isn’t high vs doing it yourself.
• deepsun 5y ago
  I use complex setups of K8S (on Google Cloud, it's simpler there), proper security, terraform etc, because I already did it before, and it as easy for me to just reuse my configs/scripts.
• bithavoc 5y ago
  I'm afraid after Heroku is not enough for you, if you don't use Kubernetes, you may end up inventing one. If you need a bit more than Heroku: 1. Try Render[0]. 2. If it is not enough, go with any Kubernetes you can afford, any docker registry and Pulumi or Pulumi+Helm[1]. For monitoring you can use either NewRelic[2] or Weave Cloud[3]. 3. If you find big cloud kubernetes too expensive, need more bandwidth, or are just willing to try more affordable alternative new players, maybe Civo[4] fits the need. In my case[5], since I've been bootstrapping for three years, a low-budget setup is key to keeping the lights on. Here are a few constraints I need to satisfy permanently-ish: - My workload is bandwidth-intensive, so EC2 egress is too expensive for my case and my situation. - It is not that cheap to run 12 micro-services in any of the existing Heroku-ish platforms, also private networking and routing of my app are a bit above "12 factor" at this point. - I can't afford the same availability as everyone else: I'm in the business of supporting other apps but I don't want to pass down the cost of AWS, so I can't afford to be "down because everyone else in AWS is down". - I don't know many things, and until I find PMF, keeping a low-cost setup gives me the resilience that I need as a founder. Therefore to solve my problem I use an unholy combination of Ansible and Pulumi to manage 3 clusters of Kubernetes for both stateful and stateless workloads. Why am I doing this to myself? I have mad respect for people who can run things in a VPS, use the hosting daily backup, and don't see the need to apply patches because they're behind Cloudflare; I just can't. More details: - Dedicated servers, 32GB RAM, 8 Cores, 500GB SSD(RAID 1), /29 ipv4, 30TB of Bandwidth for $70/mo each. XCP-NG[6] as hypervisor. - To assign public IP to virtual machines, I run a tiny DHCP ubuntu VM managed by Ansible. - On top of XCP virtual machines, k3s cluster managed with Ansible. - To manage XCP and virtual machines, I use Xen-Orchestra[7] managed with Ansible. - Pulumi+AWS ECR builds and pushes docker images, secret and configuration management using paraphrase secret provider. - Hashicorp Vault backed by Postgres+AWS KMS managed by Ansible. - Ansible+Pulumi+Helm to self-host Apache Pulsar with ZFS snapshots, a cron-job and rsync.net alerts. WAL-G+AWS S3 to backup and archive Postgres/TimescaleDB managed with Ansible. - All logs aggregated by Gelf and Fluentbit into a self-hosted instance of Seq[8] managed with Pulumi+Helm. - Tailscale between all virtual machines - DNSimple for domains - Cloudflare for all things except the self-hosted docker registry. - Github Actions for CI but not a hard a requirement, I can run ops in my local computer when Github Actions is down. Cost: $210 /mo. Estimated cost in AWS: $2,800 /mo [0] https://render.com https://render.com [1] https://newrelic.com/platform https://newrelic.com/platform [2] https://www.weave.works/product/cloud/ https://www.weave.works/product/cloud/ [3] https://www.pulumi.com/docs/reference/pkg/kubernetes/helm/v3/chart/ https://www.pulumi.com/docs/reference/pkg/kubernetes/helm/v3... [4] https://www.civo.com/pricing https://www.civo.com/pricing [5] https://widemesh.com https://widemesh.com [6] https://xcp-ng.org https://xcp-ng.org [7] https://xen-orchestra.com https://xen-orchestra.com [8] https://datalust.co/seq https://datalust.co/seq
• more_corn 5y ago
  In what specific ways are you beyond Heroku? I call bullshit on that. I've got a big client who went from 6 infrastructure engineers to zero. They survived for half a year without any real problem. You either pay a company to manage your infrastructure for you or you pay a person. Devops people are expensive. Things I've done to automate DevOps Make sure you pay attention to automatic testing and deploys with Ci/CD. I like automated deploys on Gitlab. Trigger staging release on merge into main, and deploy prod on merge from main into a deploy branch. Infrastructure as code with Terraform. K8S is more trouble than it is worth (though it is robust after you invest 3 months into getting it tuned right, although a lot of people make critical mistakes in their config, I've seen a half dozen insane mistakes.)
• balabaster 5y ago
  It depends very much what your tech stack looks like, but I use Git, CircleCI and Octopus Deploy to Azure for my stack. It's cheap in terms of maintenance and because I know them all like the back of my hand it's quick and easy for me to set up. As for rollback, I don't really use rollback. I have a roll-forward strategy. My DB deployments are migrations and so long as there's no data-loss caused by a migration (my migrations are always backwards compatible with the previous version) there's no need to roll back. Azure brings point-in-time restore which can be triggered if necessary. I think the key is like with anything. There's a learning curve on your toolset. Once you've overcome that hurdle and you know to set up automation for everything from the outset. You build your pipeline shell. You never do anything manually. If you need infrastructure of any form, it's always code-first, included in source control and done the right way. Never do anything manually and rely on your automation to carry you. It seems labour intensive up front, but once you're there, it's sustainable. It takes discipline not to cut corners. The minute you start to cut corners and get lazy is the path leading to your doom.
• christophilus 5y ago
  I lead a team of 3. We use render.com and have a handful of bare metal servers for CPU-intensive background jobs. The bare metal servers are auto-patched Ubuntu servers. It’s pretty much 0 maintenance. The server setup is automated via a single provisioning bash script. Render.com is quite nice, if a bit limited in its options vs Heroku. Highly recommended.
• RapperWhoMadeIt 5y ago
  I have this friend who recently founded a start-up with two other guys. But he basically is the only one doing any DevOps, so we could say he is a one-man show regarding the technology side of things. Two weeks ago we talked about how he is developing his product and he told me he will go full FaaS as much as possible, because it lets him forget about the whole Ops side of things and focus in real product development (what will ultimately bring revenue to the company). I am a big fan of playing as much sysadmin as possible for my own developments, and I like the CI/CD mentality associated with microservices and containers. All I am saying is that that conversation with that guy really opened my eyes to the fact that maybe in some scenarios (specially at the birth of a company) going full serverless might improve the time to market of your product. I am in no way saying that serverless also has a ton of disadvantages, e. g. vendor lock in.
• oneplane 5y ago
  "It depends"... For simple monoliths it's a non-CI method plus scp, for infrastructure-only it's SaltStack and Terraform. For something where the DevOps part is the deliverable, it can go wild pretty fast: GitLab CI with various CI steps (static analysis, building, unit tests, integration tests, browser tests, blob storage of the result), then CD steps depend on what it is that was built; docker images just get pushed to a registry and another async job listens for pushes and does the deploy in Fargate or Kubernetes. For some jobs with larger needs you get into ArgoCD and Kubernetes resource deployments that way. Essentially it depends on the zoom level: Zoomed in to the max: just a build.sh or makefile, optionally called by a CI or IDE of your choice so the building is exactly the same no matter what invoked it. Zoomed out a little: the results of the build need to end up somewhere, sometimes a step in between like a package registry, docker image registry etc. Zoomed out to level 3: once the deployment gets actually done, the runtime itself is responsible for checking for database migrations, locks, maintenance mode, so that has to be part of the startup. If it's a multi-instance workload it depends on database migrations if there has to be downtime. Zoomed out to level 4: if it also needs to deploy 'other things', like rules and records to Cloudflare, resources in AWS, resources in K8S and even update some stuff in buckets or DynamoDB, that requires some orchestration and DSL like Terraform, SaltStack, Ansible and application-specific CD like ArgoCD or Flux or even just the aws cli tools to trigger a refresh. Customer decides how much money they want to spend and that scopes what zoom level they get. Usually depends on how many developers or ops people they have themselves and how involved they want to be and what continuity or guarantees they are looking for. The only thing that is not optional is that everything has to be in Git. If a customer doesn't want that, they cannot be my customer. I'm done living in the 90's.
• marcrosoft 5y ago
  Servicing over 1MM API requests / week using API gateway and lambda. It’s been set and forget for 6 years or so. AWS lambda is incredibly maintenance free if your use case fits the lambda constraints.
  • aristofun 5y ago
    What about costs?
• dyeje 5y ago
  I would never switch from a PaaS unless I was expanding the team beyond just myself. Working on DevOps brings 0 value to my users.
• maxk42 5y ago
  Scaled multiple self-hosted products up to tens of millions of hits / month. Scaled several for clients up as far as the hundreds of billions of hits per month. (Yes, hundreds of billions.) Never seen a kubernetes deployment that was (A) necessary or (B) simpler than some alternative. Unless you're doing something very slow / processor-intensive, you'll probably never even need an autoscaler before you can afford to hire an expert to do it for you. You'd be surprised how far you can get with a small VM running your whole stack.
• tomcooks 5y ago
  Whenever I do anything on a VPS I religiously write down each step in a README file, so that then I can create bash scripts and automate processes. The more I do this, the more I learn, the more I automate, the less I worry.
• domk 5y ago
  Don't. Use a PaaS like Heroku, hook it up to your GitHub and a CI platform. Get your database hosted on Heroku or somewhere similar - plenty exists for anything you might need (Postgres, MySQL, Redis, Mongo...). Don't do things that increase your operational overhead like using microservices or running unusual databases for which a good hosted solution doesn't exist. Stay away from anything k8s. It will be a bit more expensive but well worth the extra money, compared to the time you'd spend on operations.
• barelysapient 5y ago
  Keep it stupid simple. Use tech that you understand and minimize the layers of complexity between deploying, starting and troubleshooting an instance. For me, I like digital ocean droplet(s). Stupid simple IAM. SSH scripts for CI deployments. Everything sits behind a Cloudflare cache. The backend sends key events like restarts and fatal errors into a dedicated Slack channel. Its my confirmation a deployment worked and first alert if something crashed. My business partner and I are currently running a $1m/arr business with this setup on a single digital ocean instance, Python3 and Nuxtjs.
  • whitepoplar 5y ago
    100% this! Just curious--what's your business?
    • barelysapient 5y ago
      We're in the hospitality sector.
• PaywallBuster 5y ago
  probably deploy infra manually (ec2+rds or vps at random provider) Deployments can be automated I usually work with PHP, so I already have playbooks to configure/install php/nginx and others (just change few variables) deployment only needs to add a few steps like git checkout ( easy to reuse since you're probably using the same framework across projects) some examples here https://github.com/MalpraveCorp/ansible https://github.com/MalpraveCorp/ansible in specific https://github.com/MalpraveCorp/ansible/blob/master/mal-be-p.yml https://github.com/MalpraveCorp/ansible/blob/master/mal-be-p...
• mfrye0 5y ago
  I agree with most people here. Definitely keep it simple. With that said, I run a semi resource intensive operation, so I've invested a bit into dev ops to keep our costs down. My setup is currently on AWS, primarily using ECS, RDS, Elasticache. Infra is managed via Terraform. I felt ECS was a nice balance vs K8s, as it's much simpler to manage, while getting the benefit of maximizing resource utilization. For CI / deployment, I use Github Actions to build an image and push, then start a rolling refresh to update the containers to the new version. It was pretty easy to setup. On DBs, RDS handles all the backups and maintenance. For migrations, I use https://github.com/amacneil/dbmate https://github.com/amacneil/dbmate. Happy to answer any other questions you have, as I've learned a lot through trial and error.
• blablablub 5y ago
  For a couple of persons team it is important to keep the changes small, so errors can easily identified. That means daily or bi-daily upgrades. It also helps if you focus on a certain small project and finish that, before moving on to the next project. SVN checkout development, meld into testing. Copy production db into testing. Run a db upgrade script in testing. Test the new function for a day. If all ok stop the production HAproxy at 3am. run production DB backup, Checkout testing into the production server. mount the new dir into the NGINX path. Upgrade+start the DB, start production HAproxy. Except for the meld and testing part, this is all automated. A rollback on the app servers is pretty simple, just remount to the previous location. Even though there is a backup of the DB, a rollback there is not really feasible. Everything is running in LXC containers, which I treat as cattle. Creating new ones is either automatic based on time/demand/failover or a few clicks in a custom web interface. HAproxy automatically pics them all up via DNS. SVN and production DB are always backed up nightly and copied to a S3 compatible storage. Everything older than 6 months gets deleted. All the free space on the servers is globbed up with glusterfs to give a decent amount of fast,free storage. Using Bash as scripting language, there is literary no problem in devops which hasn't been solved with bash and the solution is somewhere in the internet.
• bredren 5y ago
  Solo founder, sold my last SaaS I had set up using "spooky arcane oldhat sysadmin deploy techniques" h/t to @Fiahil for that shade. On that it was a Django product with deploy scripts for staging and production, i.e. `./deploy_production` ran: #!/bin/bash ssh web@[production ip] << 'ENDSSH' cd /home/web/ ./pg.sh ENDSSH Where `pg.sh` pulled from `master` ran static and migration management commands and restarted the web server. Last summer I moved away from the spookies and now do all deployment using Github Actions. Commits to either a feature branch or `develop` run tests and all necessary deployment steps to a staging instance. (my stuff so far does not require k8s) and all containers are put up using docker compose. Commits to `main` (or `master` depending on the project) I blended a few guides written by Michael Herman to build my GA-based CI, and it took several weeks to work on the scripts and learn Github Actions. But it was time absolutely worth investing because my DevOps is modern enough now. No more bare metal stuff. No more scripts. Nice visual display of deployment success, "free" VMs for the deployment and CI UI integrated alongside my issues and code all on Github. Importantly, I still use git flow. I tried moving to trunk based, but the macros inherint in git flow (including the pycharm plugin) make this production / staging // master / devops or feature/ work very well (I'm not making PRs to myself)
• FpUser 5y ago
  Here is one particular but typical example - enterprise backend server written by yours truly, PostgreSQL and single page frontend application in Javascript. I do it for my own company and also develop in a same manner for my clients. 1) Server is high performance native C++ application running on rented native hardware on Linux. It processes thousands of requests/s in sustained manner hence no real need to use any of that k8s / docker / etc. There is also separate rented standby server. 2) I maintain a reusable single script that can rebuild complete system from the scratch with a single command. I periodically test it on local VM. It installs all needed compilers, tools, PostgreSQL, restores database from the latest backup, checks out from VCS and builds that C++ server, registers it as a daemon and refreshes Javascript Frontend. 3) I also wrote small server that subscribes to webhooks from VCS and can build artifacts. I only use it to pull and build Javascript part. It could do main server as well but being super cautious I trigger that step manually fingers crossed. 3) for DB I use automatic scheduled backups. All in all after I've debugged and tested new version locally the release takes seconds. Script is written years ago and does not require much maintenance other then updating artifacts to new versions (replace apt-get install XXX-31 with apt-get install XXX-32) and register and build ZZZ server instead of YYY server. Compared to insanity I saw in some orgs my setups are piece of cake.
• e2e4 5y ago
  Caprover on a decent VPS like Hetzner. p.s. previously did quite a bit of AWS, Google, firebase; but the above had been working the best for about a year
• deleted 5y ago
  [deleted]
• b20000 5y ago
  this is why everytime i post here on HN that i want to make my toolset smaller, and i’m not interested in learning more tools. i catch a lot of flak, but i’ve been in this boat now for more than 15 years and I can say that if you need to manage everything with 1 or 2 people, you need to approach this differently than what a normal startup does especially if you are not able to raise funding.
• rolisz 5y ago
  For a recent small project, I tried out deta.sh. It takes out the pain of devops entirely, but it does constrain you on the architecture and database side of things.
• divan 5y ago
  I try to use simpler/friendlier tools, so it can be managed with a simple SSH scripts for as long as possible. Use static binaries whenever possible. My typical stack is Go+gRPC+Flutter – so it's really easy to ship it as a single binary (for the web builds). For the DB I also started to use EdgeDB on some projects and it's such a relief in term of migrations.
• aranchelk 5y ago
  My strategy: get really clear on business needs/objectives/internal SLAs for infrastructure. In my experience a lot of time-consuming stuff revolves around uptime, data recovery (RTO, RPO), security, compliance, and scaling. Depending on your business and the stage you’re in, a lot of that may not matter, so don’t build anything you don’t need now. I’m a solo-founder, with many years working in IT, and I focused on DevOps for part of it; I know what best practice looks like and it would be easy to fall down that rabbit hole doing an unnecessarily complex buildout. Currently I’m doing a small private beta, I’m avoiding 99% of standard practice. I’ve got a single EC2 node running Redis as the only datastore and NodeJS + Nginx + certbot, a cron to do backups to S3. No CI (there are no other devs to integrate code with) I run all tests locally and push/rollback with rsync. All code, assets, and server config (except creds) are in a monorepo. If the server goes offline, I will have a little downtime, that’s fine. If I run out of memory for Redis (not likely to happen soon), I’ll change to a different datastore or scale up the node. If I lose data, I can restore from S3, and additionally the architecture is such that clients will re-push their latest changes. Do the bare minimum to support the business, stick with what you know, outsource what you can, and properly value your time.
• welder 5y ago
  I'm a solo founder with a product used by 300k+ people. Keep it simple, don't solve problems before they become problems, and keep all config in Git. I would use Heroku, but my usage requirements makes that cost-prohibitive. Heroku is good for low-traffic projects aka Enterprise B2B. I don't use k8s, I just use shell scripts that import each other for setting up new DigitalOcean servers. I separate my shell scripts into 3 categories: 1. common (Google leap second smearing, increase max open connections, misc security and performance changes) 2. infra (install Nginx, install RabbitMQ, install Postgres, install SSDB) 3. product (run common stuff, then infra stuff, then any stuff specific to a single server) Sometimes it breaks after upgrading Linux, but it's easy to fix. For CI, I use GitHub Actions. Deployments are mostly built with GH Actions. If deployment fails it just keeps the old server code, there is no need for rollbacks I just re-deploy with working code. If deployment passes but there's still a prod bug, I don't rollback. I just commit a revert and deploy that or commit a fix. LetsEncrypt renewal, log aggregation, db backups is all custom with combination of RabbitMQ and CRON + Shell scripts. I miss out on autoscaling, so someday I'll invest the time to migrate web servers to hosted k8s. With limited time as a solo-founder I have to prioritize for impact meaning incremental improvements usually never get done. Not having k8s and autoscaling isn't causing any problems, won't increase revenue, and won't save any meaningful expenses so it's currently only an incremental improvement for me. Finally, I have a small Slack community for bootstrapped founders. Let me know if you want to chat in there about specifics.
• vikR0001 5y ago
  I'm a javascript developer. My build tool is Meteor and I host on Galaxy. Deployment is one line of code in the terminal. Scaling is handled automatically. :)
• SergeAx 5y ago
  First of all: no microsrvices, no Kubernetes. The whole business logic is a two binary (server + cli) monolith in a single container image. For CI GitLab is a bliss. Lots of examples and articles around the web. I made a test-build-dockerize-tag pipeline once and now just taking it from one project to another with minimal changes. For deploy - Docker Compose is enough. Maybe Docker Swarm some time in the future. Gitlab hosts docker images, rollback is a simple change of container version. For Postgres backups I am using WAL archive + daily snapshots from replica.
• redis_mlc 5y ago
  You can just use EC2 with yum update and RDS to do 90% of what you mentioned.
• kureikain 5y ago
  I divided my infrastructure into two part. 1. Tricky and unfrequent changes: - mail server, load balancer, database etc: for this I just write bootstrap.sh, rsync to server and run. - It's literally just a for server in `foor bar`; scp ; ssh ` done 2. Deployment/Frequent changes: - docker-compose to spin up everything. It's super nice. Again, the deployment is done with a `rsync` then `docker-compose up -f docker-compose-prod.yml` Eventually when deployment changes very frequent and need scale/ha I added in Kubernetes. K8S is way easiser to setup than you think and it handle all other suff(load balancer, environment variable etc). And my deploy now become: `kubectl apply -f` One trick I used is to use `sed` or `envsubst` to replace the image hash. For backedup, I again, literally setup cronjob from an external server, `ssh` into database and run `pgdump`. I also have a nice NFS server to centralize config and sync back to our git repo. I used this whole setup to operate https://hanami.run https://hanami.run an email forwarding service for the first 3 months before I added Kubernetes.
• theptip 5y ago
  A lot of good answers here to the actual question in the OP, of "how do you do devops". And while it's good to look at other stacks for inspiration, particularly as a solo/small-team founder, the "correct choice" such as there is one, is in my opinion to do what you can do quickly/efficiently with your current skillset. I'd say the meta-level answer here should be to timebox your investment based on the tools you know, and an honest assessment of your uptime requirements. Also, know that what feels like a janky solution will usually get you much further than you think it will. For an initial PoC you should probably not be spending more than an hour or two setting up your infra, so Heroku or a single VM is probably the way to go. But you still _might_ want to start with k8s if it's the tool you know best; you can set up a k8s cluster in GKE with about as many clicks as it takes to set up a single node, and if you have a full "django k8s yaml" or equivalent in your toolkit then you could be up and running in an hour or two from scratch. I wouldn't recommend this to most folks though. Once you have some traction, beyond PoC stage, then you should be thinking about failure modes, their impact to your business, and how much time it's worth to spend to avoid them. If your users won't notice a few hours a week of downtime, then you don't need a multi-node setup like k8s or nodes behind a LB; a single node should suffice. If you have simple user data (like say comments) where in the worst-case losing everything since last night's backup would not sink your business, then running your DB with no standby replica and nightly backups might suffice. If you're storing financial transactions that's obviously not an option and your first DB with customer data needs replication (or however you want to get to RP0). Given all that, here are my suggestions for your specific questions: For CI/CD I'd say don't bother with CD, just set up Github/Gitlab to run your UTs on your PRs and on merges to master. Honestly you can just run UTs manually while building at PoC stage, but when you have paying customers I'd say you should take the 15 mins to wire up simple UTs. Getting a full Continuous Delivery pipeline set up won't save you much time vs. just manually deploying your artifacts, and at the beginning your tests are unlikely to be good enough to continuously deploy safely. At the point where you start having quality issues that your customers are complaining about, you might want to push for CD as a discipline thing; it forces you to write good-enough tests that you're happy to deploy if the tests are green. But it's not mandatory to deploy everything if your tests are good enough to do so. For rollback on your stateless instances, I like the pattern of having your single-node running your apps in a Docker container (I had one for Django and one for Nginx), that way the upgrade/downgrade is simply `docker pull <newimage>; docker stop <old image>; docker run <new image>`, and rollback is the inverse. But as long as you're just upgrading code and not OS dependencies, then you can do an equivalent operation by swapping out your code artifacts, e.g. checking out the new version of your python app & restarting your WSGI workers. For running your DB, I see a few folks suggesting running your DB locally on your node -- if you are an experienced DB operator that might be a good option, but if you're not, I would personally recommend using whatever managed SQL offering your cloud provider has; RDS or Cloud SQL will give you a durable instance with backups & easy restore with a few clicks in the UI, which is hard to beat in terms of ROI. Possibly-controversial recommendation: in general unless you've spent a lot of time working with Terraform/Ansible/etc. I would recommend against using infrastructure-as-code in the early days. It's hard to beat the ROI of a couple clicks in your cloud provider's UI to get your infrastructure set up, you might spend 10-100x more time getting your repeatable build implemented, even if it's just a couple mins vs. a couple hours. Clicking in the UI is not reproducible, but it will get you further than you think. As above though, if you're fluent in these tools and the difference in time-to-provision is very small, then absolutely do this now instead of later. But don't feel like you MUST do it "the right way" from the beginning.
• sharps_xp 5y ago
  As a thought exercise, I thought about how I would architect a web application as a 1 man show with a relatively small amount of users. I tried to optimize for cost and very little devops at all. I still mull about it but so far, here is what my architecture would look like. - no servers to manage - static website hosted on CDN, and even dynamic user-specific pages would be built statically every 8 minutes - all writes go to an SQS queue - every 8 minutes, a lambda is spun up to batch read the SQS queue and all writes go through a single writer process that writes to a SQLite file on EFS - every 8 minutes, a new static version of the site is built using the above hopefully updated SQLite file The one thing I hate about this is that I can't in good conscious say to the user that their write request succeeded b/c no state has actually changed on the backend yet. but I might be okay with this tradeoff and just simply showing only them their most recent write. The other tradeoff is when things go wrong, it'll probably go terribly wrong.
• rajacombinator 5y ago
  The whole point of devops or any other devthing is efficiency. If doing some devops stuff will increase your efficiency then do it. But you probably just need to find someone who will pay for something.
• cgarethc 5y ago
  I wrote a series of Medium stories about my experience with tech stacks for solo work - including deployment (start here: https://medium.com/geekculture/surely-building-software-cant-be-as-hard-as-it-used-to-be-5155341bae52 https://medium.com/geekculture/surely-building-software-cant...). My needs are slightly different to OP, because I'm not assuming everything will have to scale, so I'm optimising for "scale to zero" (only pay for what I use) and low maintenance... but I'm taking approaches that I know will scale if they have to (scale technically, not scale to have a team working on it). In short, I lean heavily on serverless and Github Actions. Each story covers a different pattern in my stack and I apply these principles: -I only want to pay for what I use -I don’t have a lot of time available for learning or building -I don’t have time for maintenance activities -I’m not a good UI designer or front end engineer
• bckygldstn 5y ago
  For my API SaaS I run everything non-critical (marketing website, analytics, customer dashboard) on a single server, either part of a single django monolith or using an official docker image with some command-line arguments. Only the API gets duplicated and load-balanced with Cloudflare. Dedicated servers, specced to provide lots of capacity for spikes. A 20-node k8s cluster could fit on 2 beefy dedicated servers for about the same cost. Decreased infrastructure redundancy but massively increased operational stability through simplicity. Everything runs in a docker-compose project: one for the API, another for the everything else monolith. I've worked for a couple of small companies that ran with docker-compose, so have a good sense of the weaknesses and footguns (breaking your firewall, log rotation, handling secrets, etc). CI is running `make test` on my dev machine. Deployment is `git pull && docker-compose up --build`. Everything sits behind haproxy or nginx which is set to hold and retry requests while the backend is down, so there aren't any failed requests in the few seconds a deployment takes, just increased latency. I only deploy to the API once or so per week, that stability reduces headaches. DB backups are done with cron: every hour a pgdump is encrypted then uploaded to backblaze. Customer subscription data is mirrored from stripe anyway so an out of date DB backup isn't the end of the world. Error if the backup fails after a few retries. Sentry on everything for error alerting. All logs go into New Relic.
• jokethrowaway 5y ago
  Deployments: GitHub ci + docker stack + docker swarm + docker machine + docker-flow-proxy + docker registry There are a few rough edges you face once and then forget about but minimal resource impact and it was fairly simple. I have another project that is using hacky scripts and it's a pain when something is off or I need to run something else on top of the machine. Some stuff is not terribly maintained but everything works. If you're small, it's definitely not a full time job. I didn't pick k8s because I initially deployed on a single machine and didn't want overhead. Things kept working afterward and I'm reconsidering whether I need k8s at all. Rollbacks are just deploying an old version + optional down migrations if needed. I try to minimise them and fix the code and redeploy anyway. For migrations, I guess it's very application specific, but most tools in most languages are fine these days. I use typeorm, a node.js library. There are some practices you can follow that you probably heard working in some normal company. Eg. I don't break the db structure immediately, I deprecate columns and then delete them after a while.
• jpgvm 5y ago
  k3s on single VPS. cert-manager and external-dns for maintenance free LE certs. nginx ingress (I would probably go Contour+Envoy if I was to set it up now). Kotlin w/Gradle. Build + Deploy single command `gradle deployStage`. jib for container images, don't need Docker, reproducible super fast builds. Prometheus + Grafana for metrics and alerting. Things I like about it: - JVM is super low maintenance runtime, tune it once and pretty much good to go. Very easy to diagnose memory or performance problems in production. I even have the debugger agent loaded in prod listen on localhost so I can use kubectl port-forward to access it and diagnose bugs that only occur in production or hung threads etc. - Prometheus seems like overkill but it's just so much less effort because everything already has a prom exporter. - Using k8s API saves a bunch of time again. No need to screw around with docker-compose or some godawful config management from the 2000s like Ansible (ick). Things I don't like: - It's not HA. The DB pods push their WALs to S3 and I have script to spawn a new PG from S3 using PITR. This is how I test new code so it's known working but it still means that if my server goes down it's down until I do something about it. - The box is a bit snowflaky. I didn't set it up using a script, it was hand configured up until the k8s level. - I rarely upgrade k3s because I have to do it manually. If I was using EKS or GKE I would probably press the upgrade button more often. - Grafana is a chore. grafonnet helps (I also use Tanka w/jsonnet for k8s manifests).
• smackeyacky 5y ago
  A short list of things that save me time: Databases: use AuroraDB on AWS. Pros: no server maintenance, no backups, no maintenance at all really. Automatic snapshopts set to 30 days. I back it up once a week to an S3 bucket for longer term storage. That S3 bucket is "aws s3 sync" to a local drive on a debian machine that has a tape drive in it. I do the tape backups just after the weekly DB backup. Code deployment: For static assets I use GitHub actions. For the containerised servers I haven't migrated to GitHub actions yet (.NET Core app running in an Ubuntu container) but since the system is usually deployed with the AWS docker tools that should be easy to automate. Lambda functions just use the AWS dotnet command line tools. This is all much easier when you use a debian machine as your main dev/integration box. You can use WSL2 as a reasonable substitute. The command line AWS and dotnet tools are much nicer in a unix shell as you can do things like use a Makefile or a shell script to capture the more complicated actions. PowerShell is an abomination.
• deleted 5y ago
  [deleted]
• sillycube 5y ago
  Solo founder working on 2 Shopify apps. I tried to keep my setup simple with almost no build process: 1. 1 dev VM, 2 prod VM. ~$30 / m 2. Python / Django apps hosted with docker compose 3. Git pull to prod, just reload gunicorn to deploy most of the time unless I need to add new python libraries 4. Auto backup db daily with a docker container. Weekly backup by digital ocean No k8s for sure.
• viferga 5y ago
  I try to avoid any complicated tool and simplify my life with NoOps tools. Using Kubernetes or AWS from scratch is probably going to kill your startup. In my case, I have tried MetaCall: https://metacall.io https://metacall.io
• davidbanham 5y ago
  I run three different saas products as a solo. Here’s some things I’ve picked up: Some things can be simpler when there’s only one of you. Take those opportunities when they arise. You don’t need remote CI when there’s only one dev. Just make running the test suite part of your deploy script. Use hosted services. Sure it would be cheaper to rent a server from OVH and do it all yourself. You don’t have time for that. You’re saving money on staff, spend a little on a hosted database, k8s, etc. I love Google Cloud Run. Love it. Seamless deploys, dead easy rollbacks, inbuilt secret management, etc. Micro services are about scaling teams, not technology. You’re a team of one, that’s probably the number of services you should have.
• alek_m 5y ago
  Shameless plug: Try https://treblle.com/how-it-works https://treblle.com/how-it-works Treblle allows you to test API requests with one click. Because Treblle knows what data was sent and what was returned it can replicate any call that the user has made and quickly allow you to re-run specific requests. Treblle also provides you with a way of running manual tests from our platform. It also has an auto-generated documentation feature. Treblle can generate the required documentation for each endpoint just after 1 API call. It understands your JSON responses, can detect various authentication methods, group variable URLs in a single endpoint, support different documentation versions, and similar. https://www.youtube.com/watch?v=3_W7zHrsM7E&ab_channel=Treblle https://www.youtube.com/watch?v=3_W7zHrsM7E&ab_channel=Trebl... Here is a quick overview. Let me know what you think.
• XCSme 5y ago
  I try to keep it as simple/basic as possible. Most of my deployments are Gulp tasks, usually in the form of: build & upload via SFTP. Whenever there is a task that I have to manually do when building (eg. committing build hashes to a git repo, or manually replacing the version number in the UI), I create a gulp task to do that for me. Not only this saves time when deploying, but it also dramatically reduces the mistakes I can possibly make.
  • XCSme 5y ago
    To further add to this, when working at a company we were using a more complex deploying system, but in the end it it was a lot slower and failed a lot more often than the simple tasks that just build and customize locally and upload to the servers.
• pawelkomarnicki 5y ago
  I am a solopreneur running my own and clients' projects and the best way to NOT do devops is using Heroku (for simple/early projects) and/or AWS ElasticBeanstalk (for bigger/complex projects) — nothing beats the simplicity.
• matthewadams 5y ago
  When it comes to managing your git repo's support for releases, you might like our alternative to git-flow, which we jokingly call "git-ebb": https://gitlab.com/northscaler-public/release-management https://gitlab.com/northscaler-public/release-management It's a fairly low-tech set of shell scripts that implement a release management strategy that is based on one release branch per minor version. All it does is manage version strings, release commits, release branches & release tags. You can hook your CI/CD into it whenever you're ready for that. We've used it to great effect on many client projects. The workflow is pretty simple for a new release (assume a Node.js project in this example): 0. Your main ("main", "master", "trunk", "dev") branch is where all new features go. Assume our next version is going to be "2.3.0", so the version in the main branch starts out at "2.3.0-pre.0". If you need dev prereleases, issue them any time you'd like with `./release nodejs pre`. This will bump the version to "2.3.0-pre.1", "2.3.0-pre.2", etc each time. 1. Ceremony: decide that you're feature complete for your next release. 2. Use the release script to cut a release candidate ("rc"), say, with `./release nodejs rc`. You'll end up with a new branch of the form vmajor.minor, so v2.3 in this example, and the version in that branch will be 2.3.0-rc.0. Appropriate git tags will also be created. The version in the main branch is bumped to 2.4.0-pre.0, for the next minor release. 3. Test your release candidate, releasing more release candidates to your heart's content with `./release nodejs rc`. Meanwhile, developers can start working on new features off of the main branch. 4. Ceremony: decide you're bug-free enough to perform a "generally available" (GA) release. 5. Perform a GA release with `./release nodejs ga`. This will tag a release commit as "2.3.0", push the tag, then bump the version in the release branch (v2.3) to "2.3.1-rc.0". 6. If you find a bug in production, fix it in the release branch, issue as many RCs as you need until it's fixed, then finally release your patch with `./release nodejs patch`. You'll get a release commit & tag "2.3.1", and the version will be bumped to "2.3.2-rc.0". Lastly, cherry pick, (often, literally "git cherry-pick -x <sha>") the change(s) back to the main branch if the bug still applies there; 99% of the time, it will. 7. Repeat ad nauseum. This allows you at least manage your versions, branches & git tags in a sane & portable way that's low-tech enough for anyone to work on and understand. It's also got plenty of idiot-proofing in it so that it's hard to shoot yourself in the foot. Further, it's very customizable. After years of use across lots & lots of projects, we recommend using "dev" as your main branch name and as your main branch's prerelease suffix, and using "qa" as your release branch's prerelease suffix. The defaults are "pre" & "rc", and too many folks are using these scripts nowadays for us to change the defaults.
• mgalgs 5y ago
  > Just properly setting up and maintaining a small k8s cluster looks almost like a fulltime job. The managed offerings floating around these days (DigitalOcean, AWS, and Google all have them, probably more) are insanely easy to set up and maintain. A managed cluster at DigitalOcean starts at $10/month. I definitely wouldn't recommend standing up your own cluster. For new projects, I actually find it easier to whip up a deployment.yaml and toss it in my shared k8s cluster (DigitalOcean) than setting up a new VPS where I have to manage updates (including migration when the support window for my OS ends), service restarts, etc. Github Actions for CI. Makefiles for deployment (`docker build+push` and `kubectl apply`).
• herbst 5y ago
  I do it KISS. Deployment for me is always a git hook, usually with a private gitea server. So I have versioning and rollbacks available with a cute UI if necessary. DBs run side by side and are simply backed up regularly with whatever backup solution the VPS offers. That's it. Out of school I put way more effort in building my infrastructure but in reality a good VPS, something like Cloudflare and some app based caching can run millions of daily views over 20+ different apps without issues. Edit:// it's mostly rails apps I host btw. For my wordpress installs i use dedicated VPSes because those scale horrible.
  • playpause 5y ago
    I like this answer, but could you expand a bit on why gitea? Never used it myself, but it looks like self-hosted OSS clone of GitHub? If so, my instinct (in terms of KISS) would be to just use GitHub, so I don’t have to set up and maintain my own thing. What made you decide to go with gitea, out of interest?
    • herbst 5y ago
      GitHub would be the opposite of kiss in my opinion. External dependencies and pushing to remote servers are just pain points. > git init --bare On your remote server is all you need to spawn a repo to push to, usually hook examples already included. The hook can then directly checkout to my live directory, run migrations and restart the server without weird remote automations after every (or specified) push. Gitea is just a little extra (setup once, touch never thingy) for quick edits and checks (it's a wonderful lightweight piece of software too)
  • iKevinShah 5y ago
    > Out of school I put way more effort in building my infrastructure but in reality a good VPS, something like Cloudflare and some app based caching can run millions of daily views over 20+ different apps without issues. I read such messages (and also agree to it mostly) and then I encounter my peers wanting active-active clusters with K8s on top or something and I feel I like a dinosaur. Refreshing to see KISS still being used. Also gitea. What an awesome piece of open source software.
• cpursley 5y ago
  That's easy, Render.com (it's like if Heroku were designed in 2021, and a lot cheaper). And can handle docker and distributed Elixir out of the box as well as free static sites). Took me just a few hours at most to set up and has been very reliable (I think it's k8s on Google Cloud under the hood but it's all abstracted away).
• deleted 5y ago
  [deleted]
• schappim 5y ago
  I use https://www.hatchbox.io/ https://www.hatchbox.io/ , like Heroku (git based workflow), but you can choose what infrastructure you run on. Hatchbox manages the deployment, backups, rollback etc.
• prohobo 5y ago
  I've been using Beanstalk for my new project, which is like an in-between for full on AWS and Heroku, but if you need you can decompose it into individual AWS components. It's more difficult to get started and there are way less sensible defaults, but I still got everything deployed for my POC in less than 4 hours. As for DBs / engines and such, I try to go for hosted services as long as they're not stupidly priced. I'm currently using Elastic cloud. If I need a relational DB I'll probably deploy Postgres on AWS and install some scripts to back up the data to an S3 bucket every day or so. I chose NOT to use k8s or Docker because, like you said, it's basically a fulltime job to maintain. I don't think it's needed at all unless you're a large organization. So basically: a mixture of roll-your-own, streamlining platforms, and hosted services. Do whatever works and is easiest, and don't worry so much about having ALL the goodies.
• fxtentacle 5y ago
  I run my own Heroku clone which is then licensed by my other companies. Also, I review how much time each product / feature consumed each year and then decide what to deprecate. Each April I then have my own little party where I ceremoniously turn off last year's time wasters. I'd say it all boils down to "know your customer lifetime value"
• lbayes 5y ago
  Ex-Googler here and 20+ year veteran of SF startups. It sounds like you have a great set of intuitions, and you're right, all that infrastructure is a nightmare to set up and manage. Step 1: Question every requirement Step 2: Drop everything that is not critical Step 3: Profit! Git hooks and Makefiles are great! I have an HP workstation I bought from eBay under my desk that has 32GB of RAM, 2x 12 core CPUs and a static IP address. It's probably 8 years old, but fast enough to serve whatever we need for a long while. The machine (Old Blue), hosts our Git repos, web app, database and integration service (git hook that calls, 'make publish'). We're not serving Google scale traffic, so we don't need Google scale infrastructure. Keep it simple whenever possible and don't let the modern stack complexity creep in until you absolutely need it. Even going to the cloud, when you've done it 10 times and know how, is way more work than you need when just starting out. Take on those costs and complexities only when your traffic requires it, and you may just find out that you never have to pay rent for compute.
  • djakaitis 5y ago
    I hope you’re backing up customer data if you store any…
    • lbayes 5y ago
      Definitely. In fact, it's super easy to back up as it's in a SQLite database (for now).
      • beagle3 5y ago
        How do you backup the SQLite? I used to rsync them, but when traffic picked up, I started to fear an inconsistent copy (rsync does verify at the end there are no new changes, and restarts - but i don’t know the exact strategy so I am wear if relying on it). I switched to doing .backup and rsync that instead - but am considering switching to litestream instead.
• atom_arranger 5y ago
  I had a K8s setup running on Google cloud, using their managed database service. This was kind of nice, but expensive, cost ~$100 a month, and I was using very low specs. I have multiple projects, none of them making money currently, so $100/month for each basic app is not good. I switched to using Dokku on Vultr, $12 a month. You can easily create Postgres databases and link them to Docker apps. I haven't bothered to setup CD yet but it looks like it should be simple, for now I just push to it when I want to deploy. Liking this setup so far. For frontend I use Netlify, I use their redirects to proxy the backend.
• malwrar 5y ago
  Not a solo founder, but I do a lot of ops professionally. Unless you have a specific reason, I say it’s best to avoid complicated tooling until later. k8s, ci, etc are all really useful and solve a lot of hard problems, but god are they a bitch to set up and then monitor, fix when something weird happens, secure from nasty people outside, patch when some update becomes necessary, resist the urge to “improve”, etc. This doesn’t include the time it takes to work these tools into your project, like k8s for example really requires you grok its model of how stuff it runs is supposed to look. There’s a reason entire companies exist that simply offer managed versions of these services! If you’re using fewer than like 10 boxes and don’t work with many people, you can get away with spooky arcane oldhat sysadmin deploy techniques (scp release to prod hosts, run deploy script, service or cron to keep running) and honestly it’s pretty fun to just run a command and poof you’re deployed. Then you can just wait until it starts making sense to use fancier tooling (more load, more sites/DCs, more services, more people/teams, etc). Monitoring is always good to start with, for other stuff a good litmus test is that price points for hosted k8s & ops stuff will start to make sense, you’ll be paying in time in any case. Who knows, maybe when that happens something new will pop up that makes even more sense. Until then, reducing the amount of stuff you need to worry about is always a good strategy.
  • noir_lord 5y ago
    Agree with all of this. If you want an intermediate step you can just make docker images of your application and deploy and run those manually (again with shell scripts). You can get a long way that way without having to deal with kubernetes.
  • dbbk 5y ago
    Or just deploy it on Heroku and call it a day?
    • clairity 5y ago
      yah, this (or something similar) is the most straightforward solution. the whole point of heroku is to abstract away devops for solo/small dev groups and make deployments “one-click”. it’s unclear why that option was explicitly ruled out here.
• namenotrequired 5y ago
  Unpopular answer alert, but it worked well for my context for the few years before I hired a team. I didn’t do CI/CD. I just reviewed my own code the next day, improved it and put it live. As the only dev I knew more or less how my code worked and what risk factors to prepare for in each deploy, and serious, business-impacting production issues were rare. A simple stack of Meteor on Galaxy + Mongo on Atlas meant all my tools integrated well together and a whole lot of devops things were easy enough that I rarely had to think about them after they were set up once. I can think of many cases where this would not work but it worked well for my context.
• kingkool68 5y ago
  Bash scripts!
• easytiger 5y ago
  Don't use them. They require time to maintain and if you are old like me half of them make very little sense anyway.
• iwd 5y ago
  If the hours you spend implementing and maintaining your DevOps exceed the hours of downtime you prevent, you're probably not making good use of your time. The less you build, the less you have to maintain. I've been running a Python-based, highly custom web store solo since 2007, and it supports multiple people doing fulfillment. I host on Opalstack so as to outsourcing patching, email config, database maintainence, etc. I run directly in a Git repo (actually Hg, it's that old) and releases are "git pull && ./restart.sh". Rollbacks are "git checkout ...". I've had to migrate/rebuild the VM about every 5 years. Tech changes enough in that time that no automation will still work unmodified. So I just keep good notes about what I did last time, and figure out what the new equivalents are when I finally have to do it again (updating the notes, of course). Database and Conda are easy to port. It's usually DNS and email integrations that are a pain. As others have said, KISS is key. Industry DevOps is for a work setting with a decent-sized team, where you can afford the overhead of maintaining it all in order to make the overall team more efficient.
• kaino128 5y ago
  Pieter Levels (the guy who made Nomad List) reached $1 000 000 ARR with a single VPS, all his code in an index.php file and using FTP to transfer the latest code to his server... I think if you’re a solo founder you have to accept on day one you’re at a time disadvantage when it comes to the actual hours you’ll spend on moving the business forward vs admin, ops, legal stuff etc etc and so you have to really fight smart! I.e K8S? Definitely don’t need it. Complex CI/Deployment workflow? Nope, just the minimal shell scripts to do clean deploys and rollbacks. Pay for a HA RDS so you never need to worry about DB replication etc. If you just work with the minimal set of tools you’re comfortable with and relentlessly keep things simple it’s possible. I’ve built high availability stuff as a solo founder and still found (at least some...) time to work on sales and the business.
  • alberth 5y ago
    This! Exactly my point that I was making here in this thread: https://news.ycombinator.com/item?id=28840902 https://news.ycombinator.com/item?id=28840902
  • qw 5y ago
    I think you are right. There's a big difference between the technology I would use in a larger organization compared to solo projects. Unless you build a side project to learn a new technology, you should use something "boring" that you know well. Here's my personal choices in preferred order: 1. Static web sites - Just upload to S3 behind a CDN and forget about it 2. Server rendered website - PHP. It may not be something you brag about, but in my experience it just "works" and has excellent uptime. Note: If I had to recommend something to a larger organization, I would probably suggest Next.js to attract developers. PHP is often associated with "old school" tech 3. REST service - Spring Boot. I have so much experience with it and there are so many Spring projects that supports a huge range of technologies from monitoring to data 4. Scripts - Python
  • epolanski 5y ago
    What is a HA RDS?
    • 5y ago
• scottydelta 5y ago
  I have been working on https://quantale.io https://quantale.io since a year and half. It's a web based alternative to Bloomberg Terminal. I personally manage the CI/CD infra as well as the data infrastructure for my startup. Here is my setup: 1. Self-hosted Drone.io instance connected to github(the instance is behind a firewall and can only be accessed via a self-hosted VPN. Github IPs are whitelisted so that auto build/deployment can be initiated via github hooks) I have drone.io config that helps me manange the versioning, deployment to stage/production, run tests, do rollbacks etc. for eg: merging to master on github auto builds the master branch on drone and deploys to staging along with pulling the latest db backup from S3 and adding to stage for testing 2. Self-hosted production and staging server that hosts my web/django app. It has a local postgres db and is encrypted and backed up on S3 every hour. 3. Self-hosted Elasticsearch cluster using Ansible playbooks with daily snapshot backups to S3. beyond this, I self-host 3 VPNs to protect my entire infrastructure. read my old HN comment about it: https://news.ycombinator.com/item?id=28671753 https://news.ycombinator.com/item?id=28671753 All this is one time infrastructure setup and it all has been running smoothly for more than a year without any hiccups. I would be happy to help you setup a similar setup if you want. hit me up at vikash@quantale.io
  • iKevinShah 5y ago
    > 2. Self-hosted production and staging server that hosts my web/django app. It has a local postgres db and is encrypted and backed up on S3 every hour. Did you have any instance where you needed to restore the DB / Have you done restoration tests to ensure this is a feasible solution? Also, since you are using local postgres DB, do you not run into issues like transaction id wraparound ? Also, how many days of backup you keep on S3? Considering you create 24 backups a day.
    • scottydelta 5y ago
      > Did you have any instance where you needed to restore the DB / Have you done restoration tests to ensure this is a feasible solution? Whenever a build is deployed to stage server, it automatically pulls the latest version backup in last 1 hour and restores it on the stage for testing with the latest data. I have config to do the same restore to the prod if needed. I have been using the same setup for over 2 years to support an internal web app of a company. > Also, since you are using local postgres DB, do you not run into issues like transaction id wraparound ? This problem only arises when there is heavy load on the database, specifically high WRITE/UPDATE/DELETE operations. Quantale uses postgres to store only user info and all the other data is stored in Elasticsearch(twitter, reddit, news, SEC, Financial data on over 15000+ stocks). This is intentional because with the scalable server resources, my postgres setup can handle a large number of users before I see that problem. I can always scale up later when needed rather than over-engineering it right now. >Also, how many days of backup you keep on S3? Considering you create 24 backups a day. I have 7 days backups on S3 with versioning switched on for 60 days. So at anytime I can access the version backed up in last 60 days.
• ignas- 5y ago
  This seems to (strangely) be an unpopular answer here, but I've spent a weekend to terraform everything. I run a couple of different projects which are yet to pick-up traction, so prototyping quickly is essential. What I have now is terraform on Github which handles Networking (VPC, subnets), ECS/ECR, ALB for my backend and S3/CloudFront for frontend. Everything is on AWS because I'm familiar with it, so it's faster. And I compensate for costs with AWS Activate + Free Tier. This combination is usually enough to understand whether the project will get traction or not. Actually spending little time on IaC was one of the best recent investments. But I guess it holds more value when you have a couple of projects, still exploring and these tasks get repetitive
  • roland35 5y ago
    I find this helpful for my side project as well. It's amazing how quickly a) things break down and b) I forget how to set things up and get running again if I have to shelve and restart work on a project.
  • mleonhard 5y ago
    I spent a lot of time on Terraform + Packer code to set up VPC, RDS, Route53, AWS Cert Manager, ELB, EC2, my API server, S3, CloudFront, and CloudWatch. The setup had a lot of complexity and footguns. Plus there was the risk of accidentally running up a huge AWS bill. Later, I realized that Heroku + Netlify are good enough. Switching was easy. The hardest part was admitting to myself that I had wasted so much time on AWS. I felt relieved after I deleted the Terraform and Packer code.
    • ignas- 5y ago
      AWS bill is a concern. I put alerts on billing but they do come with a delay. I was thinking Digital Ocean. Or Heroku for the current project, but ended up with a design with quite a few serverless pieces using lambda. Which has a great free tier and is very useful for my use-case. Would not say that AWS is too much complexity though.. When you go over things at least once and understand how thing fall together
• diiq 5y ago
  Lots and lots of best practice exists specifically to help teams, and especially teams with some normal amount of turnover. The problems of a solo dev are very different than a dev on a team. Knowledge silos don't exist. Distributed expertise doesn't exist. There's no one to mentor, no shared vision to maintain, no intertia to combat. I consult on big complicated team projects. I also manage multiple solo projects. On solo projects, deployment is a script I run from my dev machine. I'm the only person who deploys; anything else would be solving a problem I don't have. The only "CI" is running the tests before I run the deployment script. I'm the only one who needs to see the outcome of the tests. Anything more would be solving a problem I don't have. Architecture is whatever makes the most sense to me personally -- which is often miles away from what I would recommend to a client, who needs something any new hire will recognize and be able to work with. I pay a service to manage backups so I can walk away from a solo project for months and know it's ticking away. The point is: solve problems you actually have. Don't try to run a "professional" operation by doing what big teams do all by yourself. Big teams have different problems.
  • count 5y ago
    'knowledge silos' in a one person team are real: Me today and me from 2 years ago when I set a thing up are different people, and I don't remember that any more than a team mate in a multi-person team would know it. This is where 'professional' approaches can help considerably: using 'standardized' stuff, instead of bespoking everything (even if bespoking is much easier), or making sure you write a TON of documentation for anything you don't do daily...
    • ohashi 5y ago
      I'm a fan of the documenting everything even on stuff I wrote and maintain for myself. Because, like you said, old me was way smarter than current me. When he was writing this code, he had the domain space modeled in his head better than me now, 2 years later trying to fix/maintain something.
  • tarunm 5y ago
    > I pay a service to manage backups so I can walk away from a solo project for months and know it's ticking away. Any recommendation of such service?