5 ms·

> But the heart of Codeberg – Gitea – is still on a rented cloud instance > We have a remote Ceph filesystem mounted on the old cloud VPS They had one (1) VP

> But the heart of Codeberg – Gitea – is still on a rented cloud instance > We have a remote Ceph filesystem mounted on the old cloud VPS They had one (1) VPS running a service that they charge for with 38k users? That seems not great for reliability and redundancy.

xena 4y ago
It turns out you can get surprisingly far with a single server. You'd be amazed.
- capableweb 4y ago
  Indeed, people vastly underestimate how far you can go with a single server. Even Digital Ocean offers a 48 vCPUs/96 GB instance for ~1000 USD which you'll be able to host most of the businesses seen on HN with. Obviously, if you're doing storage intensive stuff, you wouldn't use the built-in storage, if you're doing really CPU intensive stuff you might outsource that to a different instance and so on. But people tend to want to distribute their app with 5k monthly users to 10 really weak instances instead of just 2 medium ones or 1 big one. And in that, they think it makes it more reliable but fail to account for the added complexity of distributed architectures and start to fail when doing deployments and changes instead. Another part people underestimate is the performance of dedicated instances vs "virtual" instances. Many times I've seen people shocked at how last applications work when running on dedicated instances, and how far you can scale with them as you won't have to upgrade as often as you think. Cloud is mainly meant (for me at least) for software you need to be able to scale really fast up AND down, not just up as you get more users. But most businesses I've worked for, with and started myself, never really had that need.
  - adql 4y ago
    Well, you probably want at least 2 for redundancy, but active/passive setup on 2 nodes can cover great many use cases if you're not running some memory hogging Rails blob that needs hundreds of MB just to render a site.
    - capableweb 4y ago
      In reality, complexity of deploys and introducing new changes are way more likely to shut down your service than anything else, unless you're dealing with very large scale. And for those problems, it doesn't matter if you have 1 or 10 instances, shit would go down anyways as you push out your change to all of them. Services falling over themselves for no reason is really uncommon in my experience, and usually are because A) introduced changes, B) hitting performance bottlenecks that are really bugs in the code or C) downtime of 3rd parties that someone failed to account could actually be down so that brought down "our" application too.
      - adql 4y ago
        Complexity of rsyncing your app onto 2 nodes instead of one isn't a problem tho. The DB is the hard part (not with automation but that's more to learn), and storing the files to be seen by all nodes (if app needs to create files during its normal work) is, but in most cases delegating that part to cloudy cloud isn't a problem. > In reality, complexity of deploys and introducing new changes are way more likely to shut down your service than anything else, unless you're dealing with very large scale. And for those problems, it doesn't matter if you have 1 or 10 instances, shit would go down anyways as you push out your change to all of them. Right, but updating OS is one of those changes, can't do that hitless if there is only one node in the system. And you do update your OS (or whatever FROM you got your container), right ? Sure, most downtimes will be more "developer fucked up" than "something happened with hardware". But those fuckups are usually on smaller scale than "well, server needs to be rebuilt from scratch/backup/CM manifest" and "just" need revert. > And for those problems, it doesn't matter if you have 1 or 10 instances, shit would go down anyways as you push out your change to all of them. If you have tens of instances you can be responsible developer and do some kind of staged deploy to cut those fuckups by order of magnitude or two.
    - ta988 4y ago
      In practice for most projects being inaccessible for a couple hours is not the end of the world. And you loose less money than falling in the DevOps SRE hole where you pay for everything 5 times plus the people that take care of it all of that for it to go down anyway because of a long tail event.
  - noirscape 4y ago
    The reality of it is that a lot of people vastly overestimate the activity of what they build will actually reach. You can easily host a lot of stuff with just a "basic" SQL database and a single frontend as long as you have proper rate limiting on your APIs and make sure your frontend is sane. NGINXs original design goal was to handle 10,000 connections at the same time and nowadays it can handle up to ~400,000 without needing reconfiguration. Even a basic cache layer on nginx can often completely circumvent the need of calling the backend, especially for open APIs. Postgres can easily handle dozens of connections at the same time with the right config flags. To give a bit of an estimate on how far basic infrastructure can carry you; the entire RetroArch project[1], which I'd say has about the same userbase as any small tech company is entirely hosted, buildbot (so CI) included, on a single 40$ DigitalOcean instance. Most software is suprisingly efficient/resilient if you just configure it properly. The only place where I've seen horizontal scaling as an actual necessity vs. just properly configuring the tools you're given in most cases is Rails. There's no real efficiency fix for Rails tooling from what I can tell. It's just slow as molasses and you're gonna foot an ever expanding server bill as your userbase grows. [1]: No endorsement.
- adql 4y ago
  Think they are complaining about lack of redundancy not computing power
- danpalmer 4y ago
  My point is not performance, I'd agree you can get far. My point is that when selling a service it seems somewhat irresponsible to run it on such a clear single point of failure.
  - capableweb 4y ago
    Is it actually making things less reliable to run things on a single instance though? In my experience, why things go down tend to be because humans deploy changes, not because one instance went down because of hardware failures or connectivity issues. I've outlined more of this thought here too: https://news.ycombinator.com/item?id=33593361 https://news.ycombinator.com/item?id=33593361
    - salamander014 4y ago
      It's more about what happens once there is an issue. I agree if something works and remains unchanged it will probably stay reliable, the issue is once you introduce a change, even nowadays you can possibly break something that isn't easy to undo (or troubleshoot without taking services down). For low traffic stuff, might seem OK, but no redundancy for a service that users pay for seems irresponsible, however Gitea might just be for them to track internal stuff, so maybe a little less so.
      - capableweb 4y ago
        > It's more about what happens once there is an issue But again, when I've seen distributed infrastructure, it's very uncommon to see issues hitting one instance while others are not being hit by the very same issue. When you introduce change, you introduce it to all running instances, all at once or over a period of time. You might be in luck if you're doing blue/green deployment, but you can achieve the same thing with a staging + production environment mostly.
        markstos 4y ago
        Hardware failures will hit one twin and not the other. Same for some kinds of high load triggers. Major changes like OS upgrades can be applied to one twin and then the other.
jannes 4y ago
> that they charge for I think it is a free service. They are a non-profit organisation.
- danpalmer 4y ago
  Ah yes, fair enough. I had spotted they were non-profit, but not that they were actually free on top of that.
bluedino 4y ago
One vps had enough file system or were they using something else to supplement it?
bayindirh 4y ago
> That seems not great for reliability and redundancy. A single server, esp. when virtualized, offers great reliability. Add regular backups in terms of disk snapshots, and you'll have a "thing" which can restart in 10 seconds and rollback in half a minute. Yes, you can do failover pretty simply, but a single server really goes a long way. We redundantly install critical servers, but the backup doesn't kick-in unless the other one halts and catches fire (sometimes literally).
- hedora 4y ago
  If you want to go the enterprise route, you can mount the VM on an on-prem block store with a secondary ~ 10-20ms away, and set it to async or sync replication mode. If the primary DC burns down (hey, it happens!), you just fail over and lose ten or zero seconds of transactions, respectively. Downtime is roughly time to detect the outage + time it takes to reboot the primary. You can also do this in the cloud, of course, but I've found that grounding the implementation in a physical example clarifies exactly what I'm talking about. Some people claim this isn't a "high availability" solution, since you need to take the service down while you reboot for OS patches, etc, and during those windows, all requests fail. However, that's often fine. In particular, these sorts of setups can easily hit five nines, which is as good as an AWS region. However, in normal operation, they don't spuriously generate error codes like S3, etc do. If you count your SLA as "service went 1 minute without generating a spurious error for any client", then this will have many more 9's than a microservice architecture. If you count your SLA as "at least one request returned 200 in a 1 minute window", then the two approaches will probably be comparable, at least until you start running the microservices in multiple AWS regions (or whatever) and eliminate outages due to human operator error.
- rdsubhas 4y ago
  What about simple stuff, like updating the kernel for the latest security patch and needing a small restart?
  - bayindirh 4y ago
    None of the pets we run complain when the service is not available for a minute or so, which is required to restart a system in the worst case scenario. Even we don't blindly restart our cattle servers. We drain & restart them, which means the load is already shifted to other servers in the fleet.
andmarios 4y ago
That's what cloud providers want you to believe. :) I have had both VPS and collocated servers chugging along for over 5 years without any interruption. In the worst case scenario where I would lose one of them, given proper backups, it would take me less than a day to restore. Few of the deployments I've seen in cloud providers, kubernetes, etc, could do 5 years without downtime. Sure, there are other costs, like scaling, finding people who can work on them, or when the dreaded time for an update comes... But still, a single VPS can get you very far for a very small cost.
jwildeboer 4y ago
They don’t charge. You can donate or become a paying member of the association. Or not. Up to any User to decide. (I am a paying member since quite some time)
Kukumber 4y ago
i can run demanding AAA games on my PC there is no reason software can't run on a single PC time to study "software performance and optimizations"
remram 4y ago
A VPS is probably running on replicated network drives, so no single hardware failure would bring it down. It would probably restart on a different host in seconds. I am not sure that adding logical replication on top would improve reliability or redundancy that much.

xena 4y ago

It turns out you can get surprisingly far with a single server. You'd be amazed.

capableweb 4y ago
Indeed, people vastly underestimate how far you can go with a single server. Even Digital Ocean offers a 48 vCPUs/96 GB instance for ~1000 USD which you'll be able to host most of the businesses seen on HN with. Obviously, if you're doing storage intensive stuff, you wouldn't use the built-in storage, if you're doing really CPU intensive stuff you might outsource that to a different instance and so on. But people tend to want to distribute their app with 5k monthly users to 10 really weak instances instead of just 2 medium ones or 1 big one. And in that, they think it makes it more reliable but fail to account for the added complexity of distributed architectures and start to fail when doing deployments and changes instead. Another part people underestimate is the performance of dedicated instances vs "virtual" instances. Many times I've seen people shocked at how last applications work when running on dedicated instances, and how far you can scale with them as you won't have to upgrade as often as you think. Cloud is mainly meant (for me at least) for software you need to be able to scale really fast up AND down, not just up as you get more users. But most businesses I've worked for, with and started myself, never really had that need.
- adql 4y ago
  Well, you probably want at least 2 for redundancy, but active/passive setup on 2 nodes can cover great many use cases if you're not running some memory hogging Rails blob that needs hundreds of MB just to render a site.
  - capableweb 4y ago
    In reality, complexity of deploys and introducing new changes are way more likely to shut down your service than anything else, unless you're dealing with very large scale. And for those problems, it doesn't matter if you have 1 or 10 instances, shit would go down anyways as you push out your change to all of them. Services falling over themselves for no reason is really uncommon in my experience, and usually are because A) introduced changes, B) hitting performance bottlenecks that are really bugs in the code or C) downtime of 3rd parties that someone failed to account could actually be down so that brought down "our" application too.
    - adql 4y ago
      Complexity of rsyncing your app onto 2 nodes instead of one isn't a problem tho. The DB is the hard part (not with automation but that's more to learn), and storing the files to be seen by all nodes (if app needs to create files during its normal work) is, but in most cases delegating that part to cloudy cloud isn't a problem. > In reality, complexity of deploys and introducing new changes are way more likely to shut down your service than anything else, unless you're dealing with very large scale. And for those problems, it doesn't matter if you have 1 or 10 instances, shit would go down anyways as you push out your change to all of them. Right, but updating OS is one of those changes, can't do that hitless if there is only one node in the system. And you do update your OS (or whatever FROM you got your container), right ? Sure, most downtimes will be more "developer fucked up" than "something happened with hardware". But those fuckups are usually on smaller scale than "well, server needs to be rebuilt from scratch/backup/CM manifest" and "just" need revert. > And for those problems, it doesn't matter if you have 1 or 10 instances, shit would go down anyways as you push out your change to all of them. If you have tens of instances you can be responsible developer and do some kind of staged deploy to cut those fuckups by order of magnitude or two.
  - ta988 4y ago
    In practice for most projects being inaccessible for a couple hours is not the end of the world. And you loose less money than falling in the DevOps SRE hole where you pay for everything 5 times plus the people that take care of it all of that for it to go down anyway because of a long tail event.
- noirscape 4y ago
  The reality of it is that a lot of people vastly overestimate the activity of what they build will actually reach. You can easily host a lot of stuff with just a "basic" SQL database and a single frontend as long as you have proper rate limiting on your APIs and make sure your frontend is sane. NGINXs original design goal was to handle 10,000 connections at the same time and nowadays it can handle up to ~400,000 without needing reconfiguration. Even a basic cache layer on nginx can often completely circumvent the need of calling the backend, especially for open APIs. Postgres can easily handle dozens of connections at the same time with the right config flags. To give a bit of an estimate on how far basic infrastructure can carry you; the entire RetroArch project[1], which I'd say has about the same userbase as any small tech company is entirely hosted, buildbot (so CI) included, on a single 40$ DigitalOcean instance. Most software is suprisingly efficient/resilient if you just configure it properly. The only place where I've seen horizontal scaling as an actual necessity vs. just properly configuring the tools you're given in most cases is Rails. There's no real efficiency fix for Rails tooling from what I can tell. It's just slow as molasses and you're gonna foot an ever expanding server bill as your userbase grows. [1]: No endorsement.
adql 4y ago
Think they are complaining about lack of redundancy not computing power
danpalmer 4y ago
My point is not performance, I'd agree you can get far. My point is that when selling a service it seems somewhat irresponsible to run it on such a clear single point of failure.
- capableweb 4y ago
  Is it actually making things less reliable to run things on a single instance though? In my experience, why things go down tend to be because humans deploy changes, not because one instance went down because of hardware failures or connectivity issues. I've outlined more of this thought here too: https://news.ycombinator.com/item?id=33593361 https://news.ycombinator.com/item?id=33593361
  - salamander014 4y ago
    It's more about what happens once there is an issue. I agree if something works and remains unchanged it will probably stay reliable, the issue is once you introduce a change, even nowadays you can possibly break something that isn't easy to undo (or troubleshoot without taking services down). For low traffic stuff, might seem OK, but no redundancy for a service that users pay for seems irresponsible, however Gitea might just be for them to track internal stuff, so maybe a little less so.
    - capableweb 4y ago
      > It's more about what happens once there is an issue But again, when I've seen distributed infrastructure, it's very uncommon to see issues hitting one instance while others are not being hit by the very same issue. When you introduce change, you introduce it to all running instances, all at once or over a period of time. You might be in luck if you're doing blue/green deployment, but you can achieve the same thing with a staging + production environment mostly.
      - markstos 4y ago
        Hardware failures will hit one twin and not the other. Same for some kinds of high load triggers. Major changes like OS upgrades can be applied to one twin and then the other.

jannes 4y ago

> that they charge for I think it is a free service. They are a non-profit organisation.

danpalmer 4y ago
Ah yes, fair enough. I had spotted they were non-profit, but not that they were actually free on top of that.

bluedino 4y ago

One vps had enough file system or were they using something else to supplement it?

bayindirh 4y ago

> That seems not great for reliability and redundancy. A single server, esp. when virtualized, offers great reliability. Add regular backups in terms of disk snapshots, and you'll have a "thing" which can restart in 10 seconds and rollback in half a minute. Yes, you can do failover pretty simply, but a single server really goes a long way. We redundantly install critical servers, but the backup doesn't kick-in unless the other one halts and catches fire (sometimes literally).

hedora 4y ago
If you want to go the enterprise route, you can mount the VM on an on-prem block store with a secondary ~ 10-20ms away, and set it to async or sync replication mode. If the primary DC burns down (hey, it happens!), you just fail over and lose ten or zero seconds of transactions, respectively. Downtime is roughly time to detect the outage + time it takes to reboot the primary. You can also do this in the cloud, of course, but I've found that grounding the implementation in a physical example clarifies exactly what I'm talking about. Some people claim this isn't a "high availability" solution, since you need to take the service down while you reboot for OS patches, etc, and during those windows, all requests fail. However, that's often fine. In particular, these sorts of setups can easily hit five nines, which is as good as an AWS region. However, in normal operation, they don't spuriously generate error codes like S3, etc do. If you count your SLA as "service went 1 minute without generating a spurious error for any client", then this will have many more 9's than a microservice architecture. If you count your SLA as "at least one request returned 200 in a 1 minute window", then the two approaches will probably be comparable, at least until you start running the microservices in multiple AWS regions (or whatever) and eliminate outages due to human operator error.
rdsubhas 4y ago
What about simple stuff, like updating the kernel for the latest security patch and needing a small restart?
- bayindirh 4y ago
  None of the pets we run complain when the service is not available for a minute or so, which is required to restart a system in the worst case scenario. Even we don't blindly restart our cattle servers. We drain & restart them, which means the load is already shifted to other servers in the fleet.

andmarios 4y ago

That's what cloud providers want you to believe. :) I have had both VPS and collocated servers chugging along for over 5 years without any interruption. In the worst case scenario where I would lose one of them, given proper backups, it would take me less than a day to restore. Few of the deployments I've seen in cloud providers, kubernetes, etc, could do 5 years without downtime. Sure, there are other costs, like scaling, finding people who can work on them, or when the dreaded time for an update comes... But still, a single VPS can get you very far for a very small cost.

jwildeboer 4y ago

They don’t charge. You can donate or become a paying member of the association. Or not. Up to any User to decide. (I am a paying member since quite some time)

Kukumber 4y ago

i can run demanding AAA games on my PC there is no reason software can't run on a single PC time to study "software performance and optimizations"

remram 4y ago

A VPS is probably running on replicated network drives, so no single hardware failure would bring it down. It would probably restart on a different host in seconds. I am not sure that adding logical replication on top would improve reliability or redundancy that much.