23 ms·

I'm at OVH as well (in the BHS datacenter, fortunately). I run my entire production system on one beefy machine. The apps and database are replicated to a backu

I'm at OVH as well (in the BHS datacenter, fortunately). I run my entire production system on one beefy machine. The apps and database are replicated to a backup machine hosted with Hetzner (in their Germany datacenter). I also run a tiny VM at OVH which proxies all traffic to Hetzner. I use a failover IP to point at the big rig at OVH. If the main machine fails, I move the failover IP to the VM, which sends all traffic to Hetzner. If OVH is totally down, and the fail over IP doesn't work, I have a fairly low TTL on the DNS. I backup the database state to S3 every day. Since I'm truly paranoid, I have an Intel NUC at my house that also replicates the DB. I like knowing that I have a complete backup of my entire business within arm's reach.

warrenm 6y ago
Yet another cheer for Hetzner!
AmericanChopper 6y ago
> I like knowing that I have a complete backup of my entire business within arm's reach. It could also provide a burglar a fantastic opportunity to pivot into career in data breaches.
- baggy_trough 6y ago
  It all depends on your paranoia level of data hacking burglars vs. vaporized data centers.
- tgsovlerkhgsel 6y ago
  This problem is usually solved through encryption.
  - AmericanChopper 6y ago
    If I were to ask my CISO if I was allowed to bring the production database home, I’m pretty sure his answer wouldn’t be “as long as you encrypt it”.
    - sneak 6y ago
      Once you're big enough to afford a CISO, you're likely big enough to afford office space with decent physical security to serve as a third replicated database site to complement your two datacenters. These solutions are not one-size-fits-all. What works for a small startup isn't appropriate for a 100+ person company.
      - AmericanChopper 6y ago
        Yes, I agree. Small companies typically are very bad at security.
        sneak 6y ago
        I am really good at security and I too keep encrypted backups on-site in my house.
        _joel 6y ago
        Not in my experience. Worked at some small shops that were lightyears ahead in terms of policy, procedures and attitude compared to places I've worked with 50k+ employees globally.
        AmericanChopper 6y ago
        Large organisations tend not to achieve security compliance with overly sophisticated systems of policy and controls. They tend to do it using bureaucracy, which while usually rather effective at implementing the level of control required, will typically leave a lot to be desired in regards to UX and productivity. Small organisations tend to ignore the topic entirely until they encounter a prospective client or regulatory barrier that demands it. At which point they may initially implement some highly elegant systems. Until they grow large enough that they all devolve into bureaucratic mazes.
        _joel 6y ago
        I'm aware, but that's not been my experience. I've been in large places where there's been a lassiez faire attitude because it was "another team's job" and general bikeshedding over smaller features because the bigger picture security wasn't their area or was forced from a dictat from above to use X because they're on the board, whilst X is completely unfit for purpose. There's no pushback. However I've worked at small ISPs where we took security extremely seriously. Appropriate background check and industry policy but moreso the attitude... we wanted to offer customers security because we had pride in our work.
- axaxs 6y ago
  LUKS is your friend.
- Thorrez 6y ago
  But it does protect somewhat against ransomware on the servers.
- dredmorbius 6y ago
  For small firms, CEO / CTO maintaining off-sites at a residence is reasonable and not an uncommon practice. As with all security / risk mitigation practices, there is a balance of risks and costs involved. And as noted, encrypted backups would be resistant to casual interdiction, or even strongly-motivated attempts. Data loss being the principle risk mitigated by off-site, on-hand backups.
notaharvardmba 6y ago
Bacula has some really cool features for cloud backups. https://bacula.org https://bacula.org
- yongjik 6y ago
  Ow what an unfortunate name. https://en.wikipedia.org/wiki/Baculum https://en.wikipedia.org/wiki/Baculum
  - schoen 6y ago
    It just means "stick" in the original Latin!
  - 867-5309 6y ago
    but fortunately also within arm's reach
  - waheoo 6y ago
    LGTM
johnchristopher 6y ago
It seems your setup follows the three rules back-ups with at least two in different physical location. https://www.nakivo.com/blog/3-2-1-backup-rule-efficient-data-protection-strategy/ https://www.nakivo.com/blog/3-2-1-backup-rule-efficient-data...
vanviegen 6y ago
Are you me, by any chance? :-) I also run our entire production system on one beefy machine at OVH, and replicate to a similar machine at Hetzner. In case of a failure, we just change DNS, which has a 1 hour TTL. We've needed to do an unplanned fail-over only once in over 10 years. And like you, I have an extra replica at the office, because it feels safe having a physical copy of the data literally at hand.
- 7ewis 6y ago
  Not done any research into it, but I always thought OVH was supposed to be a very budget VPS service primarily for personal use rather than business. Although thought it was akin to having a Raspberry Pi plugged in at home. Again, I may be completely wrong but why would you not use AWS/GCP? Even if it's complexity, Amazon have Lightsail, or if it's cost I thought DigitalOcean was one of the only reputable business-grade VPS providers. I just can't imagine many situations where a VPS would be superior to embracing the cloud and using cloud functions, containers, instances with autoscaling/load balancers etc.
  - Symbiote 6y ago
    OVH is a European equivalent to Digital Ocean. It has twice the revenue, and is the third largest hosting provider in the world.
    - nixgeek 6y ago
      Twice the revenue of DigitalOcean still puts it < $1B ARR, or am I missing something? I can’t see how that’s the third largest in the world, or does your definition of “hosting provider” exclude clouds?
      - Symbiote 6y ago
        I took it from the top of their Wikipedia page. In any case, they aren't "primarily for personal use". https://en.wikipedia.org/wiki/OVH https://en.wikipedia.org/wiki/OVH
eb0la 6y ago
Are you truly paranoid? If my money and/or job depended on having something running without (or with minimal) disruption I would be as paranoid as you, too. BTW - Some people call this business recovery plan, not plain paranoia ;-)
- buran77 6y ago
  It's not paranoia if you're right. All of the risks GP is protecting against are things that happen to someone every day, and they should be seen like wearing the seat belt in a car.
  - klingon78 6y ago
    I have a reliability and risk avoidance mindset, but I’ve had to stand back because my mental gas tank for trying to keep things going is near empty. I’ve really struggled working with others that either are both ignorant and apathetic about the business’s ability to deal with risk or believe that it’s their job to keep putting duct tape over the duct tape that breaks multiple times a day while users struggle. I like seeing these comments reminding others to a wear seat belt or have backups for their backups, but I don’t know whether I should care more about reliability. I work in an environment that’s a constant figurative fire. I also like to spend time with my family. I know it’s just a job, and it would be even if I were the only one responsible for it; that doesn’t negate the importance of reliability, but there is a balance. If you are dedicated to reliability, don’t let this deter you. Some have a full gas tank, which is great.
    - CraigJPerry 6y ago
      This resonates with me. I notice my gas tank rarely depletes because of technology. It doesn’t matter how brain dead the 00’s oracle forms app with absurd unsupported EDI submission excel thinga-ma-bob that requires a modem ... <fill in the rest of the dumspter fire as your imagination deems>. Making a tech stack safe is a fun challenge. Apathetic people though, that can be really tough going. It’s just that way “because”. Or my favourite “oh we don’t have permission to change that”, how about we make the case and get permission? _horrified looks_ sometimes followed by pitch forks.
venj 6y ago
are your domains at ovh too ? If yes, I'd consider changing this: this morning the manager was quite flooded and the DNS service was down for some time...
extrasolar 6y ago
this is the way. I do the same. Not paranoid at all.

warrenm 6y ago

Yet another cheer for Hetzner!

AmericanChopper 6y ago

> I like knowing that I have a complete backup of my entire business within arm's reach. It could also provide a burglar a fantastic opportunity to pivot into career in data breaches.

baggy_trough 6y ago
It all depends on your paranoia level of data hacking burglars vs. vaporized data centers.
tgsovlerkhgsel 6y ago
This problem is usually solved through encryption.
- AmericanChopper 6y ago
  If I were to ask my CISO if I was allowed to bring the production database home, I’m pretty sure his answer wouldn’t be “as long as you encrypt it”.
  - sneak 6y ago
    Once you're big enough to afford a CISO, you're likely big enough to afford office space with decent physical security to serve as a third replicated database site to complement your two datacenters. These solutions are not one-size-fits-all. What works for a small startup isn't appropriate for a 100+ person company.
    - AmericanChopper 6y ago
      Yes, I agree. Small companies typically are very bad at security.
      - sneak 6y ago
        I am really good at security and I too keep encrypted backups on-site in my house.
      - _joel 6y ago
        Not in my experience. Worked at some small shops that were lightyears ahead in terms of policy, procedures and attitude compared to places I've worked with 50k+ employees globally.
        AmericanChopper 6y ago
        Large organisations tend not to achieve security compliance with overly sophisticated systems of policy and controls. They tend to do it using bureaucracy, which while usually rather effective at implementing the level of control required, will typically leave a lot to be desired in regards to UX and productivity. Small organisations tend to ignore the topic entirely until they encounter a prospective client or regulatory barrier that demands it. At which point they may initially implement some highly elegant systems. Until they grow large enough that they all devolve into bureaucratic mazes.
        _joel 6y ago
        I'm aware, but that's not been my experience. I've been in large places where there's been a lassiez faire attitude because it was "another team's job" and general bikeshedding over smaller features because the bigger picture security wasn't their area or was forced from a dictat from above to use X because they're on the board, whilst X is completely unfit for purpose. There's no pushback. However I've worked at small ISPs where we took security extremely seriously. Appropriate background check and industry policy but moreso the attitude... we wanted to offer customers security because we had pride in our work.
axaxs 6y ago
LUKS is your friend.
Thorrez 6y ago
But it does protect somewhat against ransomware on the servers.
dredmorbius 6y ago
For small firms, CEO / CTO maintaining off-sites at a residence is reasonable and not an uncommon practice. As with all security / risk mitigation practices, there is a balance of risks and costs involved. And as noted, encrypted backups would be resistant to casual interdiction, or even strongly-motivated attempts. Data loss being the principle risk mitigated by off-site, on-hand backups.

notaharvardmba 6y ago

Bacula has some really cool features for cloud backups. https://bacula.org https://bacula.org

yongjik 6y ago
Ow what an unfortunate name. https://en.wikipedia.org/wiki/Baculum https://en.wikipedia.org/wiki/Baculum
- schoen 6y ago
  It just means "stick" in the original Latin!
- 867-5309 6y ago
  but fortunately also within arm's reach
- waheoo 6y ago
  LGTM

johnchristopher 6y ago

It seems your setup follows the three rules back-ups with at least two in different physical location. https://www.nakivo.com/blog/3-2-1-backup-rule-efficient-data-protection-strategy/ https://www.nakivo.com/blog/3-2-1-backup-rule-efficient-data...

vanviegen 6y ago

Are you me, by any chance? :-) I also run our entire production system on one beefy machine at OVH, and replicate to a similar machine at Hetzner. In case of a failure, we just change DNS, which has a 1 hour TTL. We've needed to do an unplanned fail-over only once in over 10 years. And like you, I have an extra replica at the office, because it feels safe having a physical copy of the data literally at hand.

7ewis 6y ago
Not done any research into it, but I always thought OVH was supposed to be a very budget VPS service primarily for personal use rather than business. Although thought it was akin to having a Raspberry Pi plugged in at home. Again, I may be completely wrong but why would you not use AWS/GCP? Even if it's complexity, Amazon have Lightsail, or if it's cost I thought DigitalOcean was one of the only reputable business-grade VPS providers. I just can't imagine many situations where a VPS would be superior to embracing the cloud and using cloud functions, containers, instances with autoscaling/load balancers etc.
- Symbiote 6y ago
  OVH is a European equivalent to Digital Ocean. It has twice the revenue, and is the third largest hosting provider in the world.
  - nixgeek 6y ago
    Twice the revenue of DigitalOcean still puts it < $1B ARR, or am I missing something? I can’t see how that’s the third largest in the world, or does your definition of “hosting provider” exclude clouds?
    - Symbiote 6y ago
      I took it from the top of their Wikipedia page. In any case, they aren't "primarily for personal use". https://en.wikipedia.org/wiki/OVH https://en.wikipedia.org/wiki/OVH

eb0la 6y ago

Are you truly paranoid? If my money and/or job depended on having something running without (or with minimal) disruption I would be as paranoid as you, too. BTW - Some people call this business recovery plan, not plain paranoia ;-)

buran77 6y ago
It's not paranoia if you're right. All of the risks GP is protecting against are things that happen to someone every day, and they should be seen like wearing the seat belt in a car.
- klingon78 6y ago
  I have a reliability and risk avoidance mindset, but I’ve had to stand back because my mental gas tank for trying to keep things going is near empty. I’ve really struggled working with others that either are both ignorant and apathetic about the business’s ability to deal with risk or believe that it’s their job to keep putting duct tape over the duct tape that breaks multiple times a day while users struggle. I like seeing these comments reminding others to a wear seat belt or have backups for their backups, but I don’t know whether I should care more about reliability. I work in an environment that’s a constant figurative fire. I also like to spend time with my family. I know it’s just a job, and it would be even if I were the only one responsible for it; that doesn’t negate the importance of reliability, but there is a balance. If you are dedicated to reliability, don’t let this deter you. Some have a full gas tank, which is great.
  - CraigJPerry 6y ago
    This resonates with me. I notice my gas tank rarely depletes because of technology. It doesn’t matter how brain dead the 00’s oracle forms app with absurd unsupported EDI submission excel thinga-ma-bob that requires a modem ... <fill in the rest of the dumspter fire as your imagination deems>. Making a tech stack safe is a fun challenge. Apathetic people though, that can be really tough going. It’s just that way “because”. Or my favourite “oh we don’t have permission to change that”, how about we make the case and get permission? _horrified looks_ sometimes followed by pitch forks.

venj 6y ago

are your domains at ovh too ? If yes, I'd consider changing this: this morning the manager was quite flooded and the DNS service was down for some time...

extrasolar 6y ago

this is the way. I do the same. Not paranoid at all.