11 ms·

Total data loss after botched GitOps and failed backups

• readthenotes1 3y ago
  It's odd to me when people say backups are a feature. They are not. Restoration, however, is. And has to be tested...
  • Thorrez 3y ago
    It sounds like they did test restorations, but didn't correctly verify the results. It's not very clear though. >While manual backup and restore tests were run once a month to ensure our backups were functioning, they were run manually.
    • ComputerGuru 3y ago
      They didn’t test the backups they were actually using, but a separate backup made completely separately to reassure themselves that <something>.
    • markburns 3y ago
      I think they are saying they manually tested what they thought the automated process was doing, then found out their assumption about what it was doing was wrong. So the manual tests were moot.
• macintux 3y ago
  The more layers of automation we add, the more invisible points of failure. Magic is great until it isn’t. I feel their pain. The sinking feeling when you realize that data is gone and not coming back is an awful experience.
  • noobermin 3y ago
    This is a dumb question, but why can't people just make things simpler? Tools are supposed to make work easier and make things more efficient, but this sort of complexity just seems to hurt, doesn't it?
    • livueta 3y ago
      I wonder if it has to do with mismatches between -as-cattle and -as-pets philosophies and usecases. From the post, it looks like they were using k8s. It's hard to tell given the, you know, total data loss, but it looks like the instance had at max triple-digit users. In my experience running small community stuff like this, the -as-cattle tooling, even though I'm familiar with it from work, is way overkill and introduces far more brittleness than it prevents. My go-to for stuff like this is -as-pets: gimme a box, a daemon, some rc scripts and maybe a backup client. Admin complexity and burnout are huge problems for volunteer sysadmins, and IMO the burden of -as-cattle tooling unnecessarily exacerbates both. I've had an irc daemon running on the same metal for almost 15 years now, uninterrupted besides occasional OS upgrades and patches. There is basically nothing else happening on the host. It is dumb simple and just works. Some of that may be on devs, actually: distribution-as-docker-container at the expense of real packaging seems to be on the rise.
      • noobermin 3y ago
        This sounds like a good explanation. As an outsider may be complexity seems unnecessary but it's needed for scale (an analogy is what is needed in a kitchen for a spaghetti meal to what is needed in a resturaunt to what is needed in a canned spaghetti factory), but problems can arise when you try to shoehorn methodologies and tools for scaled production to smaller systems I guess.
      • lamontcg 3y ago
        > Some of that may be on devs, actually: distribution-as-docker-container at the expense of real packaging seems to be on the rise. You could just run docker or podman or whatever alone on a VM and start/stop containers entirely manually.
      • tetha 3y ago
        At work, we're running a hybrid container/vm setup. We host the applications from our dev-teams as containers, since there is a lot of them and this allows them to do most of the work to integrate new apps with the stack. However, the actual data stores like postgres, glusterfs and such are simple VMs for a simple reason: It has less failure modes and many of the failure modes on a VM are less subtle and strange to deal with. And very few failure modes on a VM go straight to irrecoverable dataloss. In fact, the company I work at has deployed production like that for many years when our needs were simpler with just 1-2 applications. 1-4 application VMs provisioned by a config management, maybe a loadbalancer, maybe 1-2 DB VMs behind it. Worked like a charm until the new parent company started throwing loads of complexity at it.
  • deleted 3y ago
    [deleted]
  • inferiorhuman 3y ago
    The more layers of automation we add, the more invisible points of failure. Bullshit. Doing things by hand is more error prone. Without automation this situation would've played out like: well we got used to running things with e.g. --force or --yes or just hitting yes manually at every prompt. Unfortunately we just nuked our data store. Alternatively they would've looked at the dashboards, seen perhaps low CPU or memory utilization for the data store namespace and manually nuked it to save some money.. While this smells like a process issue it's mostly an architectural one. It's good that things were namespaced, however, for persistent data the infra should be more or less air gapped from a tooling POV. Updating the persistent data store should be a whole separate CI job, ideally with more intervention required to effect change.
    • politelemon 3y ago
      The opposite of complexity is not manual. It is possible to host a setup that is simpler than the one that was chosen.
• pdntspa 3y ago
  Wow... tooling that deletes a bunch of stuff because a manifest file is missing. Just... wow. I feel for the admin here. This seems equivalent to an accidental `sudo rm -rf /`.
  • kuchenbecker 3y ago
    Yeah, it's a hard lesson. We would always safe-expire or soft-delete to avoid this so each operation is undo-able for ~2 weeks.
    • nighmi 3y ago
      How do you set this up?
      • kuchenbecker 3y ago
        In a nutshell, we "abandon" the resource and clean up abandoned resources after N days. To delete something, you need to explicitly check it's name into a list of resources to delete. So deletions are explicit and a mistake is just un-abandoning.
  • defaultcompany 3y ago
    Ugh I did something like this once when I wrote a script to rsync one volume to a backup volume. Everything was great until someone accidentally rm -r ‘d the source volume and the script happily deleted everything on the backup volume to keep it in sync.
    • ryanjshaw 3y ago
      That was never a backup volume then, it was a mirrored volume. (Ask me how I learned the distinction.)
      • philipswood 3y ago
        How did you learn the distinction?
• birdyrooster 3y ago
  I don't understand the backup failure, how did you not notice your PVCs weren't being backed up? Like wouldn't you realize they were too small or happening too fast? It seems like the root cause is people not having their work checked from first principles.
• E39M5S62 3y ago
  Oof. I've been there. The rush of adrenaline and then just feeling completely wrecked and empty is something that sticks with you for a long time. Take care, lone admin - don't be too hard on yourself.
• neilv 3y ago
  Is it all fediverse data? If so, could the data be reconstructed from copies scattered amongst caches on other parties' servers?
  • sydbarrett74 3y ago
    Was wondering the same thing.
  • bronikowski 3y ago
    You could probably restore posts but with private keys gone you can't redo the instance, right?
• devmor 3y ago
  Automation is wonderful for creating things, configuring things and moving things. Automation should never "clean up". Do your cleanup manually, or this is what you get.
  • yowlingcat 3y ago
    Or said another way -- deletion is dangerous. I've observed that mixing the convenience of automation with the risk of hard deletion is fraught with peril. If you really need automation around deletion, it's best to set up roadblocks and use tombstoning approaches, where archive/backup data is sent to S3 Glacier or something of that ilk. Storage is cheap enough these days that there's no reason not to.
  • sp332 3y ago
    The point of the tooling is that you describe what you want your deploy to look like, and it updates the deploy to match the description. If you delete something from the description and it stays running, that would be very confusing.
    • Dylan16807 3y ago
      Running is different from existing.
    • jakelazaroff 3y ago
      I haven’t used their particular tool (ArgoCD) but the ones I have used include an option to keep specific pieces of infrastructure around even when it’s deleted from the description. That’s absolutely what you should be doing for anything that stores data.
      • fhaldridge7 3y ago
        ArgoCD will also not delete resources by default when the file is removed from the git repo. The author explicitly enabled auto-pruning.
    • viraptor 3y ago
• bg24 3y ago
  Very sad. But also a miss on admin's part, even if unfortunate. They did not realize during manual restore testing that volume was not being backed up. "Yes and also apparently no. We use #Velero to capture backups of our cluster every 6 hours. From what I had seen our backups had been running successfully. I discovered once the incident started that backups had captured everything but the Persistent Volume Claim data. While manual backup and restore tests were run once a month to ensure our backups were functioning, they were run manually. After digging into why our restores were not coming up with data, I found that our recurring backups were missing the flag to run volume backups with Restic which snapshots PVC block volume data."
  • belltaco 3y ago
    > manual restore testing They didn't test restores of the actual data being backed up. They did a manual restore test of data that they manually backed up.
    • appplication 3y ago
      I want to feel bad for them, and I do, but really I’m not sure what you expect. They never tested their backup. They tested their ability to manually back up, but I’m not sure why they would ever think to conflate that with automatic processes. If they had automatic backups, why were they relying on manual ones for testing?
      • gabereiser 3y ago
        Yeah. Kudos to being open about it but this is an amateur mistake. I ran cloud infrastructure for large companies, small companies, and small companies with large data. Disaster Recovery, Disaster Response, Data Backups, Time to restore, etc are all table stakes to running anything in “production”. They tested their ability to do a backup, but never went through with actually backing it up - automating it - let alone making it a process to do before you make changes in “production”. Amateur. I will give this credit though. It’s hard. Not everyone can do it. Running production that is. Anyone can build for production but actually keeping it live, running, and working nominally with proper SLA and DR? It’s hard. I’m curious what the volumes were even for? I tend to not use storage outside of blob, databases, queues.
• pbjtime 3y ago
  Anyone who would feel a "crippling loss" if their data disappeared must execute a real world real failure test at least annually. Disconnect the storage, attach new storage, and go.
• stym06 3y ago
  That sucks! Now I'm thinking my github action that is doing a `git pull repo && cd repo/ && rm -rf dist/ && populate.sh && git push` in my bash script will totally go rogue one day and kill everything. Any idea, peeps?
  • bagels 3y ago
    Separate your code from your data. Be very careful with your data, and make sure you have backups of your data and that you test and validate your backup procedure. At the very least, verify the integrity of your data backups.
  • deleted 3y ago
    [deleted]
• scarface_74 3y ago
  The only data that I really care about are my photos and videos and they are backed up to four different providers. But this also made me realize how upset I would be if my blog over at micro.blog was accidentally deleted. It’s more of a journal of my digital nomadding with my wife across the US than anything else. I immediately went over and started a JSON export in bar (?) format.
  • andruby 3y ago
    If it is a public blog, make sure the url is in the wayback machine as an extra external backup
    • RetroTechie 3y ago
      Archive.org often captures sites excluding certain media like images or video. So you'd think of that as an extra backup. But in reality might be just html, text-based stuff, or some other subset of file types.
• simula67 3y ago
  > While manual backup and restore tests were run once a month to ensure our backups were functioning, they were run manually. After digging into why our restores were not coming up with data, I found that our recurring backups were missing the flag to run volume backups with Restic which snapshots PVC block volume data. Can someone explain this? How did they test restores, if the actual restore failed to come up with data?
  • deleted 3y ago
    [deleted]
  • depereo 3y ago
    Velero can be configured to run on a schedule, but the scheduled command was apparently not the exact same command they were using to perform the manual tests - the scheduled job was missing part of the command, basically.
    • VectorLock 3y ago
      Sp they were manually doing a backup and then testing that backup, rather than testing their automated backups? If thats what they were doing that just makes me wonder... why?
      • fbdab103 3y ago
        I am not sure about the pricing structure of their provider, but maybe it is a Hotel California situation? Data egress price of the provider was such that they did not want to pay to retrieve the full backup "just for a test"? Instead, repeat the backup command, but swap it out for a local location.
      • depereo 3y ago
        It's possible that the backup restore/test process was still 'automated' to some degree, like a manually-run pipeline or playbook etc. Obviously there were mistakes made in process design and tools deployment but I don't think this is 'baffling incompetence' more than it is a couple of small mistakes that compounded each other to have a pretty spectacular impact.
• pnw 3y ago
  That sucks. Everyone has been there. On my first sysadmin job I did an rm /* as root and then discovered my backups across the network to a remote tape unit had a buffering issue and were basically useless.
  • nicbou 3y ago
    I know that this is the time to say "you should have tested your backups", but I never tested mine. How does one test a (personal) backup strategy?
    • LoganDark 3y ago
      > How does one test a (personal) backup strategy? Boot into one. (Clone it onto a spare drive and use that as a boot target.) If it works flawlessly, there's your backup. If it doesn't, fix your backups so they're bootable.
    • hnbear 3y ago
      Personally my personal files like important docs go to: - iCloud (automatically) - Local nightly rsync to a zfs share from local Docs folder - Backblaze - Some crit stuff to OneDrive - TimeMachine on a local Linux server at home Periodically I check them all, verify file counts and sizes, spot check files. Periodically isn’t that often, but I’ve actually needed backed up files and used it. Data volume are low though. I keep Photos libraries similarly backed up. That’s harder to verify, although sizes is pretty good. Even if the library metadata gets fried the raw files still exist in the folder structure.
    • j7ake 3y ago
      Buy an extra computer that is blank and periodically restore your back ups into the extra computer.
      • politelemon 3y ago
        On similar lines, I like to reformat every year or so at least. It forces a backup test.
• fnord77 3y ago
  something like this almost happened where I worked - an atlantis/terraform apply started tearing down our cassandra cluster, which was not backed up. Caught it in time before it got the volumes
  • mdaniel 3y ago
    You're in good company: https://news.ycombinator.com/item?id=36665004 https://news.ycombinator.com/item?id=36665004 The ol' adage "to err is human, to wipe out production requires <s>automation</s> terraform"
  • Anon_Forever 3y ago
    Terraform has had the "prevent_destroy" functionality for quite some time. Highly suggest using it for everything you're going to miss if it's gone. [0] https://developer.hashicorp.com/terraform/language/meta-arguments/lifecycle#prevent_destroy https://developer.hashicorp.com/terraform/language/meta-argu...
• oneepic 3y ago
  >I won’t personally be bringing back outdoors.lgbt or firefish.lgbt. Being an admin has been one of the most fulfilling things I have done in a long time and you all have made it such an amazing experience., however, I need to take a step back. I would love to hand the domains over to someone with a similar passion for creating a safe and welcoming community. I hope the author isn't only quitting because of a feeling of guilt or shame. If that's true, it's certainly ok to give it another try. Tech is toxic enough depending on where you look, and that's not even counting the self-hurt that some of us do to ourselves.
• lamontcg 3y ago
  This all seems way overly complicated for what could probably be a few services running on a VM or four. Why does Kubernetes/Argo/Helm need to get involved? Why couldn't the whole architecture diagram look a lot more like HN? I feel like we've entirely lost our way with complexity.
  • havnagiggle 3y ago
    IMO it's because of this default in k8s storage class: https://kubernetes.io/docs/concepts/storage/storage-classes/#reclaim-policy https://kubernetes.io/docs/concepts/storage/storage-classes/.... Almost everything in a k8s cluster can be ephemeral... except your data! But they make the defaults insane and it eats a lot of people's lunches. It's honestly very sad. I fortunately haven't "lost" data in 25 years, but I still remember the pain when I made this kind of mistake.
    • Kadin 3y ago
      That explains why the K8S defaults are bad, but why use K8S at all here? Personally it seems like a lot of shade-tree server admins are way too eager to bust out much more tooling than is actually necessary for many tasks, just because it's the way that $BIGNAME company does it. Realistically, most people running a personal or small community's IT system aren't going to need the sort of crazy scalability that container orchestration systems are designed to provide. And if you're not running them at significant scale, the containerization and orchestration-system overhead are often quite significant fractions of the overall resource footprint. Plus there's the unnecessary complexity due to all the levels of abstraction that just don't need to be there. E.g. for a Mastodon instance, I wouldn't touch Kubernetes or complex orchestration unless I was out of other, more traditional options for scaling. The server side is already broken into a number of components (RoR app, PosgreSQL, Redis, Sidekiq, node.js) which you can separate out onto their own servers, and from there each component has preferred ways of scaling based on need. And while nothing is safe from failure, backing up a bog-standard PostgreSQL VPS is a lot more straightforward than K8S. If you're doing deployments dozens of times a day, of course automation is desirable. But if you're doing it once, I suspect most people would be hard-pressed to make back the investment of time and additional testing required (well, should be required) of working through a complex container-management architecture.
• gorgoiler 3y ago
  Kudos to the admin for writing this up and making it public. Everyone’s lives will involve an irretrievable loss of something, on some scale, at some point. A friend, partner, parent, child — those are the big ones. Your home, job, your pet, a precious object (or in this case, data), or an enjoyment of something: the accumulation can be a lot to handle as time goes on, but it’s also human nature, and it’s important to learn to tackle grief without ignoring it. Grief is so core to being human that, thankfully, it is an aspect of life where you can find a lot of support. Religious communities, for example, can provide a lot of help that at the same time will be orthogonal to their core mission of worship. You can benefit from the former without having to engage with the latter. Don’t worry about being transactional in looking for help — people will want to help you.
• futuretaint 3y ago
  hey, it's a mistake the admin will never make again, so. be a cup half full person... you don't have to worry about the data any more also.
• serpix 3y ago
  This would also mean they did not have a staging environment for verifying a botched deployment. Very unfortunate and a double disappointment for finding out backups were missing.
• SeanAnderson 3y ago
  Off-topic, but this website has some pretty crazy dark-patterns! I went to the user's profile and right-clicked their photo. The contextmenu action is intercepted and replaced with a custom menu mimicking the browser's context menu. This fake context menu has the option "Open in Window". I click it. The website doubles-down on the charade and opens a "popup" inside itself - maximize/minimize icons in the action bar and everything. It's the sort of thing you're taught phishing websites do to lure you into inputting your credentials into the wrong website. https://i.imgur.com/lMiH3uH.png https://i.imgur.com/lMiH3uH.png
  • bentinata 3y ago
    It's not a website anymore. It's a web app. Might be related: I recently joined a company using Flutter web and notices similar stuff. Yes, the software engineers are wondering: "Why do we reimplement browser stuff?" But the business ask for it and we don't hear complaints from the end user.
    • omniglottal 3y ago
      "We don't hear complaints" != "there are no complaints"
  • gnyman 3y ago
    That's not a "dark pattern", it's just the result of of making the browser a OS and giving websites free rein of anything inside. There is an aptly named browser extension to help combat this called StopTheMadness [1] , I recommend it. Let's try to save the term Dark Patterns for the things that that deserve it, where the goal is to steer the user to make a unfavourable choice at the benefit of someone else. https://underpassapp.com/StopTheMadness/ https://underpassapp.com/StopTheMadness/
    • Hakkin 3y ago
      Firefox has a built-in method to get around context menu blocking by holding down the "shift" key while right clicking. I wish there were similar built-in overrides for things like blocking selecting text and copying, you commonly find stuff like this on corporate blogs or lyric sites where they want to "protect" their text content, of course you can just get around it by opening the page HTML and copying from there.
• shnkr 3y ago
  Disclaimer: Author to a competitive k8s backup solution. I personally don't think Velero is a solution for production workloads or anything serious. Only an established backup company/devs will have expertise to implement and handle all cases and take care of all data loss scenarios. Ideally k8s authors should have stopped at providing a tool (they have snapshots like any db/fs) rather than writing their own backup piece. Unfortunately many of the industry solutions are wrapped over Velero except a few (two?), one I implemented from scratch for Commvault.
• puffyblogger 3y ago
  [flagged]
• mkl95 3y ago
  > We use #Velero to capture backups of our cluster every 6 hours. From what I had seen our backups had been running successfully. I discovered once the incident started that backups had captured everything but the Persistent Volume Claim data This is why for non hobby stuff I advocate for RDS, or its counterpart on your favorite provider. Running a production db on Kubernetes is looking for trouble unless you really, really know what you are doing.
  • speedgoose 3y ago
    I would also advice to have some backups available outside of your favourite cloud provider. Your cloud provider may fuckup, it has happen many times. You may encounter issues such as your account deleted or temporarily unavailable. Or the whole datacenter and many zones are on fire.
    • jrockway 3y ago
      For example, the "delete" button on GCP deletes the backups along with the database you're deleting. I would highly recommend not relying on those backups for anything mission-critical. Transfering the backups to another cloud provider is essential if your risk model involves losing your cloud account somehow.
  • marcus_holmes 3y ago
    We had a database downtime on RDS. I queried it, because we were paying good money for a failover database. We were told that the downtime was because the database was being updated, and all databases are upgraded at the same time so the failover was also being upgraded. Never using RDS again. I'll stick with running Postgres on something I can actually manage myself.
    • sofixa 3y ago
      That sounds unlike my experience with AWS RDS, and I don't think there would be many users of it if it were generally true. When was this and was it AWS?
• pech0rin 3y ago
  Very sorry to hear about this. Working in ops for 10+ years I know that drop kick in the stomach feeling when everything is going wrong and nothing is working as expected. I hope the author continues forward and takes this as a lesson. My own two cents which others have echoed is this is extremely overly complex setup for this situation. Its been told 100x times on HN about this but there is a reason. The more complex a system is the more points of failure you have. For all the tools you were running you should have had a team of ops people. In reality a database, some servers and a load balancer would get you 99% of the way there. Modern engineering is a dumpster fire of complexity mostly hocked by shills working to sell contracts to enterprises. You have learned this lesson the hard way but less moving parts means less things that can go wrong. Basics like monitoring, backup, testing etc go only so far if your system is a rube goldberg machine. Hope as an industry someday we can get back to simplicity
  • trustingtrust 3y ago
    > Hope as an industry someday we can get back to simplicity The services upon services with no idea what underlying layers are doing and pretty much enterprises collaborating to get stuff 'out the door' means we may not go back to simplicity easily. At this point I am only finding work who are basically middleware and rarely doing something unique even there. Basically getting contracts and servicing a business need is all that I see when I try to find jobs these days. Initially I was doing something quite unique in networking but now that the client needs have changed its basically just another product that is servicing clients which big companies like Arista can't do for them as there are custom requirements.
  • JeremyNT 3y ago
    You're right of course that complexity makes it harder to reason about, which makes it easy to "fat finger" a destructive operation with unintended consequences. However I'd say the truly disastrous issue here wasn't really that. Data loss is something we've always had to plan around, whether due to mechanical failure, bug, or operator error. The actual death knell here was a classic ops blunder: > From what I had seen our backups had been running successfully. I discovered once the incident started that backups had captured everything but the Persistent Volume Claim data. While manual backup and restore tests were run once a month to ensure our backups were functioning, they were run manually. After digging into why our restores were not coming up with data, I found that our recurring backups were missing the flag to run volume backups with Restic which snapshots PVC block volume data. So they weren't actually backing up the data they thought they were, which is a class of error that is familiar to everybody who has been working in ops for a while. Unless you test the restore of an actual backup process, you have no idea whether you have useful backups.
• deleted 3y ago
  [deleted]
• TekMol 3y ago
  I have been building and running web applications which are used by millions of users for over 20 years now. But reading this post, I feel like I am looking into a completely different world. None of the following is something which ever crossed my path: - GitOps repository cleanup - yaml manifests that create our namespaces - ArgoCD - Helm deployments - Persistent Volume Claims - Velero - Restic - PVC block volume data - Vultr I simply write data to MariaDB (a fork of MySql) and back it up via mysqldump. I wonder if that would have been an option here too?
  • sofixa 3y ago
    I can see the point you're trying to make, but you're choosing an interesting way of conveying it, and I disagree. Restic is a backup tool. Velero is a backup tool for Kubernetes. Vultr is a low cost but still decent "cloud" provider. GitOps is a philosophy which makes sense even on small projects. None of those are "wrong" or "overcomplicated" options. The elephant in the room is Kubernetes, which is indeed quite complex, and often gets used as the go-to even where it doesn't make sense [1] either because it's popular or because that's what people know, or because of the ready-made tools from others (e.g. if all you want to deploy exists in the form of Helm charts, it can save you lots of time) but it has its place and brings a lot to the table. You just have to be aware of the risks the complexity brings. Disclaimer: I work at HashiCorp, I'm a massive fan of Nomad and think it's a better fit than Kubernetes in many cases, but dismissing Kubernetes outright is wrong. 1 - https://atodorov.me/img/nomad/kubernetes.jpg#center https://atodorov.me/img/nomad/kubernetes.jpg#center
    • politelemon 3y ago
      > You just have to be aware of the risks the complexity brings. I'd say most are not aware. It is not enough to say that people should "just" be aware. That is something said from a position of knowledge and awareness, which helps nobody. Completely agree with the parent comment, simplicity is the first thing that should be reached for. K8s ought to be dismissed by default, because then you have to justify its inclusion. That's probably a way to increase awareness before plunging in.
    • 3y ago
• dusted 3y ago
  While my knee-jerk reaction is to blame bad backup practices, that's not what I believe happened here. Not entirely at least. The setup is too complex to easily verify and restore. Yes, a bespoke box is annoying and "inelegant" but it also works, and if it's backed up following even 80's tier best practice, it can be restored by anyone with a pulse.
• lifty 3y ago
  As DevOps Borat put it: “To make error is human. To propagate error to all server in automatic way is devops.” Sorry this happened, I’m sure it’s gut wrenching.
• huksley 3y ago
  that's why DB should a permanent instance out of reach of CI/CD
• hnarn 3y ago
  I’m trying very hard to not be flippant here, but I can’t shake the feeling that this Kubernetes norm has to end. I’m not saying Kubernetes needs to disappear, but people need to stop treating it as the new normal, as if VM:s and config management is somehow an outdated and incapable alternative. To me, this is an example of the complexity of Kubernetes coming back to blow your foot off. Remember, Kubernetes exists to make scaling and redundancy easier, but it’s only easier if you fully understand the implications of every configuration that you make. Complexity causes incidents, so my mantra will always be that if you propose to introduce complexity, have a justification ready for why its inherent risks are outweighed by the benefits. If you’re deploying your side projects in an infrastructure this complex, I would strongly suggest taking a step back and questioning if the same benefits really couldn’t be achieved in a simpler way.
  • CommanderData 3y ago
    If the application is containerised, like most are these days, how do you propose running these on said VM? If we're talking about something a little more complicated then a single container, then you arrive at the same problem Kubernetes is solving.
    • aequitas 3y ago
      I have setups where the container orchestration was done completely by Terraform or Puppet. Docker Compose also comes a long way if you are not running a FAANG webscale service. If you manage your VM's basics with configuration management and infrastructure as code anyways, it's easy to bolt containers onto that and treat them like any other installable piece of code. Kubernetes does bring a nice abstraction if you have DevOps teams and separate Ops for your bare metal or Cloud.
    • hnarn 3y ago
      It's a very complex discussion, almost complex enough to be unsuitable for text, because for every reply ten more "what-ifs" seem to open up. It's hard to be general enough that what you're saying is useful, while being specific enough that you're actually answering the question. If we're talking about a multi-container application with moving parts, then yes, Kubernetes is suitable exactly because it was created to resolve the issues arising from that architecture. But I don't think it's fair to start a general discussion about architecture from a point of "Step 0: The application is made to run in Kubernetes". By the way, is the statement that "most applications today are containerized" really true? I would claim that you can only count applications into this category that are published as "containerized by default", and to me that certainly doesn't seem true. I've encountered a few, but they're exceedingly rare. If you also count "applications that are available as a container", then sure, but then the statement should be that "most applications today *can* be containerized", or they are "available" in containerized form. Whether something is "more complicated than a single container" depends not only on the application, but how you choose to deploy it. If you take a hypothetical web application running nginx for PHP with a Postgres backend, you can deploy both of those as containers, but you certainly don't have to. You could skip using containers entirely, or you could deploy the nginx component as unmanaged containers as if they were an RPM/DEB-package, while not involving containers for your database at all. Yes, it requires more configuration for all the things that are no longer "magic", but with static infrastructure the work you put in comes back to you as "simplicity": the moving parts are easier to understand, which means problems are easier to troubleshoot. Yes, you will lose "auto-healing", but you also lose "auto-nuking", and so on. I want to be clear again that I'm not saying Kubernetes is objectively bad, but I am saying that I think a lot of people using it have not considered whether it's a suitable solution for their current problems (never-mind imaginary, future problems). To me, it's like comparing a lawn mower to the space shuttle, and then trying to motivate building the latter by saying that a lawn mower will never be able to get into space: if you're running a web site that is looking at a few hundred thousand users, you're still (probably) not "going to space". Fundamentally, I think the over-use of Kubernetes is nothing more than an example of premature optimization. The idea goes something like: "If we become the most successful /thing/ in the world, the infrastructure is already made in such a way that we can scale" -- but there are no decisions when it comes to infrastructure where you get the pros but can escape the cons, and those cons for Kubernetes specifically seem to rarely be discussed out of fear of sounding like a Luddite.
• testemailfordg2 3y ago
  One learning for all of us that I see every other day at work causing issues, the testing is not planned or not done based on what is present in production but something which makes the tester feel...Works for me.... Probably need a similar solution like Docker for this works for me disease in testing.
• tamimio 3y ago
  Honestly, it shouldn’t feel that bad, what was the loss again here? A bunch of users and their posts/memes?! They can create other users it should not be an issue. The real fuckups IMO are stuff that involves people’s lives, things like industrial automation, robotics, autonomous vehicles, aircraft systems and what not, unfortunately, usually you don’t see such an accountability like in OP when things go south in those domains.
  • manifoldgeo 3y ago
    It sounds like this failure led to the complete destruction of 3 online communities. While this might not be as bad as, say, a fatal multi-car accident on the highway, I bet it led to some heavy grief as users mourned the losses of their communities. It might not sound like a big deal because it's just "a bunch of users and their posts/memes", but many people get the vast majority of their social interactions through social media. A sudden, total loss of such a site could be devastating to someone's social life. They might've lost contact with an intimate friend group they knew only via aliases with no other way to reach out. Even for older generations like Baby Boomers, life is an increasingly online experience: - How many senior citizens are reliant on Facebook for tons of their daily interactions with old friends? Probably a lot! - Entire degrees from reputable universities can be earned fully remotely, and the transcripts, curricula, forum posts, and even proof that you earned such a degree exist in databases and distributed systems vulnerable to the exact same types of flaws outlined in the OP. - Dating apps like Tinder / Bumble connect people who form long-lasting and loving relationships, and the sudden loss of such platforms could quickly end your potential lifelong romance. - LinkedIn networks are of huge importance to some for continued, gainful employment. Are there memes there? Sure! But does that mean it's "just a bunch of posts and memes" whose loss would be fine? Hardly, in my opinion. My use of the above platforms as examples is not an endorsement of them, BTW. I just wanted to illustrate that the position you're defending will become more and more indefensible as people's lives become increasingly enmeshed in the digital world. Socializing online is still socializing, and a loss can still be painful even if there are memes and usernames involved.
• roydivision 3y ago
  Props to the author for being so open and honest, and for writing it up as a cautionary tail. None of us are perfect, thankfully. The stigma around failing should be eradicated from society.
• hyperman1 3y ago
  Backup and restores are hard. Last week, I upgraded my home laptop's SSD. I decided to take out the old one, do a fresh install on the new one, and restore the backup from a Synology NAS. The good news: Backup seems complete. It's not the first time I migrated like this, and an older hardware upgrade went a lot less well, so I am happy. The bad news: It was far from a trouble free process. I succeeded in the end, but had to restore a few times because the first tries were botched and there were some gotchas to learn.
• kozzz 3y ago
  At least they had fun with trendy tech. Who wants to ssh to servers any more? Throw in ten different tools you barely read documentation for, season it up with some yaml you found on gist.github and join the cool guys!
• mmcnl 3y ago
  Unfortunately, anything that's powerful enough to simplify complex configuration is also powerful enough to accidentally make fatal errors.
• zzyzxd 3y ago
  Here we go again, a lot of comments saying k8s and gitops are too complex. IMHO, your ops team can accidentally delete your data one way or another regardless. Today it was moving files to a wrong directory in git, yesterday it could have been human executing `rm -rf` against the wrong path. The highlighted item here should be the bad backup process. You can't say your data is backed up unless you actually verify successful restoration from the backup data, on a regular basis. This was true 20 years ago for your SQL database instance, and this is still true today for your k8s PVCs. FWIW, I have been running GitOps long enough that any PR that moves files around raise the highest alerts in my head. The fundamental issue I have seen in many places is that engineers stores infra code in git and call it "GitOps". When you adopt GitOps concept, the most important thing is to train your engineers to switch to a different mental model, where your git repo is the _desired_ state of your infra -- it's not the actual state of the infra, and it's not a store of your imperative commands to manipulate the infra. When your desired state of your namespace is to not to exist, your gitops engine will try to make that happen!
• samcat116 3y ago
  If this was with Argo shouldn’t finalizes prevented this?
• redhale 3y ago
  Here we go again, a lot of comments saying the giant ball of knives is too dangerous. I work with the giant ball of knives every day, wrapped in my custom-tailored full-body kevlar bubble, and I barely ever have any catastrophic life-threatening accidents! The giant ball of knives is not the issue. Your carelessness when working around the giant ball of knives is the issue. Look inward.