55 ms·

Gandi loses data, customers told to use their own backups

• webrobots 7y ago
  Dear customer, This mail is a follow-up to the previous email we sent (on January 8th, 2020) on this topic. As a reminder, yesterday, we experienced an incident on a storage unit at our LU-BI1 datacenter, located in Luxembourg. Despite the replication systems in place, and the combined efforts of our technical teams throughout the night, we were unable to reover the data that was lost on the impacted storage unit. We sincerely apologize for the inconvenience that this situation has caused. This type of incident is extremely rare in the web hosting industry. In the event that you have a backup of your data, we suggest that you to use it to recreate your server at a different datacenter. To help you in this, we have provided you with a promo code that will give you one free month for an instance, so that you can create a new Simple Hosting instance in a different datacenter: XXX
  • petee 7y ago
    Wow, for a company that boasts "no bullshit", only offering a month after destroying data and backups seems a little tone deaf Edit: in fairness, I'm not sure how exactly you would quantify such a loss anyway...
    • Shalle135 7y ago
      It sounds like they didn’t have any backups at all but rather relied on a active-active replication link to a secondary storage. Edit: who knows it may be related to the HPE issue. https://www.bleepingcomputer.com/news/hardware/hp-warns-that-some-ssd-drives-will-fail-at-32-768-hours-of-use/ https://www.bleepingcomputer.com/news/hardware/hp-warns-that...
      • jnwatson 7y ago
        In other words, RAID is not backup.
      • mschuster91 7y ago
        What baffles me is that there seems to be no way for either the customer or a data-recovery company to flash a new firmware onto the drive after it has failed. Someone there wanted to spare the few millicents of copper trace for a JTAG port?!
        • marcinzm 7y ago
          Probably to prevent supply chain firmware changes for hacking, espionage, etc.
      • deelowe 7y ago
        Hmm... I wonder what the "incident" was. If it involved something akin to an "rm -rf," then of course their replication link didn't protect them.
      • fencepost 7y ago
        Perhaps they were depending on snapshotting and were not prepared for some kind of hardware failure taking out the entire storage system.
    • joepie91_ 7y ago
      Reputable hosting providers typically don't try to quantify such a loss, but rather outright offer a credit/compensation that is very obviously generous (say, a year or even two of free service). Especially when a small set of your customerbase is affected, it won't cost you that much, and "overcompensating" like that means that virtually noone is going to criticize you for quantifying it wrong; instead, the public narrative will be centered around "well, shit happens, they did their best and generously compensated".
  • mmoez 7y ago
    A promo code in exchange of your data loss. What a bargain!
    • atlasAUPrivacy 7y ago
      “Please keep trusting us to host your data”
      • scohesc 7y ago
        You really shouldn't trust anyone hosting your data. Always have backups!
        • oefrha 7y ago
          Often times the backup provider is the hosting provider, whom you have to trust. (This extends all the way from big clouds like AWS and GCE to small providers like Linode and DO). Having an external backup can be unreasonably expensive due to ridiculous egress costs.
          • arpa 7y ago
            You can still back up to the same providers' different data center. Two data centers failing simultaneously is very unlikely.
            • oefrha 7y ago
              Not always an option. For instance, I use Linode’s backup service and it can only back up to the same data center (although it is said to live on a separate system).
              • jdmguit 7y ago
                Coming from a Linode employee, I can confirm this is true. Linode's backups live in the same data center as the server, but the systems are separated so that they don't directly affect one another.
  • hedora 7y ago
    Interesting. The public status page says they’re still waiting for the recovery process to complete.
  • manuelmagic 7y ago
    I could understand the incident (I would _at least_ start questioning myself about the quality of the service I'm paying), but IMHO this is not something that can be addressed with a casual e-mail that contains few lines of excuses and a "promo code" like it's everyday business. That's astonishing. Worse than a bad incident there is only bad management of the following situation.
  • abtinf 7y ago
    > This type of incident is extremely rare in the web hosting industry. Why would they include that sentence? Are they trying to imply it is rare for them because it is rare for the industry? Are they saying they are not as good as the industry, so customers should move to other providers? Or are they trying to show they apply the same inattention to their customer communication as they apply to their data backup/recovery practices? This kind of data loss should simply never happen. It’s one thing to say “it will take us up to 30 days to restore your data because our fast recovery options aren’t working and we have to bring up cold archives”, it’s entirely another to say “your data is gone, tough”.
    • SmellyGeekBoy 7y ago
      I'm not sure why you've been downvoted for this. I thought the same. I read it as: "This type of incident is extremely rare in the web hosting industry, because apparently the overwhelming majority of our competitors aren't capable of fucking up as badly as we just did." Doesn't inspire confidence at all, IMO.
    • ben509 7y ago
      > Why would they include that sentence? They're a French company; it may be a non-native speaker not catching the implication. It's also possibly an editing error, e.g. they started writing something like, "these types of incidents are extremely rare and when they happen etc" and most of it was dropped without considering how that changed the implication.
    • jermops 7y ago
      If you're not paying for backups... what archive?
      • icebraining 7y ago
        They say you can backup by using their snapshoting tool, but they lost those snapshots too.
        • yjftsjthsd-h 7y ago
  • passivepinetree 7y ago
    Is this a response from the company or are you putting it forth as an example response for how to handle this incident better? It’s unclear from your post.
  • jandrese 7y ago
    > This type of incident is extremely rare in the web hosting industry. I read this as "so maybe you should consider one of the other web hosting companies that doesn't have problems like this."
• djmobley 7y ago
  Gandi have something of a cult following, but in my only experience with them they literally lost my domain name during an inbound transfer. Their response was awful and rude and completely unprofessional. I never got my domain back. Based on that experience, this incident doesn’t surprise me at all.
  • quaquaqua1 7y ago
    In the year 2020, it's becoming increasingly impossible to trust anyone to do nearly anything (in my opinion of course). The courts are too expensive. The culture of taking pride in one's work maybe is disappearing. For the most crucial parts of doing business/living life, we are required to trust someone else. For example, I can't just go and make my own cell phone tower or ICANN. And yet I can't even trust those entities to get it right.
    • specialist 7y ago
      Decreasing trust increases transaction costs. There's got to be a measurable (negative) economic impact.
  • vegannet 7y ago
    I’ve always been a little confused about their cult following given their unfriendly terms — arbitrary domain cancellation based on adult material for example — which are fair terms to have if that’s their ethics but it seems at odds with the typical pro freedom expectations many people in technology hold.
    • tgsovlerkhgsel 7y ago
      Any details on this? All I found while searching for this was Gandi explicitly advertising gTLDs designed for adult content... Do they have that in their terms? Independently of that, do they have a history of doing that?
    • SmellyGeekBoy 7y ago
      They put a rude word on their homepage, that makes them edgy and cool and anti-corporation!
• anarcat 7y ago
  I understand people might be upset because they lost data, but as a sysadmin, my reaction is "ooh shit, poor guys, that must be a horrible week"... And honestly, if you don't keep data of stuff you host on a server provider like this, you kind of get what you deserve...
  • bigwavedave 7y ago
    > You get what you deserve Sure, let's blame the victims here; that's effective and helpful.
    • anarcat 7y ago
      Oh god, the victims, really? You host your data on someone else's computer to save on costs and get rid of the burden of dealing with metal and stabbing yourself with screwdrivers , and you're the victim when they fuckup? Give me a break... It's not like anyone died here. There's a reason I host my own shit. Problems happen, errors are made, and data is lost. It's also your responsibility to deal with data permanence, even if your provider has all the promises in the world.
      • cblades 7y ago
        > You host your data on someone else's computer to save on costs and get rid of the burden of dealing with metal and stabbing yourself with screwdrivers , and you're the victim when they fuckup? A company violates their agreement with you in a way that costs you time, money, and potentially business, and you're not the victim?
      • yibg 7y ago
        Well yea that’s why you pay them, to do a job. That payment comes with certain expectations and when they aren’t met you incur cost. In this case downtown and effort and time to restore from your own backup. Victim may be a bit strong but of course it’s Gandi’s fault and not their customers’.
      • rvz 7y ago
        Exactly, its like you somehow give your original private keys to a cloud hosting provider or a service like Gandi, they have a problem and lose your mission critical data and you later blame them for their responsibility. They are fools on their side for failing to preserve user data, but you end up being the bigger fool for trusting them to do this for you without preserving a backup plan yourself.
• Eikon 7y ago
  > We sincerely apologize for the inconvenience that this situation has caused. This type of incident is extremely rare in the web hosting industry. Why are they speaking of the "industry" as a whole when they are to blame? It's even crazier they are not even explaining the source of the data loss and why the "replication systems" didn't help. IHMO they are trying to sweep this event under the carpet. They should instead explain why they should be trusted in the future and why this would not occur again.
  • briffle 7y ago
    In the last few years, I have seen many people confuse replication with backups. People see them as the same thing, but they really aren't. Even with snapshots, if the devices are the same, they might have the same firmware bug, etc.
    • deleted 7y ago
      [deleted]
    • dv_dt 7y ago
      Just to further explore that a bit, would you say replication adds independent copies for failures of media, while backup adds copies made by independent software against failures of process / software / media.
      • slrz 7y ago
        Replication covers durability (and availability) in face of system or media failure but does nothing whatsoever against software bugs or human error.
      • abtinf 7y ago
        Replication increases risk of data loss when implemented incorrectly, because added resources increase the probability of bit errors. This applies to both replicated disks (RAID) and servers. Replicated servers must use ECC memory as well as checksum blocks and periodically scrub data to ensure integrity (e.g. what ZFS does for you). If they don't then a bit error corrupts the data on all servers, because you have no way of know which copies are pristine or how to piece together pristine parts.
• aosaigh 7y ago
  I don't have hosting with Gandi, but I do use them for domains and DNS. I'll be considering migrating my domains from them after this. Their response to this is exceptionally poor. To say essentially "this could happen to any other web host" it nonsense. I've never had this happen with any of the providers I've used for hosting and I'd be very angry if I had just lost an entire VPS. The fact that they've lost all snapshots as well (which are advertised as backups of the underlying volume) is unforgiveable.
  • netule 7y ago
    I've been burnt several times now by smaller players claiming a higher degree of privacy that suddenly charge high fees, sell to a competitor, or sell my data. As of last month, I've moved my domains to Google. Better the devil you know than the devil you don't.
  • eg312 7y ago
    I use Gandi for domains & DNS too. I've never had any problems so far but I don't want any surprises... Where do you want to migrate? What is a better alternative?
    • Grimm665 7y ago
      I use Hover for DNS and domain registration, never had a problem and their interface and support is top-notch.
    • danr4 7y ago
      Same here. Wondering what alternative there is. Heard good things about https://porkbun.com/ https://porkbun.com/
      • jdhawk 7y ago
        I wish porkbun allowed easier DNS record management. It's very cute, but I cant edit a bind style file, which means a lot of extra clicks.
      • donmcronald 7y ago
• babycake 7y ago
  What kind of data was affected? Was Email messages affected?
• EnderMB 7y ago
  Oof, this Twitter thread looks particularly bad, especially the response from the official Gandi account. https://twitter.com/andreaganduglia/status/1215199147701231616 https://twitter.com/andreaganduglia/status/12151991477012316... While I appreciate that there are real people behind these companies that are probably having a really rough time right now, the criticism that Gandi are getting as a company is justified - and if Gandi are truly a "no bullshit" company they need to put something out to their customers asap.
  • binarysaurus 7y ago
    Julie Pelloille, responsible for comms, appears to be going a bit too far with this.
    • rhacker 7y ago
      The Cersi thing is wrong on so many levels.
    • mprev 7y ago
      Wrong or not let’s be careful about using full names as it’s how these things get whipped up into pile-ons.
  • duxup 7y ago
    The number of people who have control of social media accounts for companies who do not understand how to relate to people / basic customer service / can predict how their post will be received is shocking. I worked at a company of 5K+ people and one of the folks in control of the twitter account(s) would come to me with questions. Now I applauded them for coming to me for technical questions before posting, that was great, but they absolutely did not have the self awareness / understand what to say / when to say it and etc. But hey they were tied to a high ranking person (who also had no clue) so they had access to the account. In my early days I worked PC customer support... I feel like that comes in handy all the time.
    • the8472 7y ago
• pinewurst 7y ago
  Does anyone know what Gandi is using as a “filer”?
  • corford 7y ago
    ZFS by the looks of the status updates: >we have a problem to import zfs pool on the unit storage. Our engineers are still working on it.
• matthewaveryusa 7y ago
  Looks like their backups only consisted of in-region backups on systems that were homogeneous. Common pitfall. While technically a 3-node distributed system may provide disaster recovery from one node failing, in practice, an accidental rm -rf from an ansible script targeting all three machines, or a bug in the software that's doing the replication, will leave you without a backup plan. If you're in such a situation, The easiest is to do filesystem level backups with something like zfs and ship the backups to a third-party system that only has write/append-only semantics (better yet, use a write-once-read-many (WORM) disk to really guarantee it.).While there will still be _some_ data loss, it'll let you recover since the last snapshot. If you don't have zfs, a database backup that runs the db dump script and scp/sftps it to a server running as a cronjob can also be an immediate remedy while you get your shit together (and by that I mean buy yourself a product with an immaculate reputation like aurora or cockroachdb to manage the db for you) Harder but better would be to tee the log of the changestream (all distributed systems have such a log) to a third-party system. This is ideal because if it's done synchronously it'll let you recover since the last committed transaction. And of course, test your backups, because backups are subject to code rot as well.
  • namibj 7y ago
    What backup strategy are you implying for the case of cockroachdb? Streaming the changefeed (including timestamps) to an external append-only system while slowly and incrementally iterating through all tables using as of system time to reduce impact on active transactions and know how late this shard of a "full backup" can be inserted into the "agumented" changefeed you'd generate by interleaving these shards into the changefeed. For replay you'd use the stream from the oldest shard up to the select min(a) from (select max(timestamp_resolved) as a from changefeeds group by table) newest timestamp you know you have the transactions complete changesets for (the resolved timestamp can be periodically emitted to confirm that no further records in the same feed(/table) could have a transaction timestamp earlier than it, inducing a partial ordering). You could replay the (combined,sorted,agumented) changefeed in-order, or shard it on the table's primary key to ensure per-key monotonicity when applying the streams in parallel threads/transactions/nodes.
• totaldude87 7y ago
  shit happens...but the way their philosophy is fine tuned makes me wonder.. Above all, "no bullshit" is our golden rule—to treat our users how we want to be treated. It's a promise to respect your rights and to level with you about our shortcomings. https://www.gandi.net/en-US/no-bullshit https://www.gandi.net/en-US/no-bullshit ex: https://twitter.com/andreaganduglia/status/1215199147701231616 https://twitter.com/andreaganduglia/status/12151991477012316... (thanks op) We will listen to you, and be honest in our replies, even if it means you won’t always like what we say.
  • ChikkaChiChi 7y ago
    > We will listen to you, and be honest in our replies, even if it means you won’t always like what we say. They are actively treating their customers like shit, and that tone starts at the top. No bullshit does not give creative license to be assholes to people that are panicked because of something you directly caused.
• terom 7y ago
  From the incident timeline: > we have a problem to import zfs pool on the unit storage I really want to know what went wrong to a) break ZFS b) prevent recovery from backup.
  • terom 7y ago
    Re myself, it looks like they're using FreeBSD-based ZFS filers with iSCSI/NFS exports using a user-spce NFS server: * https://news.gandi.net/en/2019/09/exporters-detect-micro-incidents-and-improve-storage-performance/ https://news.gandi.net/en/2019/09/exporters-detect-micro-inc... > Gandi’s storage infrastructure consists of two environments: one for IaaS and one for PaaS. Both are based on FreeBSD-based storage units (filers), that stock each volume (disk) as though it were a ZFS volume. * https://news.gandi.net/en/2019/03/tracking-a-storage-issue-led-to-software-change/ https://news.gandi.net/en/2019/03/tracking-a-storage-issue-l... * https://www.bsdcan.org/2016/schedule/attachments/351_FreeBSD%20based%20high%20density%20filers-bsdcan2016.pdf https://www.bsdcan.org/2016/schedule/attachments/351_FreeBSD... No mention of what they're doing for backups / "replication systems", unsurprisingly/unfortunately. I'm anxious to know what the failure mode for `zfs send | zfs receive` replication is here?
    • tomatocracy 7y ago
      Sounds very much like they weren't doing zfs send | zfs receive to anything sufficiently physically separated. For example, if you send and receive in the same pool, it's replication but still leaves you vulnerable to issues where the pool can't be imported due to corruption in the wrong places (it can happen) or significant hardware failure (eg a PSU fault that takes out too many of the drives in the pool).
• deanmoriarty 7y ago
  I have a custom domain with Gandi and take advantage of their mail forwarding option to forward the emails sent to the custom domain (my “no lock-in” email address) to my personal Gmail account. Considering how critical email is for me, seems like I won’t be trusting their MX servers to process all my inbound mail anymore and will soon be looking for another solution that works well with Gmail (don’t want to pay for GSuite), and possibly also transfer my domain to another registrar. That support tweet is such bad taste.
  • fchu 7y ago
    Depending on your email volumes, mailgun could be a viable alternative
    • deanmoriarty 7y ago
      Thanks, I'll absolutely look into this soon.
  • blibble 7y ago
    gandi's response notwithstanding: email is hardly reliable unless you run all the MXes (and can prove otherwise): you're likely having emails dropped all the time already
    • deanmoriarty 7y ago
      For sure, but everything is relative. Ignoring for a second the lock-in factor of using a @gmail.com address, I would trust Google's MX servers any day over Gandi's, especially after this last incident (trust == reliability in this context).
      • blibble 7y ago
        Google's MXes are notoriously strict and drop or permanently delay emails all the time for reasons beyond your control as a recipient an example from 5 minutes ago from one of my MX'es (which only forwards, after heavy greylisting and spam filtering): Jan 9 18:05:59 mail postfix/smtp[26197]: to=<ABC@gmail.com>, orig_to=<XYZ>, relay=alt1.gmail-smtp-in.l.google.com[209.85.233.26]:25, delay=25000, delays=25000/0.01/1.6/0.16, dsn=4.7.0, status=deferred (host alt1.gmail-smtp-in.l.google.com[209.85.233.26] said: 421-4.7.0 Our system has detected that this message is 421-4.7.0 suspicious due to the very low reputation of the sending domain. To 421-4.7.0 best protect our users from spam, the message has been blocked. 421-4.7.0 Please visit 421 4.7.0 https://support.google.com/mail/answer/188131 https://support.google.com/mail/answer/188131 for more information. ABC - gsmtp (in reply to end of DATA command)) that's not my reputation (which is high), that's the reputation of the sender's From address and it doesn't send it to the spam folder, it just delays the email forever until my MX gives up
• markorigho 7y ago
  Just a quick question, how do I transfer my site adress to another hosting company, is that too is lost? Sorry for being a philistine about this... Cheers
• markorigho 7y ago
  Hey Guys sorry for being a philistine about this but does that mean we have lost our domain name and how can we migrate it to another hosting platform? Cheers
  • mratsim 7y ago
    If you only had a domain you're likely not affected (metadata). If you also have a website hosted there (data), you may be.
• moralestapia 7y ago
  Whoops, so long with the "no bullshit" policy. I stopped using them a while ago but for a different reason. I used to use their website to check availability/whois for domains that I was interested in buying. If it was available I didn't buy it at the time but until I finished the website/app whatever I was going to put there, this took me a few months obviously. It happened to me that when I was finally ready the domain had already been sold to someone else. This repeated five times during six or so years. Now, I know, "someone else could have thought the same thing" but I find it very hard to believe that it happens so often. These domains were a bit of niche words that were not hot topics at the time, some of them using fairly uncommon TLDs (like .one). Another weird thing is that they were always registered to someone living/or doing business at India, and it was a fairly simple landing page with a "contact me" link. I'm a bit superstitious so I don't think it was a coincidence. Now, I don't think this is a GANDI problem per se, but my theory is that they share this information (who is looking for which domains) with marketers or something like that, or maybe it was a rogue employee trying to make some money squatting domains. I would have expected this from BigDaddy or similar sharks, but from a company whose motto is "no bullshit" I had much better hope. Anyway, I decided to move (to namecheap if you're wondering) and surprisingly the problem went away.
  • Jasper_ 7y ago
    I've heard that this isn't actually the registrar's fault, it's the registry's fault. So your TLD is sharing the "is registered" query with other parties. That said, everything about the domain registration industry seems designed to appear sketchy as all hell, so who knows.
  • craze3 7y ago
    This happened to me with GoDaddy and Namecheap before, which is why I switched to using Gandi for all my domain searches... Now I'm regretting it! But as @Jasper_ said, this could be a problem with the domain name registry selling/leaking that info (AKA all their 'is_available' queries), and not the registrar.
    • Hoasi 7y ago
      You shouldn't regret it. GoDaddy was and still is way worse than this, there is no comparison.
• whorleater 7y ago
  Gandi is never really impressive, but they're one of the few registrars where I can get .af domains without a hassle.
• privateSFacct 7y ago
  There have been a number of threads suggesting places like Gandi over AWS because they are so much cheaper. I've always been skeptical about building key apps on these types of places but folks INSIST it's the right choice. 3TB at Gandi costs $6 + you get compute with it. 3TB of bandwidth at AWS might be $270. Has anyone tried this instead of using cloudfront etc? Get 100 $6 hosts and pump out content for your ipv6 connecting clients etc?
• benguild 7y ago
  I hate to say it but Gandi seems like they’re in a quality freefall. I had a domain there a year or two back because they were one of the only registrars that supported that particular extension... and man, so many problems just with simple tasks like updating the WHOIS info and credit card for renewal. This is basic stuff.
• teh_klev 7y ago
  The key question is, did Gandi offer and explicit backup service for your data on their plans? I just had a look and I don't see this being offered. As a former hosting engineer, at the risk of pissing on everyone's outrage parade, but unless an explicit guarantee of a backup is included in your plan's contract, or you can pay for backups as a bolt-on, then if you've lost data it's your fault for not planning for this scenario. And I mean proper backups where you get, for example, twenty eight days of hourly backups and you can pick a specific version of file to recover in that 28 period. And where those backups are stored on different hardware or off-site. We offered this as a bolt-on (in-site and off-site). Tt was 20 quid a year for in-site, the off-site was a bit more. But a great many customers chose not to pay for this add-on, even despite the great big red bold warning text explaining that unless they paid for this add-on we made no guarantees about the permanence of their data in the event of a storage problem. Guess what.... Now that's not saying we didn't take snapshots of the hosting environment, but they were for internal use and to allow us to recover quickly in the event of something unexpected going wrong, but now and again stuff breaks. Sure, it's unfortunate some lump of storage hardware has failed and whatever mirrors they may have had have been taken out as well. They possibly could have done better but shit happens sometime. You shouldn't rely on an "implied backup" from your service provider, if you want that then you're going to be paying a shedload more for hosting your Wordpress and Woocommerce site. It's up to you to make sure absolutely sure your data is safe if it's critical to the day-to-day running of your business. Edit: ok, so this is tucked away in their docs (thanks to itake below): https://docs.gandi.net/en/simple_hosting/common_operations/snapshots.html https://docs.gandi.net/en/simple_hosting/common_operations/s... But it does say: > Snapshots do not make a backup of your databases. If you would like to perform a backup of your databases, we recommend you perform an export, or launch a dump script via crontab. The bottom line...is it guaranteed in your contract? Always check. And as per my follow up comment, those plan prices are are just too cheap for that facility to be taken seriously for business continuity. They're a convenience to quickly recover a version of a file, not a serious backup.
  • deleted 7y ago
    [deleted]
  • itake 7y ago
• M2Ys4U 7y ago
  TIL that gandi was bought by a private equity firm around a year ago.[0] This may explain some things... [0] https://news.gandi.net/en/2019/02/futureofgandi-the-adventure-continues/ https://news.gandi.net/en/2019/02/futureofgandi-the-adventur...
  • pnathan 7y ago
    Interesting. That's a very strange blogpost.
  • nachtigall 7y ago
    Where does it say it was bought? It talks about a new investor: > we have found a new investor in Montefiore Investment, who have replaced our former shareholder! Am I missing something?
• Macha 7y ago
  I have some domains here, mostly secondary domains to not have all my eggs in my namecheap basket (e.g. if anything happens to namecheap or my namecheap account). Will likely transfer those to elsewhere after this. Probably Name.com, I guess.
• blackethylene 7y ago
  All my domains are registered through Gandi. What good registrars would you suggest? I'd like to move them out.
• buboard 7y ago
  There s an upside to knowing your data is not backed up somewhere, and that when you delete them they 're really lost. They should offer that as privacy-conscious hosting.
• newscracker 7y ago
  This seems like data has been lost from servers hosting sites/services. Since Gandi is mostly known for domain registrations and DNS, I'm curious if you (as an individual who hosts websites/online services somewhere on the web) backup your site's DNS records periodically (or whenever they're changed). What if your authoritative name server lost data and all the caches of those records across geographies expire while you're asleep/away? If you do back these up regularly, how do you do it in an automated way on a *nix system? I found this article [1] when I searched about this, but it's not a simple shell script. The scripts that I did find on some of the Stackexchange sites seemed to have specific subdomain names hardcoded. [1]: http://www.programblings.com/2012/07/23/do-you-back-up-your-dns-records/ http://www.programblings.com/2012/07/23/do-you-back-up-your-...
  • dylz 7y ago
    Gandi has an API and lets you download entire zonefiles if they host your DNS.
• Hoasi 7y ago
  As a customer, I only have good things to say about Gandi.net but have to admit this is subpar customer communication right there. Lost 3 sites built with WordPress. Will rebuild as static sites repo separate from host, no more database, lesson learned.
• jchw 7y ago
  Gandi is absolutely not the company I expected this to come from. With that having been said, everyone please stop assuming your data is safe. It’s never safe, but it’s extremely not safe single homed somewhere. Make backups. Anything that’s saved locally on one machine only? Consider it gone until it’s backed up. Cloud providers may be able to give you better assurances, but if you really care about data give it at least 2 independent homes. I’ve lost data more than I care to admit. BuyVM lost one of my VPSes years ago. Who’s fault was it really? When you are ready to stop kidding yourself about your data, check out some backup solutions. I particularly like Borg Backup: https://github.com/borgbackup/borg https://github.com/borgbackup/borg And if you do not have network attached storage anywhere there are services that provide it as a service. (Note: I think needless to say it’s also a good idea to back your NAS up to other places too, although I haven’t gotten into this practice yet. Synology supposedly has a lot of features around this.)
• supernova87a 7y ago
  Yeah, generally French people suck.
  • dang 7y ago
    Please don't do this here. We detached this comment from https://news.ycombinator.com/item?id=22002923 https://news.ycombinator.com/item?id=22002923 and marked it off-topic.
• SeanMacConMara 7y ago
  Interesting reaction. Is the highly negative reaction correlated with US culture maybe ? I've used them for many years and had several complex support interactions with them. Their customer service policy is very "API-like" in that you get exactly the t&c you paid for and nothing more. Hand-holding and soothing noises are not included in the t&c. They fuck up you get a refund, you fuck up they'll tell you exactly that. Outside that they're very casual relaxed humans to communicate with. I find that far more trustworthy (in the mathematical sense) than a "slick" twitter feed. Politness does not imply trustworthiness.
• Neil44 7y ago
  The problem with cheap hosting is they want to use backups as an upsell, but you should still have backups to cover the companies ass even if the customer doesn’t get to use them. 123 Reg lost a load of customers VPS’s a year or two ago also thanks to a faulty script.
  • PHGamer 7y ago
    Not excusing them but.. with modern container hosting you really should be able to make your own. even with cheap VPS hosting. There is no reason to live in a world where a server goes down you lose anything anymore.
• zulgan 7y ago
  I had a co-worker who was super chill during outages; especially at night, we were 10-15 people on the call fixing issues related to his work almost monthly. those outages costed millions of euros, and he never picked up his phone at night, once I asked him why he never picks up, he told me: "I used to be a general surgeon, when someone calls me people die. Relax, nobody is dying during our outages." now I think I am taking myself(and my work) too seriously.
  • supernova87a 7y ago
    Given that the statistical economic value of human life is around $7m, maybe he should start picking up his phone.
  • unishark 7y ago
    "I used to be a general surgeon, when someone calls me people die." Hence he's not a surgeon anymore.
  • kbr2000 7y ago
    It's likely the only way to stay sane in a corporate environment. The problem is, you don't need much people practicing that, before it drags everyone down to the same niveau. You can choose to try to continue your quest into doing work seriously (this will likely drive you insane over the years), or to join in that kind of negligence (goodbye spine), or to quit. In the end it got us where we are now, a world filled with fake companies selling their fake little products as they were qualitative, and making a game of disrespecting their own customers. Pure facade, been there... I've done it before, but I'll recommend to you Scott Adams' book, The Dilbert Principle for some light reading about forces like that at work.
• Johnny555 7y ago
  The assessment is taking a long time because there are several TB of data on the filer Is that a lot of data? That sounds like a very small filer that could have easily been backed up.
• corford 7y ago
  I'm a long term user of Gandi for my domains but have wanted to get off them for some time now. Can anyone recommend a domain registrar "equivalent" of a Fastmail or Letsencrypt or DNSMadeEasy i.e. truly no bullshit, geek friendly and polished at the same time ? I'm not too bothered about price. I just want a well run outfit that has a wide selection of TLDs and ccTLDs (and ideally isn't a mega corp like google but is big enough that I don't have to worry about them disappearing overnight).
  • dangravell 7y ago
    NearlyFreeSpeech?
    • corford 7y ago
      Close but sadly they don't seem to offer ccTLDs :(
  • dorian-graph 7y ago
    +1 to this. I've got a few bookmarked, but I haven't tried them: Porkbn, Nuage, Hover, and Namesilo.
    • corford 7y ago
      Thanks! I'll check these out.
  • knute 7y ago
    easyDNS is pretty good.
    • corford 7y ago
      Thanks. They look exactly like what I want.
  • changethe 7y ago
    i have been using uniregistry.com for a while and have nothing bad to say. in another comment on this thread, someone pointed to cloudflares new registrar offering, which also seems good.
• anonred 7y ago
  Gandi is the absolute worst. The last time I tried buying a domain through them, they took my money and then demanded "identification" via government ID (citing some bullshit in their ToS). I refused, so they closed my account and took the domain with them. Based on that, I'm not surprised at all by their CEO's response to this incident[0]: >If we led you to believe that you had nothing to do on your side when warned multiple times to make your back ups, then we'll have to make it clearer, and stop assuming that it's an industry wide knowledge. [0]: https://twitter.com/StephanGandi/status/1215287619938062342?s=20 https://twitter.com/StephanGandi/status/1215287619938062342?...
  • TheChaplain 7y ago
    I had exactly this problem too but with NameCheap. Told them to put their id request and my money somewhere and left for Gandi. After more than 8 years with Gandi, not had a single issue with them.
• acvny 7y ago
  Never heard of it and hopefully never will
• cenourinhapt 7y ago
  Always make your own backups. Shit is going to happen any time soon.
• nojvek 7y ago
  In the world of cloud, this should be pretty trivial. Upload your daily dumps / asset metadata to S3/GCS/ABS. Set a retention policy I.e even if someone ran some delete command it wouldn’t delete. Someone with retention lock permissions is the only one that can remove the locks And delete. There is cold storage and other things even cheaper. But cloud object prices are pretty cheap per GB it’s ridiculous. I think they make most of the margins on bandwidth. Losing customer data. All customer data is pretty ridiculous. I can understand downtime. I can understand losing a day of changes. But everything? That’s just unacceptable business.
• dkarras 7y ago
  This is like living in an alternate universe, I've been heavily involved in all things programming and webdev for years, following trends and whatnot and it is literally the first time I'm hearing of this particular company. What is (was?) so special about them that they attracted the HN crowd can someone briefly explain? Why would I buy domain from them when something like namecheap, even google domains exists? Why would I even host something there?
  • porker 7y ago
    > Why would I buy domain from them when something like namecheap, even google domains exists? Why would I even host something there? If you're in Europe, they're cheap for many European countries' domains. Back 10-15 years ago they were special because it felt like a hacker kind of company. They gave free WHOIS privacy, what seemed like good DNS control/UI at the time. But it was the WHOIS privacy that got me onto them. I still use them because they're around half the price for .co.uk than many registrars - and many others I've used have become more rubbish than Gandi has. All my DNS is hosted elsewhere now, and I never understood why Gandi introduced hosting et al. I've never used it and never would, it seemed a terrible diversification for a good domain registrar.
    • thrwaway69 7y ago
      Can you explain how you not using their hosting makes it a terrible diversification? I think majority of people buy domains for hosting websites so it makes sense they would want to setup one using one click WordPress or something similar.
      • porker 7y ago
        It wasn't their skill-set. They had problems with their hosting from the start; reliability problems IIRC, not data loss.
  • technion 7y ago
    I wrote out a list of things I needed in a domain registrar and once you include U2F logons and DNSSEC support, you find yourself in a very limited space.
• l0b0 7y ago
  The "no bullshit" motto is mentioned a few times here. A motto is just another marketing device ­– a way for a company to pretend to have any sort of principles beyond making as much money as fast as possible. Why would anyone believe a motto is anything other than a marketing device? It is only believable if people follow it contrary to pragmatism. Any company is eventually going to have a fair share of people who believe being pragmatic is more important than their motto. And in Western culture at least it's usually considered rude to bring up the "big guns" and have a fundamental values discussion when everybody just wants the meetings to end and to start making more money.
  • scandox 7y ago
    In fact a motto is usually chosen to cover a weak spot. So "No Bullshit" reads to me as "We're Kinda Cowboys". "We Care" - "People Know We Don't care". Fujitsu – “The possibilities are infinite” ... "The Ways in Which we can Screw this Up Are Infinite" Intel – “Leap Ahead” and “Sponsors of Tomorrow”. "We've got to protect our entrenched position". LG – “Life's Good”. "Life is Actually Objectively Bad". Google - "Don't be Evil". "How We Actually Make Money is Evil But Our Mission is Good".
• yardie 7y ago
  Azure Shared Responsibilities [0] AWS Shared Responsibilities [1] Flipping a switch that says "Backup" does not mean you are handing your responsibility to them. At most, they will fail to meet their SLA, write you a check for according to the TOS and be done with it. At best, you'll be able to bitch about it on Twitter, possibly threaten a lawsuit (you read the ToS?) and still be in the same position because you did not share the responsibility of securing your data. [0] https://docs.microsoft.com/en-us/azure/security/fundamentals/shared-responsibility https://docs.microsoft.com/en-us/azure/security/fundamentals... [1] https://aws.amazon.com/compliance/shared-responsibility-model/ https://aws.amazon.com/compliance/shared-responsibility-mode...
• smileybarry 7y ago
  (1h22m before this comment) > Updated on Thursday, 9:58 PM +0200: > we're not sure we will be able to provide the data but we were able to recover a version of the filesystem from right before the crash Maybe it's not all gone.
• iseeyou 7y ago
  last update says they were able to restore a version Updated on Thursday, 9:58 PM +0200: we're not sure we will be able to provide the data but we were able to recover a version of the filesystem from right before the crash I have been using them for DNS and some minor hosting for a long time and I will stay with them. I think it's important to avoid the monoculture/centralisation which is otherwise happening. Sure Gandi has their flaws, they are humans. I expect they do will a proper post-mortem on what went wrong and how they managed to fix it. Seems they were using ZFS and relied on it a bit too much. Or if they indeed managed to restore the last snapshot, then their only error might have been the classic one of underestimating how long restoring/investigating several terabytes take even on modern HW.
• nachtigall 7y ago
  News post about it: https://news.gandi.net/en/2020/01/major-incident-on-our-hosting-infrastructure-in-luxembourg/ https://news.gandi.net/en/2020/01/major-incident-on-our-host... A site of mine is also hosted as their PAAS at Luxembourg, but was luckily not effected. Probably my site was on another storage unit ("on one of our ZFS storage units"). PS I also always thought that the snapshots were backups.
• YesThatTom2 7y ago
  That's why I keep all my DNS configuration in DNSControl and push the results to Gandi (and NameDotCom, and Route53, and GoogleDNS, and AzureDNS) https://github.com/StackExchange/dnscontrol https://github.com/StackExchange/dnscontrol (Terraform users have a similar benefit)
  • gingerlime 7y ago
    Sounds really cool. Do you also have NS records for all of those? or just in case you want to switch-over? (can you actually hold a domain on multiple registrars?)
• jiggawatts 7y ago
  Just to play devil's advocate: This is in no way different to how Azure, AWS, and GCP operate. They don't have backups either. They too rely on n-way replication, a bit like a distributed RAID. All cloud providers make it absolutely clear, in black & white, that protection of your data is your responsibility, not theirs. What I find hilarious is that most cloud providers only provide built-in backup functionality for a tiny subset of their services. Ask Microsoft if you they have a "backup" button for Azure DNS Zones. Or Azure load balancers. Or anything else that isn't a VM disk, App Service, SQL Database, or a Secrets Vault. I mean, look at this insanity: https://docs.microsoft.com/en-us/azure/backup/backup-azure-files#limitations-for-azure-file-share-backup-during-preview https://docs.microsoft.com/en-us/azure/backup/backup-azure-f... "Backup for Azure file shares is in Preview." After 10 years of operation, this trillion-dollar company has only a use-at-your-own-risk beta for data protection! Don't be too hasty to point fingers at Ghandi and laugh about how they're unprofessional. Whatever you're using is essentially the same. Ask yourself this: Could your organisation recover if some malicious admin simply deleted all Azure Resource Manager resources in one go using PowerShell?
  • teddyuk 7y ago
    Maybe not even malicious, maybe they just put in the wrong subscription ID :(
    • jiggawatts 7y ago
      Yup. This thought occurred to me when I was testing a bulk resource creation script. My workflow in my lab tenant was: 1) Bulk create hundreds of resources 2) Bulk wipe everything 3) Go to step #1 Turned out, I had some objects with globally unique names that were now conflicting in the production tenant, so I had to wipe my lab. I had already logged on to the production tenant, and I was so "trigger happy" that I very nearly ran my bulk-erase script against the wrong subscription. It was a terrifying moment of clarity.
  • bscphil 7y ago
    Everything you say here is true, but at the same time it's just a fact that Gandi lost a lot of customers' data, and AWS, GCP, and Azure have never (as far as I know) lost a significant amount of it at once. You can talk about theoretical responsibility for data, and it's true, you are responsible for having backups of your data, no matter how many "9s" the service has, but the basic fact is that some services have been consistently good at not losing customer data, and others haven't. Even though I'm going to back up my data no matter where it is, I'd still rather use the service that's got a better track record with it. I haven't ever even lost a file on Google Drive, which as far as I know provides no reliability guarantees at all.
• RantyDave 7y ago
  For not the first time I'm left thinking that the "big filer" model is not such a great idea :(
• aspectmin 7y ago
  I helped co-found a large Dropbox-like white label product. We used AWS and especially s3 for storage. After many many years of experience with systems, I made sure we had as many possible ways to recover user data as we could. The initial solution was a large Postgres database for all the metadata/indices and s3 for the actual storage. Despite much pushback we built in little things like an individual meta file on the file system for each file we stored. That way, if we lost the Postgres dB for any reason, we could create a script to rebuild the dB and restore access avoiding massive counts of orphaned files. A simple and probably stupid solution but... Well guess what - the DB got corrupted and after some ado, we restored all access and none of our customers lost anything. No it’s not full backups but...
• jigglypuffs 7y ago
  'Gandi' means bad in Hindi. Some coincidence huh!
• kuon 7y ago
  I have about a hundred of domains registered at gandi, I used to like the formed management interface, but I really hate the new one. Is there a registrar you would recommend as an alternative, I don't need DNS, nameservers and glue records and I'm ok. The main selling points are stability, transparency and simplicity. I don't care if it's not the cheapest.
• kup0 7y ago
  Regardless of any of the technical aspects of this disaster, the attitude of the company and its customer service means I will be staying far away from them.