8 ms·

Mea Culpa I'm in the industry. I knew we should have had backups. She is brilliant but not technical and while the backups would have been expensive and difficu

Mea Culpa I'm in the industry. I knew we should have had backups. She is brilliant but not technical and while the backups would have been expensive and difficult due to size, I should have built multiple NAS and distributed them geographically and used Syncthing to keep them updated. To be clear we were legacy G-suite users and have converted to paying customers, so it was a free service while this happened, and is a paid service (I guess is youtube part of that?) now. I just assumed if there was a problem we could youtube-dl everything, it didn't occur to me that control of the account would be suspended arbitrarily with no recourse for literally no reason. Despite reading about similar things here for years. I guess I just thought those were lightning strikes, and I was focusing my time outside of work on the new SPA rollout we were working on for the company. I understand excuses like armpits are something we all have and they all smell, so I'll just say I fucked up and own it. I appreciate all the advice and I understand why you are all saying what you are saying. This was spur of the moment post (I didn't think it through) and I hadn't updated my contact information on my profile. I've done so now. Thanks.

donatzsky 5y ago
If you're a paid G-Suite user, you might be able to get help that way. I'm pretty sure I have heard of others doing something like that.
- heavyset_go 5y ago
  In my experience, paid GSuite "support" are just underpaid contractors in developing countries who don't have much power to do anything outside of a few common actions in response to customers' requests.
  - hackermailman 5y ago
    Yep, even if you pay Gsuite you get some guy in the middle of nowhere earth just feeding you macros for support. They don't care at all big gov contracts is their business
  - greggman3 5y ago
    strange, every time Ive contacted gsuite support I've gotten timely professional support
mulmen 5y ago
Back up everything in G-Suite outside of Google. If your Youtube account is connected to your G-Suite services they are at risk as well. Geo dispersed NAS is probably the most cost effective long term but for now the important thing is getting data off Google services before you lose it forever. So take a look at other cloud providers in the short term. The more you interact with Google and ask questions the more likely it is they shut down more access. If you have your domains registered with them move it somewhere else, preferably a dedicated registrar so you can at least re-host email and website.
- antaviana 5y ago
  This cross-connection of services scares me as well. I once happily ordered a Pixel phone using my gmail account. The phone was stolen before delivery (the delivery guy handed me an empty box with a hole in it!). I reported the situation to Google and they could confirm with the parcel company the delivery was not completed successfully (plus some additional background checks they did after requesting access to my account) and they refunded the order. The thing is I was not brave enough to place a second order of a Pixel because of fear of risking closure of my gmail account if the phone in the second order was also stolen before delivery, so I ended up ordering a Nokia instead.
  - hnshit2 5y ago
Axsuul 5y ago
You did the best you could with the information you were given. I don't think anyone should have to resort to building multiple NAS and distributing them geographically. It's 2022 and this an extremely esoteric thing to do. Going forward though, Vimeo feels like a better fit for you guys. Caveat emptor, never rely on free services for critical parts of your operation.
- lobocinza 5y ago
  > Caveat emptor, never rely on free services for critical parts of your operation. Don't rely on a single service. Having the same data at two free services is probably more safe than relying on a single paid service. At least for data loss. If data is sensitive don't rely on any service, just encrypt it with your own key.
- pmoleri 5y ago
  Takeout + store encrypted in AWS S3 or even Glacier? Preety cheap storage and supposedly high redundancy. I guess it's better to not use Gmail email address to create that AWS account.
  - dbmnt 5y ago
    NAS + automated backups to BackBlaze (Amazon S3 compatible storage, but much cheaper).
- inapis 5y ago
  >I don't think anyone should have to resort to building multiple NAS and distributing them geographically. It's 2022 and this an extremely esoteric thing to do. Given how giant companies (especially Google) behave, I do not feel it is an esoteric thing to do. Anyone with even a modicum of technical skill can roll their DIY NAS. For the non-tech savvy, buying off the shelf is fine. Should not be needed in real life but this is the world we live in where robots have way too powerful ban hammers and the only way to get any support is to make sound in public forums or have a large social following.
  - 5y ago
lobocinza 5y ago
Probably not that expensive. Vimeo cheapest plan is $7/month (w/ limits) and $50 for unlimited plan and cloud storage (for backup) is cost competitive. Even having it on a local disk would probably be enough. But... it's not your fault. It's just that despite the propaganda these companies don't give a fuck about their users.
- hnshit2 5y ago
  cost competitive against free? lol
  - AitchEmArsey 5y ago
    Evidently "free" has other costs, like "we're going to delete your stuff and there's nothing you can do about it". So on balance I'd say paying for something is a pretty good deal.
  - Ekaros 5y ago
    Free is perfect for content that is free to make. If your content has clear monetary value in workhourse, why not pay at least fraction of that for actual service?
  - lobocinza 5y ago
    Free as in serfdom, you don't own a thing, you have no right, your content can be deleted for any reason at any time without recourse.
ziml77 5y ago
The important thing for your business going forward is to ensure that you have good contracts. Your contract with YouTube was their terms of service. Those terms basically give you no guarantees yet give Google immense freedom. I think Vimeo is a good choice for hosting new videos. For archival I would consider just using a single NAS of your own and then have that replicate to Amazon Glacier in at least 1 other region (unless you already have the rack space and someone to physically manage the NAS box in another region).
- WalterBright 5y ago
  I wouldn't rely on a contract for archival services. Legal action can take several years to wind through the courts. For gawd's sake, buy some terabyte drives and make a backup that you control. Mail a copy to your mom just in case.
  - ziml77 5y ago
    Hence why I didn't say to use Glacier as the only archival copy. In the setup I'm proposing, they would have a copy on Vimeo, at least 1 copy on Glacier, and a copy on their NAS. An offline backup mailed to his mom isn't a bad idea either though. The point I'm making about contracts here is not solely about having the option to sue if the other party doesn't uphold their end of the contract. It's about even having the option to agree to conditions that aren't ridiculously one-sided in the first place.
xupybd 5y ago
For write and store content would m-disc work better? It's not cheap but passive.
datavirtue 5y ago
AWS Glacier or some similar service on Azure. Super cheap. Cloudflare will store and stream 1000 minutes for $6 a month.
- hnshit2 5y ago
LightHugger 5y ago
Indeed you have become one of those "lightning strikes". What would have convinced you that it was a real possibility though? Even reading this post, many readers are will also just assume the same as you did, that it was a lightning strike and will never happen to them... Not sure i have a solution myself.
cloudking 5y ago
Since you are paid customers you should escalate your issue through Google Workspace support https://support.google.com/a/answer/1047213?hl=en https://support.google.com/a/answer/1047213?hl=en Insist that your issue be escalated to an internal bug so that it is properly triaged. Even if the videos were deleted as part of the suspension, it's highly likely that there are still multiple backups and cached versions in Google's systems. They should be able to help you restore them at the very least.
- SpelingBeeChamp 5y ago
  Fulltime YouTuber here. Just as one datapoint, I was recently [very] pleasantly surprised to learn that YouTube was able to restore a video that I had accidentally deleted. It went back to being live as if nothing had happened—though my contacting support was admittedly right after the deletion happened.
- 35mm 5y ago
  Worth a try but I’ve tried doing this before and they said it’s outside the scope of paid support.
  - ldng 5y ago
    Yet it's not outside of the Law. At least in Europe, they MUST give all your data back.
    - tremon 5y ago
      Assuming you mean the GDPR, it only obligates them to share the data they have. If their response is "we don't have that data any more", then the GDPR won't help you. It opens them up for a massive claim if it later turns out that they really did have the data somewhere, though.
      - ldng 5y ago
        The "we don't have that data any more" should only apply to "I closed my account" not to the "we unilaterally chose to close you're account and delete all your data just because we can". And if they jumped the gun and can't, they must pay for the prejudice. Period. They bet on people not suing and they're right most of the time customer don't demand. It does not mean, just because they actually get away with it by play dumb, that there are not legal basis.
WalterBright 5y ago
You can buy 16 terabyte drives for $300 or so. Cost shouldn't be a problem to back up your videos.
- iforgotpassword 5y ago
  I second this. I can't fathom that on HN of all sites people recommend renting backup space from the same big tech companies that continuously fuck over people. Setting up a NAS is not magic. Setting up two isn't either. Put one in you parents basement and check on it once every year when you visit for Christmas. If you don't need 16TB go with an SSD (although with the current contamination situation not the best timing to buy flash).
  - Cynosaur 5y ago
    Aren't you making it sound a bit too easy and straightforward than it actually is? It might be that simple for you but buying a NAS device like from synology can be pricey too, setting it all up so it is secure but network accessible, and then also handling possible drive failures down the line (and the chances of that are much higher than google fucking you over IMHO) is much more than an average non-tech-savvy user would be interested in.
    - iforgotpassword 5y ago
      Yes, I wouldn't recommend my mom do it. This was with the average HNer in mind, but even an average tech enthusiast could do it. I think the OP commented here too saying he is a techie. But tbh I think even getting and external drive and backing up to it should at least be done in parallel to any cloud service you choose.
    - lbriner 5y ago
      Maybe not really easy but it is really just due-diligence and getting a solution that is worth the money you are making by providing the service. If you weren't a mechanic, you wouldn't avoid maintaining a car and then get upset that it broke down eventually, you would pay a specialist to maintain it for you. Youtube is cool because it's "free" but you also get almost zero support from Google and they are pretty the worst in my opinion. I have got decent support from AWS and Microsoft (never tried with FB).
vero2 5y ago
> multiple NAS and distributed them geographically and used Syncthing to keep them updated As some one working in storage, please do not get tons of NAS they are pain to manage eventually. Example: linus-tech-tips could not do it properly. If you use youtube-dl how large is one video? Lets say 2 GB. I presume, every week you create have 2 videos? Then 4GB per week. Google gives you $20 for 100 GB. Every year create a new account - like - company name.2022 dump everything there. Hire an clerk to make sure the credit card is payed every year. That way you do not lose all videos at a time. Heck you can even create free Google drive account every few months and segment them.
- manigandham 5y ago
  That's unnecessarily complicated and expensive. Object storage like Wasabi and BackBlaze B2 are $6/year for 100GB, and much easier to upload, maintain and retrieve from in the future.
- Freak_NL 5y ago
  Recommending Google as a backup in case Google decides to delete your videos sounds a bit suboptimal. Go with hosted backup; sure, for this use case that makes sense. But do so with a different company.
  - vero2 5y ago
    acquiring talent to manage google drive is easy. Not cheap or easy for Amazon/B2.
    - throwuxiytayq 5y ago
      Please note that Google are perfectly happy to ban all of your accounts simultaneously. And your neighbors' accounts, apparently. https://www.reddit.com/r/androiddev/comments/ckoej1/googles_practice_of_associated_account_ban_aka/ https://www.reddit.com/r/androiddev/comments/ckoej1/googles_...
      - vero2 5y ago
tlonny 5y ago
> multiple NAS and distributed them geographically and used Syncthing to keep them updated I decided to look at using AWS S3 to archive my media library. If you use the `DEEP_ARCHIVE` storage class, I believe costs are ~$1 per month per TB. The downside of such a storage class is that it takes ~48 hours for files to be "defrosted" and ready to download. Before I push any of my media (that I have legally purchased), I also encrypt it to hedge the risk that AWS suddenly decides to start doing automated scans/fingerprinting for copyrighted content. I assume that's overkill but w/e. Here is a CLI I wrote to make the above a bit easier: https://gitlab.com/tlonny/dfrost https://gitlab.com/tlonny/dfrost
gaius_baltar 5y ago
First, IANAL. This is really dependent on where you live, but as you are paid customers I would suggest to file a lawsuit in a small claims court. Seriously! Google is not playing its part on a legally binding contract, and money exchanging hands is enough to make it valid for the courts. The amount of work poured into these videos, compared with the equivalent compensation that you would have to pay for an employee to make the same work would also qualify for damages. This can also protect you against retaliation from google. But again, IANAL and check our local legislation carefully before. Where I live it's a thing a person can do in an afternoon (usually takes less time than fighting big companies' support online), do not require lawyers, and the other part also can not bring a lawyer to conciliatory session (for big companies this is usually a sham: even if their representative does not have a lawyer present at court, be sure that s/he was heavily counseled by company legal staff before and everything presented was proofread by them too).

donatzsky 5y ago

If you're a paid G-Suite user, you might be able to get help that way. I'm pretty sure I have heard of others doing something like that.

heavyset_go 5y ago
In my experience, paid GSuite "support" are just underpaid contractors in developing countries who don't have much power to do anything outside of a few common actions in response to customers' requests.
- hackermailman 5y ago
  Yep, even if you pay Gsuite you get some guy in the middle of nowhere earth just feeding you macros for support. They don't care at all big gov contracts is their business
- greggman3 5y ago
  strange, every time Ive contacted gsuite support I've gotten timely professional support

mulmen 5y ago

Back up everything in G-Suite outside of Google. If your Youtube account is connected to your G-Suite services they are at risk as well. Geo dispersed NAS is probably the most cost effective long term but for now the important thing is getting data off Google services before you lose it forever. So take a look at other cloud providers in the short term. The more you interact with Google and ask questions the more likely it is they shut down more access. If you have your domains registered with them move it somewhere else, preferably a dedicated registrar so you can at least re-host email and website.

antaviana 5y ago
This cross-connection of services scares me as well. I once happily ordered a Pixel phone using my gmail account. The phone was stolen before delivery (the delivery guy handed me an empty box with a hole in it!). I reported the situation to Google and they could confirm with the parcel company the delivery was not completed successfully (plus some additional background checks they did after requesting access to my account) and they refunded the order. The thing is I was not brave enough to place a second order of a Pixel because of fear of risking closure of my gmail account if the phone in the second order was also stolen before delivery, so I ended up ordering a Nokia instead.
- hnshit2 5y ago

Axsuul 5y ago

You did the best you could with the information you were given. I don't think anyone should have to resort to building multiple NAS and distributing them geographically. It's 2022 and this an extremely esoteric thing to do. Going forward though, Vimeo feels like a better fit for you guys. Caveat emptor, never rely on free services for critical parts of your operation.

lobocinza 5y ago
> Caveat emptor, never rely on free services for critical parts of your operation. Don't rely on a single service. Having the same data at two free services is probably more safe than relying on a single paid service. At least for data loss. If data is sensitive don't rely on any service, just encrypt it with your own key.
pmoleri 5y ago
Takeout + store encrypted in AWS S3 or even Glacier? Preety cheap storage and supposedly high redundancy. I guess it's better to not use Gmail email address to create that AWS account.
- dbmnt 5y ago
  NAS + automated backups to BackBlaze (Amazon S3 compatible storage, but much cheaper).
inapis 5y ago
>I don't think anyone should have to resort to building multiple NAS and distributing them geographically. It's 2022 and this an extremely esoteric thing to do. Given how giant companies (especially Google) behave, I do not feel it is an esoteric thing to do. Anyone with even a modicum of technical skill can roll their DIY NAS. For the non-tech savvy, buying off the shelf is fine. Should not be needed in real life but this is the world we live in where robots have way too powerful ban hammers and the only way to get any support is to make sound in public forums or have a large social following.
- 5y ago

lobocinza 5y ago

Probably not that expensive. Vimeo cheapest plan is $7/month (w/ limits) and $50 for unlimited plan and cloud storage (for backup) is cost competitive. Even having it on a local disk would probably be enough. But... it's not your fault. It's just that despite the propaganda these companies don't give a fuck about their users.

hnshit2 5y ago
cost competitive against free? lol
- AitchEmArsey 5y ago
  Evidently "free" has other costs, like "we're going to delete your stuff and there's nothing you can do about it". So on balance I'd say paying for something is a pretty good deal.
- Ekaros 5y ago
  Free is perfect for content that is free to make. If your content has clear monetary value in workhourse, why not pay at least fraction of that for actual service?
- lobocinza 5y ago
  Free as in serfdom, you don't own a thing, you have no right, your content can be deleted for any reason at any time without recourse.

ziml77 5y ago

The important thing for your business going forward is to ensure that you have good contracts. Your contract with YouTube was their terms of service. Those terms basically give you no guarantees yet give Google immense freedom. I think Vimeo is a good choice for hosting new videos. For archival I would consider just using a single NAS of your own and then have that replicate to Amazon Glacier in at least 1 other region (unless you already have the rack space and someone to physically manage the NAS box in another region).

WalterBright 5y ago
I wouldn't rely on a contract for archival services. Legal action can take several years to wind through the courts. For gawd's sake, buy some terabyte drives and make a backup that you control. Mail a copy to your mom just in case.
- ziml77 5y ago
  Hence why I didn't say to use Glacier as the only archival copy. In the setup I'm proposing, they would have a copy on Vimeo, at least 1 copy on Glacier, and a copy on their NAS. An offline backup mailed to his mom isn't a bad idea either though. The point I'm making about contracts here is not solely about having the option to sue if the other party doesn't uphold their end of the contract. It's about even having the option to agree to conditions that aren't ridiculously one-sided in the first place.

xupybd 5y ago

For write and store content would m-disc work better? It's not cheap but passive.

datavirtue 5y ago

AWS Glacier or some similar service on Azure. Super cheap. Cloudflare will store and stream 1000 minutes for $6 a month.

hnshit2 5y ago

LightHugger 5y ago

Indeed you have become one of those "lightning strikes". What would have convinced you that it was a real possibility though? Even reading this post, many readers are will also just assume the same as you did, that it was a lightning strike and will never happen to them... Not sure i have a solution myself.

cloudking 5y ago

Since you are paid customers you should escalate your issue through Google Workspace support https://support.google.com/a/answer/1047213?hl=en https://support.google.com/a/answer/1047213?hl=en Insist that your issue be escalated to an internal bug so that it is properly triaged. Even if the videos were deleted as part of the suspension, it's highly likely that there are still multiple backups and cached versions in Google's systems. They should be able to help you restore them at the very least.

SpelingBeeChamp 5y ago
Fulltime YouTuber here. Just as one datapoint, I was recently [very] pleasantly surprised to learn that YouTube was able to restore a video that I had accidentally deleted. It went back to being live as if nothing had happened—though my contacting support was admittedly right after the deletion happened.
35mm 5y ago
Worth a try but I’ve tried doing this before and they said it’s outside the scope of paid support.
- ldng 5y ago
  Yet it's not outside of the Law. At least in Europe, they MUST give all your data back.
  - tremon 5y ago
    Assuming you mean the GDPR, it only obligates them to share the data they have. If their response is "we don't have that data any more", then the GDPR won't help you. It opens them up for a massive claim if it later turns out that they really did have the data somewhere, though.
    - ldng 5y ago
      The "we don't have that data any more" should only apply to "I closed my account" not to the "we unilaterally chose to close you're account and delete all your data just because we can". And if they jumped the gun and can't, they must pay for the prejudice. Period. They bet on people not suing and they're right most of the time customer don't demand. It does not mean, just because they actually get away with it by play dumb, that there are not legal basis.

WalterBright 5y ago

You can buy 16 terabyte drives for $300 or so. Cost shouldn't be a problem to back up your videos.

iforgotpassword 5y ago
I second this. I can't fathom that on HN of all sites people recommend renting backup space from the same big tech companies that continuously fuck over people. Setting up a NAS is not magic. Setting up two isn't either. Put one in you parents basement and check on it once every year when you visit for Christmas. If you don't need 16TB go with an SSD (although with the current contamination situation not the best timing to buy flash).
- Cynosaur 5y ago
  Aren't you making it sound a bit too easy and straightforward than it actually is? It might be that simple for you but buying a NAS device like from synology can be pricey too, setting it all up so it is secure but network accessible, and then also handling possible drive failures down the line (and the chances of that are much higher than google fucking you over IMHO) is much more than an average non-tech-savvy user would be interested in.
  - iforgotpassword 5y ago
    Yes, I wouldn't recommend my mom do it. This was with the average HNer in mind, but even an average tech enthusiast could do it. I think the OP commented here too saying he is a techie. But tbh I think even getting and external drive and backing up to it should at least be done in parallel to any cloud service you choose.
  - lbriner 5y ago
    Maybe not really easy but it is really just due-diligence and getting a solution that is worth the money you are making by providing the service. If you weren't a mechanic, you wouldn't avoid maintaining a car and then get upset that it broke down eventually, you would pay a specialist to maintain it for you. Youtube is cool because it's "free" but you also get almost zero support from Google and they are pretty the worst in my opinion. I have got decent support from AWS and Microsoft (never tried with FB).

vero2 5y ago

> multiple NAS and distributed them geographically and used Syncthing to keep them updated As some one working in storage, please do not get tons of NAS they are pain to manage eventually. Example: linus-tech-tips could not do it properly. If you use youtube-dl how large is one video? Lets say 2 GB. I presume, every week you create have 2 videos? Then 4GB per week. Google gives you $20 for 100 GB. Every year create a new account - like - company name.2022 dump everything there. Hire an clerk to make sure the credit card is payed every year. That way you do not lose all videos at a time. Heck you can even create free Google drive account every few months and segment them.

manigandham 5y ago
That's unnecessarily complicated and expensive. Object storage like Wasabi and BackBlaze B2 are $6/year for 100GB, and much easier to upload, maintain and retrieve from in the future.
Freak_NL 5y ago
Recommending Google as a backup in case Google decides to delete your videos sounds a bit suboptimal. Go with hosted backup; sure, for this use case that makes sense. But do so with a different company.
- vero2 5y ago
  acquiring talent to manage google drive is easy. Not cheap or easy for Amazon/B2.
  - throwuxiytayq 5y ago
    Please note that Google are perfectly happy to ban all of your accounts simultaneously. And your neighbors' accounts, apparently. https://www.reddit.com/r/androiddev/comments/ckoej1/googles_practice_of_associated_account_ban_aka/ https://www.reddit.com/r/androiddev/comments/ckoej1/googles_...
    - vero2 5y ago

tlonny 5y ago

> multiple NAS and distributed them geographically and used Syncthing to keep them updated I decided to look at using AWS S3 to archive my media library. If you use the `DEEP_ARCHIVE` storage class, I believe costs are ~$1 per month per TB. The downside of such a storage class is that it takes ~48 hours for files to be "defrosted" and ready to download. Before I push any of my media (that I have legally purchased), I also encrypt it to hedge the risk that AWS suddenly decides to start doing automated scans/fingerprinting for copyrighted content. I assume that's overkill but w/e. Here is a CLI I wrote to make the above a bit easier: https://gitlab.com/tlonny/dfrost https://gitlab.com/tlonny/dfrost

gaius_baltar 5y ago

First, IANAL. This is really dependent on where you live, but as you are paid customers I would suggest to file a lawsuit in a small claims court. Seriously! Google is not playing its part on a legally binding contract, and money exchanging hands is enough to make it valid for the courts. The amount of work poured into these videos, compared with the equivalent compensation that you would have to pay for an employee to make the same work would also qualify for damages. This can also protect you against retaliation from google. But again, IANAL and check our local legislation carefully before. Where I live it's a thing a person can do in an afternoon (usually takes less time than fighting big companies' support online), do not require lawyers, and the other part also can not bring a lawyer to conciliatory session (for big companies this is usually a sham: even if their representative does not have a lawyer present at court, be sure that s/he was heavily counseled by company legal staff before and everything presented was proofread by them too).