6 ms·
Backblaze seemingly does not support files greater than 1 TB
- Aeolun 2y agoThis seems at the same time reasonable (no files larger than 1TB sounds fair, I don’t think anyone reads ‘no file size restrictions’ and imagines a one terabyte file), and not, because support should realize that that might have something to do with the issue and get someone technical involved.
- swiftcoder 2y agoUndocumented limits are a classic way to blow up traffic to your support centre. Please folks, if your service has limits, document them!
- OtherShrezzing 2y agoDocumenting also helps your own engineers understand that there are limitations. The back-end team might understand that there's a 1TB limit. If the front-end team doesn't, they could cost you a tonne of bandwidth uploading 99% of a 1.001TB file to the server before it gets rejected.
- michaelt 2y agoDocumenting wouldn't hurt, I'll agree. But I doubt they're getting much support burden from this; the product targets home users, providing fixed-price single computer backups. This isn't their B3 blob storage offering. The most bloated triple-A games out there are only ~100GB. The entirety of OpenStreetMap's data, in XML format, only reaches 147 GB [1]. The latest and greatest 670-billion-parameter LLM is only 600 GB... and they ship it as 163 x 4.3GB files [2]. Among PC gamers [3] about 45% of people have less than 1TB of hard drive space, total. IMHO it's very unlikely they have a big support burden from this undocumented limitation. It's clearly a bug though - trying to upload the same file in every single backup, failing every time? There's no way that's the intended behaviour. [1] https://planet.openstreetmap.org/ https://planet.openstreetmap.org/ [2] https://huggingface.co/deepseek-ai/DeepSeek-R1/tree/main https://huggingface.co/deepseek-ai/DeepSeek-R1/tree/main [3] https://store.steampowered.com/hwsurvey https://store.steampowered.com/hwsurvey
- birdman3131 2y agoYou have never played Ark I take it. I am sitting at ~700gb between Ark: Survival Evolved and Survival Ascended.
- madeofpalk 2y agoIf it actually wasn't supported, surely the client would know this and wouldn't attempt to continiously upload it. Seems like there is a bug somewhere - whether artificially imposing a file size limit, or in not imposing the limit correctly.
- jagged-chisel 2y ago> … surely the client would know this ”Should” most certainly. However, often client teams don’t work so closely with backend teams. Or the client team is working from incorrect documentation. Or the client team rightfully assumes that when it reports a file size before uploading, the backend will produce an acceptable error to notify the client.
- baobabKoodaa 2y agoOh my, this almost sounds like it just might have unintended bad effects when you lie in your marketing about what limits your service has.
- davidt84 2y agoI read "no file size restrictions" and assume I can create a file as large as the storage space I can afford. What else would I assume? If there's a 1TB limit I would expect that to be described as "create files up to 1TB in size".
- eterevsky 2y agoI would assume some limit no higher than 2^64, since all common file systems have file size limits: https://en.wikipedia.org/wiki/Comparison_of_file_systems https://en.wikipedia.org/wiki/Comparison_of_file_systems
- 8organicbits 2y agoWhen common file systems like NTFS have limits like "16 TiB (17.59 TB) to 8 PiB (9.007 PB)", a 1TB limit seems below what is advertised.
- dpacmittal 2y agoWhy couldn't they just say 1tb limit?
- baobabKoodaa 2y agoThis is what bothers me as well. I don't think you would find a single customer who would go "what, only 1TB files?" I guess different kinds of people are drawn to different careers, and people with loose morals, hubris, and a propensity to lie are drawn to marketing. Don't even need an incentive to lie, it's just who they are.
- pbalau 2y ago> I don't think you would find a single customer who would go "what, only 1TB files?" I think you will find that there are people that see "limit <humongous>" and pick the "no limit" or no limit stated option. > people with loose morals, hubris, and a propensity to lie are drawn to marketing You are jumping the gun here.
- mort96 2y agoIt's a back-up service, not a cloud storage service. If I have a 1TB file on my machine, and I want to back up my machine, I want that 1TB file to be backed up.
- londons_explore 2y agoAnd a 1tb file is pretty common if you take a backup of a 1tb disk drive into a disk image.
- mort96 2y agoCertainly, I've regularly had single image files and tar.gz files of that magnitude on my machines which are precisely back-ups of old laptops when I get a new one etc.
- joseda-hg 2y agoNot arguing your main point, but why wouldn't you compress a backup?
- londons_explore 2y agoMy disk's tend to be full of films that barely compress.
- kaivi 2y agoThese limits are not reasonable at all. You are going to curse the Backblaze or AWS S3 before you learn to never pipe the output of pg_dump into an object store like that.
- atYevP 2y agoYev here -> we're actively looking into it, we can upload files >1TB so it's a bit tough to chase down.
- 2030ai 2y ago[dead]
- tobyhinloopen 2y agoI'm going to ask the wrong question here, but... What files are >1TB? Raw video footage?
- vman81 2y agoI'm guessing poorly configured log files?
- eptcyka 2y agoDisk images, archives.
- earth-adventure 2y agoEncrypted disk volumes from virtual machines for example.
- bifftastic 2y agoBackblaze does not back up disk images. It's a documented limitation. I just found this out the hard way.
- baobabKoodaa 2y agoThat can't be right. After all, "Backblaze automatically finds your photos, music, documents, and data and backs it up. So you never have to worry."
- bifftastic 2y agoActually to be fair on BB you can override this behaviour. But you have to know...
- ykonstant 2y agoCan you share the know?
- aosaigh 2y agoI like the log message "BadBadBadChunkRecord". I wonder what BadChunkRecord and BadBadChunkRecord are? Is there a VeryBadBadBadChunkRecord? I've been trying to replace all my various backups (Time Machine, Backblaze, CCC) to use a single tool (Arq - https://www.arqbackup.com/ https://www.arqbackup.com/) The Backblaze client is the next to go. To be honest, I haven't had too many issues with it, but the restore interface in particular is pretty poor and slow.
- Crosseye_Jack 2y agoAt least the file wasn't Michael Jackson bad (bad, bad, really, really bad).
- jagged-chisel 2y agoI think it’s a triple negative. A BadChunkRecord is, well, a bad chunk record. A BadBadChunkRecord is a chunk record that’s bad at being bad, so it’s good. One can see the logical progression that leads BadBadBad… to be a bad one. Of course, being bad at being bad could just be a different kind of bad. A good BadChunkRecord explains the problem with the chunk. A bad BadChunkRecord might have too little information to be a good BadChunkRecord. A bad BadBadChunkRecord could be an unhandled exception with no other details and the fact that a ChunkRecord is even involved is assumed and therefore questionable.
- dmd 2y agoThe møøse responsible for the BadChunkRecord have been sacked.
- handsclean 2y agoWe can’t neglect the possibility that BadBadBad is a term that the Backblaze code base now defines directly, not just in terms of its etymological root “bad”. Its name and definition may overlap the common word, but also carry additional meaning, like “logged and non-recoverable”.
- smatija 2y agoThat's a 4-base number system. Next in progression is WorseChunkRecord, then you have BadWorseChunkRecord and so on to WorseWorseWorseChunkRecord, when you finally get to HorribleChunkRecord.
- raverbashing 2y agoHonestly? Be a good customer. Help them help you and help yourself in the meanwhile Having a 1TB file sucks in way more ways than just "your backup provider doesn't support it". Believe me.
- davidt84 2y agoPerhaps you could say something concrete, rather than vague waffle and assertions?
- 542354234235 2y agoPerhaps you could say something concrete, rather than vague criticism?
- davidt84 2y agoWhat am I supposed to say in response to a comment conveying the message "be a good customer" with no further elaboration and the source "believe me"?
- aleph_minus_one 2y agoThe problem is that it is often possible to solve the problem for (your) future, but you cannot get rid of the existing large files (if only for legacy reasons). One example: A Git repository that I created contained files that are above GitHub's file size limits. The reason was a bad design in my program. I could fix this design mistake, so that all files in the current revision of my Git repository are now "small". But I still cannot use GitHub because some old commit still contains these large files. So, I use(d) BitBucket instead of GitHub.
- Symbiote 2y agoYou can rewrite the history of the repository to remove the huge files, if you wish. Of course, all the commit identifiers change.
- Borg3 2y agoThe question is, where that limit comes from.. It sounds weird. 40bit file size record? like why? I recently was fixing my own tool here and had to shufle some fields. I settled up at 48bit file sizes, so 256TB for single file. Should be enough for everyone ;)
- spuz 2y agoIt sounds like an arbitrary limit set by some engineer without too much thought and without the incentive to document their decision. I do this sometimes as well when dealing with untrusted data. Sometimes you don't want to have to think about the knock on effects of supporting large file sizes and it's easier to just tell the client please don't do that.
- redleader55 2y agoI'm speculating here based on my experience working on a storage fabric product in the past. First - it's sensible to have limits to the size of the block list. These systems usually have a metadata layer and a block layer. In order to ensure a latency SLO, some reasonable limits to the number of blocks need to be imposed. This also prevents creating files with many blocks of small sizes. More below. Second - Usually blocks are designed to match some physical characteristics of the storage device, but 10 MB doesn't match anything. 8MB might be a block size for an SSD or a HDD, or 72MB=9*8MB might be a block size with 9:16 erasure coding, which they likely have for backup use-cases. That being said, it's likely the block size is not fixed to 10 MB and could be increased (or decreased). Whether or not the client program that does the upload is aware of that, is a different problem.
- bakugo 2y agoThe Backblaze computer backup app is kind of a mess in general. The only reason I still use it is because of the unbeatable price, there are many better alternatives if cloud storage prices aren't an issue for you. I've had many issues with it, and poking around at its logs and file structures makes it pretty obvious that it's very bad code. There's numerous different inconsistent naming conventions and log formats. I once tried to edit the custom exclusions config file (bzexcluderules) and gave up shortly after because the format of the file is both completely nonsensical and largely undocumented - whoever implemented didn't know how to do proper wildcard matching and was likely unaware of the existence of regular expressions, so they came up with an unholy combination of properties such as ruleIsOptional="t" (no idea what this means), skipFirstCharThenStartsWith=":\Users\" (no comment on this one), contains_1 and contains_2 (why not contains_3? sky's the limit!), endsWith="\ntuser.dat" and hasFileExtension="dat" (because for some reason checking the extension with endsWith isn't enough?), etc.
- southernplaces7 2y ago>there are many better alternatives if cloud storage prices aren't an issue for you. What are a couple that you particularly recommend for this? I've been using B2 after abandoning the disaster that SpiderOak has become (was a long term user and stuck out of habit), and not exceptionally happy with B2 either.
- bakugo 2y agoRestic is my personal favorite for straightforward file backups. It's simple and well-designed, integrates with rclone meaning it supports any file transfer protocol or cloud storage service you can imagine, and has a decently large community surrounding it. https://restic.net/ https://restic.net/ https://github.com/rubiojr/awesome-restic https://github.com/rubiojr/awesome-restic
- southernplaces7 2y agoThanks for getting back with an answer. Still curious thogh, to what cloud system or platform do you back up? Restic is a solution for organizing and managing the backup process, but not for the remote storage itself.
- actuallyalys 2y agoThe log suggests it doesn’t immediately stop upon noticing the file is over 1 TB—the second entry comes two minutes later when it’s trying to upload the 100,001st block. This makes me wonder whether 100,000 blocks is an intentional limit on file size or whether it’s some sort of internal threshold that isn’t meant to prevent uploading files of that size but is buggy. Perhaps files bigger than 1 TB have slightly different behavior by design and that behavior was broken? Since 2 minutes isn’t long enough to upload 1 TB, it’s either looking at the blocks out of order and skipping ahead to block 100,001 for some reason or noticing that’s the first block that hasn’t been uploaded yet. Another comment suggests 10 MB isn’t a fixed limit. In that case, I wonder if 100,000 blocks is the limit and the intention of the designers is that users with such large files would increase their block size. If so, that should be documented of course, and a scenario. Although it’s still a bit strange that it would apparently try to upload the 100,001st block and not immediately warn the user that the file size is incompatible with their block size.
- deleted 2y ago[deleted]
- atYevP 2y agoYev here from Backblaze -> This is something that our engineering team is taking a look at. We've been able to successfully upload files over 1TB internally, so we're trying to chase down what's triggering those error messages. From what I can gather the client should certainly upload files over 1TB, but something wonky is going on this case. We're looking at it though!