6 ms·
On macOS, Backblaze ships with 21 identical copies of the same executable, nearly 200 MB in all, presumably because they don't realize you can just execute one
by rgovostes 5y ago
On macOS, Backblaze ships with 21 identical copies of the same executable, nearly 200 MB in all, presumably because they don't realize you can just execute one binary n times (with different argv[0] if they'd like) and they haven't written their code to be thread safe.
Needless to say it doesn't instill confidence in the quality of software engineering that I rely on for disaster recovery.
% openssl md5 /Library/Backblaze.bzpkg/bztrans*
MD5(/Library/Backblaze.bzpkg/bztrans_thread00)= 772308dbd9b8083f4dc1c31bfe6a28da
MD5(/Library/Backblaze.bzpkg/bztrans_thread01)= 772308dbd9b8083f4dc1c31bfe6a28da
...
MD5(/Library/Backblaze.bzpkg/bztrans_thread19)= 772308dbd9b8083f4dc1c31bfe6a28da
MD5(/Library/Backblaze.bzpkg/bztransmit)= 772308dbd9b8083f4dc1c31bfe6a28da
- dataflow 5y agoEh, even Clang does this. Install it on Windows and you'll get clang-cl.exe, clang++.exe, clang-cpp.exe, and clang.exe, all 90 MB executables, and almost entirely the same executables, but not bit-for-bit identical, so you can't hardlink them. I actually hate this too, but my point is it doesn't necessarily say much about software quality.
- rgovostes 5y agoThis makes slightly more sense if it's the case that they are statically linking the Clang and LLVM libraries into each tool. Each one has a slightly different main() (e.g., one parses gcc-style args) but ends up using the same library functionality.
- jfrunyon 5y agoBut these are bit-for-bit identical.
- dataflow 5y agoEdit: You're talking about the Backblaze binaries? Yeah that's why I said Clang's is almost bit-for-bit identical, so it's clear to everyone it's not 100% the same as the Backblaze situation. Not sure what your complaint is about my comment though? It's still practically the same problem, and you can solve it just as well with the argv[0] tricks or whatever. Or are you saying your heuristic for determining the underlying engineering quality changes its output based on 1 bit flip? Identical executables is awful engineering, but a few bytes different (out of 90MB) is great engineering?
- somehnguy 5y agoThe 'these' being referenced here are the Backblaze executables, not your clang executables.
- monsieurbanana 5y agoThey're not saying the clang are bit by bit identical, but the blackblaze ones.
- jfrunyon 5y agoNot sure why you think clang having multiple not-identical executables is relevant to Backblaze having multiple identical executables.
- dataflow 5y ago> Not sure why you think clang having multiple not-identical executables is relevant to Backblaze having multiple identical executables. Really? You genuinely don't see why I would think a case with >99.99% identical executables might be relevant to a case with 100% identical executables?
- 5y ago
- mshockwave 5y agoThis only happens on Windows, on *nix systems these are all pointing to `clang-<version>`
- meinersbur 5y agoThey are identical copies of each other. Since symlink support is not universal on Windows (Possible since around the time WSL was introduced, but still requires either special privileges or developer mode enabled), many tools including GNU autotools fall back to copying. Since recently (https://reviews.llvm.org/D99170 https://reviews.llvm.org/D99170), LLVM et least tries to create a symlink on Windows, but still caused problems because this is kind of unexpected by Windows. The NSIS installer used by the official Windows installer (https://llvm.org/builds/ https://llvm.org/builds/) does not support symlinks at all such that the installer includes 4 identical copies of Clang. A clean solution would be to have stub executables that link to a central libClang.dll/libLLVM.dll (-DCLANG_LINK_CLANG_DYLIB=ON), but this is not supported under Windows because a DLL can export at most 2^16 symbols. Some work would need to be invested to make process launching on Windows work differently than on other platforms, but then disk space is not that much of a problem.
- dataflow 5y agoThe Clangs are different from each other in some installations, and identical in others. In particular the Visual Studio installation doesn't have identical binaries, but the MSYS2 installation does. You'll have to check yours. For example, if you dir "%ProgramFiles%\Microsoft Visual Studio\2022\Community\VC\Tools\Llvm\bin\clang*.exe" you'll see they're slightly different sizes. (Also note that in this case you don't need symlinks per se, just identical binaries with hardlinks. Each executable could inspect argv[0] and figure out what it's invoked as, and behave accordingly.) What I don't understand is why can't they just bundle everything into a single DLL, then make whatever stub executables they want just call the exported main() in that DLL. I think that might be what MSYS2's copy already does, though I haven't checked. That DLL wouldn't need to export anything else, just the main() function that each .exe stub would forward to. And it can handle everything else internally if it so desires.
- cma 5y agoDifferent sizes doesn't always mean different sizes (I'm not sure if that is completely true using 'dir'): https://superuser.com/questions/1353064/why-does-size-on-disk-vary-when-size-does-not-with-the-same-set-of-files https://superuser.com/questions/1353064/why-does-size-on-dis...
- dn3500 5y agoEven if they're not very smart couldn't they hard link these?
- rgovostes 5y agoI had previously symlinked them all to one copy and there was no apparent consequence for my backups.
- gunapologist99 5y agoWhat about your restores ;)
- chomp 5y ago> presumably because they don't realize you can just execute one binary n times This seems needlessly dismissive. I feel like they definitely know that you can execute the same binary multiple times.
- brianwski 5y ago> > Backblaze ships with 21 identical copies of the same executable > > This seems needlessly dismissive. I feel like they definitely know that you can execute the same binary multiple times. I'm the programmer at Backblaze that made the copies on purpose, I wrote some extra code to do this, and it's meant to help us debug certain things. Yes they are identical, the installer only ships with one copy of the executable, the installer then makes the copies on purpose. I get to explain this from time to time. :-) In Windows when you want to know what is going on behind the scenes, you can bring up Task Manager and look at the different names of the different processes that are running. On the Macintosh this is called Activity Monitor, same sort of thing. The different names for the executables are for different "threads" which have different roles. Backblaze is multi-threaded to get higher performance. The parent coordination process is called "bztransmit". But when doing the actual transmission it spawns the bztrans_thread01, bztrans_thread02, bztrans_thread03, etc. So BEFORE I made multiple copies of the executables with different names, a customer would say "bztransmit is hung" or "bztransmit is using up too much memory". There was very little visibility into this. But now that I made multiple copies with different executable names, when the same customer says "bztrans_thread03 is hung". Or they say "bztransmit is using too much memory". We immediately have narrowed down what to look at. Here is a screenshot showing what "Chrome" looks like to me in Windows, and how it compares to how Backblaze's bztransmit looks like to me in Windows: https://i.imgur.com/KOJHJ9Q.jpg https://i.imgur.com/KOJHJ9Q.jpg In that screenshot, you can see there is the "main thread", and "worker threads". Meanwhile chrome is just one big list of processes all named the same thing (see the screenshot). I prefer the Backblaze system, but I understand it upsets some customers that prefer the chrome experience. That's it. It's not some huge mystery. One question asked here was do we know you can launch the same executable twice? Yes, and we do that. The bztrans_thread05 is launched for thread 05, thread 25, thread 45, thread 65, etc. It's THREAD_NUMBER mod 20. Here is what it looks like to hit 500 Mbits/sec upload speeds, this isn't photoshopped, it's a real screenshot on my development computer: https://i.imgur.com/hthLZvZ.gif https://i.imgur.com/hthLZvZ.gif Another question is: why not use hard links or symbolic links? That's the only real optimization possible here, everything else was on purpose. The answer is not an excuse, it's just an explanation if you are curious. The software we develop at Backblaze is cross platform, so what we like to do is make the most general form first that will always work, then if customers complain or we want to refine it we special purpose code in per file system or per platform. The most general thing to do is make full copies. We could then go on to make links on the Mac WHEN POSSIBLE and the equivalent on Windows WHEN POSSIBLE, but it never became a large priority. The reason I can't use one technology is we support several file systems, and not all of them are the same or support the same technology for links. Every feature we have is the result of prioritizing it over working on other things. Until recently, we did not have a lot of funding or an infinite supply of programmers, so we had to choose what order to implement each feature in. I'm not saying we got all the priorities correct, or that we did things in the correct order. I don't really even think there is one correct order. For example, some individual home user customers prefer saving 180 MBytes of their valuable boot SSD space over me implementing single sign on for our corporate customers. On the other hand some corporate customers DEMANDED single sign on or they wouldn't purchase the product at all. They are both correct, but there is only one of me, so we made some judgement calls and left the multiple copies and worked on single sign on. Some customers were happy, some are miserable. We do have open client recs for both Windows and Mac programmers, so if you would like to make a good salary, full benefits, and help us out, come join us! :-)
- KennyBlanken 5y agoThis is hilarious. Their marketing drivels on about how they employ a bunch of Apple engineers and the MacOS client is top-notch.