9 ms·
Debugging a weird 'file not found' error
- ronsor 5y agoGo is dynamically linked in order to use glibc's name resolution functions, as it doesn't support all of glibc NSS's (arguably bloated and mostly unneeded) functionality. Anyway, I too found the "file not found" error to be terribly unhelpful the first time I encountered it half a decade go. I thought I was going crazy.
- mobilio 5y agoldd is always on assistance to see why executable file can't be loaded.
- dgfitz 5y agoYeah I had this error once and ldd solved it iirc.
- justin_oaks 5y agoMisleading error messages are the worst. As a software developer, it's bad enough that I have to constantly challenge my assumptions to try to debug something. It's even worse when I have to think "What part of the system is lying to me?" Recently I upgraded Jira Service Desk (new name: Service Management) to a new version. After the upgrade I got an error in the administrative section of the app. It popped up an error message saying "Something is wrong. Contact administrator." Ugh... but I'm the administrator! And I don't know what's wrong. Maybe it's not a misleading error message, but it's certainly an unhelpful one. Since software can be a lying devil, I am happy that I can make use of so much open source software. I've often delved into the source code to understand what's going on. Once I do that I can either fix the issue, work around it, or even submit a PR upstream. It's much harder to figure out what's going wrong with closed source software. And even if I do figure it out, I'm powerless to change the situation. Now if you'll excuse me, I'll be contacting Atlassian support to help me fix the "Something is wrong" message.
- stefs 5y agowriting good error messages is hard* and a potential security issue if it leads to information leakage. for this reason in my team (not jira) we opt for "sorry, something is wrong" per default in the UI (with maybe the real error message in the logs) and only provide the real message if deemed necessary. * and good developers aren't necessarily good writers
- justin_oaks 5y agoYes, it is hard to write a good error message. Sometimes it's not necessary to write a good error message if you can provide something that the user can Google or at least something that's useful to someone who can investigate further. For example, the old "Error occurred, check the logs" is useful because it tells me that there is info in the logs. It's even better if it says "Error 4824 occurred, check the logs" because it gives me an error code to try to look up. Better yet is "Error 4824 occurred, check the foobar-error.log file for transaction_id nv3kd32dx9" which also gives you which error log to check AND gives you a unique ID to grep for. Sadly, I've NEVER seen that type of error message in any software.
- umvi 5y agoSalesforce is like this too! Cryptic error messages that say "contact admin" when I am the admin.
- anyfoo 5y agoPlaying around with a mainframe OS from the 70s (I'm weird like that), I realized that IBM has that wonderful thing where nearly every message has a code that you can use for lookup. In manuals or online, for example. If you ever used OS/2 you might have seen that, too. The problem is, the original handbooks are hard to come by[1], and while another wonderful thing (in some aspects) is that a modern, current mainframe is still compatible with that old code from the 70s, and so you can use their resources, you sometimes encounter error codes where the description is basically just "contact IBM". Most likely because IBM really does not expect you to encounter that error on a mainframe delivered since, say, the Berlin Wall fell. Even if I wanted to, I doubt IBM wants to sell me a support contract for their 4 decades old last public domain version of their mainframe OS on an emulator. [1] ... and especially on that old OS the message itself can be cryptically short, likely because even a few bytes still cost serious money in the beginning. You still see that in bootloaders.
- ghotli 5y agoThis is on the top of my list of "least helpful error messages" and isn't just a golang thing, any dynamically linked executable will throw this error in an unexpected execution environment. Cheatsheet (standard issue hello world -- hello.c) # gcc -o hello ./hello.c # file ./hello ./hello: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, BuildID[sha1]=1e41ccb72d7a36ab5ac0c2c11bcd0e994e5eb013, not stripped This bit is what throws that error if unavailable: "interpreter /lib64/ld-linux-x86-64.so.2" Also, if ldd / file are unavailable but readelf is: # readelf -a ./hello | grep interpreter [Requesting program interpreter: /lib64/ld-linux-x86-64.so.2] # readelf -a ./hello | grep NEEDED 0x0000000000000001 (NEEDED) Shared library: [libc.so.6]
- Waterluvian 5y agoSomething I didn’t realize I would love about static languages after years of Python: programs more often crash at the error site. Python often keeps working for a while until the duck isn’t shaped correctly. The number of times I’ve had to unravel a bug where we iterated on a string rather than the list containing strings…
- anyfoo 5y agoThat is exactly one of the reasons why people advocate for static languages. More so, often they don't even compile if you mixed up your types somewhere (the "more static" and "more strict" the language is, the more that is true).
- justin_oaks 5y agoI heard this referred to as a language's tendency to "keep on truckin'". Instead of causing an error by dividing by 0, JavaScript, for example, will return Infinity which corrupts any further calculations. It can be caused by dynamic typing but it also creeps into any language that allows implicit type conversion. Java, for example, is statically typed but allows some implicit type conversion in primitive types. The problem is that an implicit type conversion is easy for a programmer to miss, and thus they don't realize that that the variable doesn't have the value or type they're expecting.
- pletnes 5y agoAFAIK, division by zero in IEEE 754 floating point shall return infinity, which is a valid floating point value. For once, javascript does the right thing.
- justin_oaks 5y agoFair enough. I was trying to come up with something weird that JavaScript does with numbers and I guess I chose the wrong thing.
- 5y ago
- Thespian2 5y agoReminds me of copy-paste shell script error, between windows and mac, where first line of shell script turned out to be #!/bin/bash^M That is, a non-printing carriage return after "bash" and running the shell script "helpfully" kept telling me: "/bin/bash: no such file or directory"
- anyfoo 5y agoDoes the shell filter the carriage return out in the error for some reason? I would more expect something like: /bin/bash : No such file or directory (Not that that's more helpful to a user, but at least a trained eye would immediately get very suspicious.)
- kymaz 5y agoI guess it depends greatly on what version of bash and such, but with something recent, trying to execute the script will escape the problem characters when printing if it prints the interpreter name at all, and possibly also say 'bad interpreter: no such file or directory" ...since there's more than 2 valid line ending configurations?
- anyfoo 5y agoYeah that sounds well plausible, especially because there seem to be many (minor and major) versions of bash still around in different contexts. Not to speak of any other shell. Escaping seems best! The "bad interpreter" being there or not might be OS-specific, though. Unless some bash versions use fork+execve vs. posix_spawn or similar and that makes a difference there. I haven't checked. Overall I can see how even some current stuff might give the confusing message...
- Thespian2 5y agoThat would be a "line feed" (0x0A) - Carriage return is (0x0D) (ascii values 10 and 13, respectively) the Mac/Windows conversion is the difference between CR and CR/LF All I can guess is that in terminal, when that particular version of bash and OS interacted, it didn't go back to the start of line, and the CR was unprintable, so it all ended up on the same line. I'd expect to have seen something like what you describe with the line break, if it were a bare LF, but in my case, it wasa bare CR.
- anyfoo 5y agoI've been dealing with POSIX (or more generally "unix-y") environments for so long, when I see "No such file or directory" I automatically tend to think "ENOENT" and not actually "file not found", as in something did not exist. Many system calls (sysctl as just one example) happily return ENOENT when they did not find a thing no matter whether it is even remotely related to a file. I picked sysctl because it also can return ENOTDIR and EISDIR for something not related to directories at all. It's commonly not that much of an issue when you handle the error in code, but if the error gets "helpfully" bubbled up with its strerror description (instead of just a number), it can sure confuse people. Somewhat simplified: errnos were at some distant point in the past divined into existence, and system calls and system level C APIs had to make with what they got since then.
- assbuttbuttass 5y agoI'm intimately familiar with this particular error, having debugged it a few times on NixOS. Binaries tend to assume that /lib/ld-linux.so exists
- thundergolfer 5y agoHad the same because of using Nix. A file was mounted from an Amazon Linux 2 VM into a Nix-based container, and ran into this exact problem. One of the more interesting debugging adventures I had.
- userbinator 5y agoI’ve run into this “file not found” error a couple of times, and it feels kind of mind bending because it initially seems impossible I've seen this error often enough to know that if you're doing anything with execution, "file not found" = "the file, or one of its dependencies, was not found". On the other hand, "file not found" on an open() call when you seem to see that it exists --- which I have actually seen --- should take you down a different path of troubleshooting, such as making sure you're in the right path and on the right machine.
- kevin_thibedeau 5y agomax-width: 45em;
- 1vuio0pswjnm7 5y ago"step 4: use readelf" "$ ldd ./serve" "not a dynamic executable" No need to use ldd, can use readelf, as in step 4 $ readelf -d ./serve There is no dynamic section in this file. http://www.catonmat.net/blog/ldd-arbitrary-code-execution/ http://www.catonmat.net/blog/ldd-arbitrary-code-execution/ "And statically linking in this case doesn't even produce a bigger binary (for some reason it seems to produce a slightly smaller binary?? I don't know why that is)" To force use of built-in net libs instead of system ones, and remove debugging info to make the binary a little smaller, something like this $ CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -tags netgo -ldflags '-w -extldflags "-static"' -o serve *.go re: "File not found" I encounter this "weird bug" all the time because I like to build smaller, self-contained, no dependency systems and started doing so long before Docker existed. Static binaries are one way to solve the missing linker problem. Chroots and the like are another. I will sometimes create an alias that runs the program using chroot. Then I can keep the chroot directory as small as possible, populating it with only those resources the program actually needs. This is praticularly fun when it is an older program, no source available, and requires older versions of glibc, and other libraries. Dynamic linking in a time of GBs of core memory and TBs of storage is something I still question. People say dynamic linking is useful for the library "version control" it allows, but this just becomes a PITA for someone wanting to use older programs. Lots of folks compiling static binaries today for their Github repos only do so with glibc, which is hostile toward static linking. For someone using musl, this is no better than a dynamically-linked binary.
- drewg123 5y agoWhy does strace not show the kernel looking for, and failing to find, the rtld? (/lib64/ld-linux-x86-64.so.2) When fooling around running linux binaries on FreeBSD, ktrace/kdump show the (FreeBSD) kernel looking for the rtld..
- loeg 5y agostrace just watches syscalls, like truss. execve(2) returns ENOENT and that's it.
- chrismorgan 5y ago$ echo > /nonexistent-dir/filename zsh: no such file or directory: /nonexistent-dir/filename $ echo > /nonexistent-dir/filename bash: /nonexistent-dir/filename: No such file or directory In the middle of a shell script or makefile recipe, this can be hard to spot, and it may take a while to realise why it’s actually failing—that it’s the directory that doesn’t exist, not the file (which of course it doesn’t exist, you’re trying to create it!). I recently hit a messy race condition related to this: https://github.com/alexforster/libbpf-sys/commit/e65b4962e7de5fa0755b274d707d19c64c176913 https://github.com/alexforster/libbpf-sys/commit/e65b4962e7d... (described in the commit message). I’ve been wondering about filing bugs against zsh and bash to be a little bit cleverer about their error message in this specific case, to only mention the nonexistent directory rather than the full path.
- saagarjha 5y agoI work on iSH, which ships Alpine Linux for iPhone, so I know this error by heart: people constantly try to run software that relies on glibc, which fails on Alpine because of the missing interpreter. I guess it’s certainly a way to teach people about the differences in libcs, but it’s generally hard for people to figure out for exactly the reason given in the blog post. I wished that shells did a bit more here and tried to figure out if the ENOENT was from a missing interpreter or binary, but maybe I’m asking too much…
- pletnes 5y agoThis reminds me of k8s’ «out of memory error» which turns out to be «kill -9» / SIGKILL which can also happen by k8s’ own «kill process when health check fails». I have no idea who thought this was a good idea, but this needs to be fixed. Somehow.
- malka 5y ago> It works! I checked, and that’s an alternative way to fix this bug – if I just set the CGO_ENABLED=0 environment variable in my build container, then I can build a static binary and I don’t need to switch to the golang:alpine container for my builds. I kind of like that fix better. With a static binary, you can even start your Dockerfile with `from scratch`. Feels good to have 5kb docker images
- TYMorningCoffee 5y agoSuppose there was no Google result, and you were the first to ask about this. What would have been your next step?
- unstatedAnswers 5y agoSuch a good blog post, from start to finish. There aren’t a lot of expertly-written in-the-weeds-tech blog posts, but this one was crafted so well. I guess that’s why Julia Evans’ name is so recognizable.