8 ms·

> There are bugs where you know exactly how to reproduce it, but it takes a long time (like 20 minutes or way longer) to reproduce the bug. This is hard because

> There are bugs where you know exactly how to reproduce it, but it takes a long time (like 20 minutes or way longer) to reproduce the bug. This is hard because it’s hard to maintain your focus: maybe you can only try 1 experiment per day! Ugh I hate these. It's almost worse than the "only happens sometimes". I'd rather walk down a longer path fast than a shorter path miserably slow. > there’s no output at all Hangs!! I hate hangs. I hate them. Crashes give information. Hangs could be so many things, but it is the absence of information. > 4. one of your assumptions is wrong #1 assumption I always make sure to check now is "is the code that's running actually from the source code I'm looking at". So may "impossible" bugs I run into are in fact impossible... in the codebase I'm looking at. And it turns out what's deployed is some other code where it is very much possible.

poidos 5y ago
> the code that's running actually from the source code I'm looking at Super common when working with Clojure(script)! Bites me in the ass all the time.
- darksaints 5y ago
  Speaking of clojurescript and bad assumptions: Clojure and clojurescript are both dynamically typed, but clojure is strongly typed (type errors throw exceptions), and clojurescript is weakly typed (type errors are technically valid code). Too many clojure enthusiasts act like they're both the same language, but that is an absolutely massive difference. Try the following code out in each: (+ "1" 1). In clojure you get an exception, in clojurescript you get "11" (maybe a warning at compile time which won't show it's face at runtime when the data is fed to you from an API call). That bug silently corrupted analytics data for 3 months for a service I worked on. That alone was the reason I stopped using dynamically typed languages.
  - laurencerowe 5y ago
    > That alone was the reason I stopped using dynamically typed languages. Nitpick, but weakly typed not dynamically typed is the problem here. Python would throw an exception (though I know mostly prefer JS with TypeScript to Python nowadays.)
    - darksaints 5y ago
      Yes, in this particular instance that would be correct. But tracking down that bug made me realize something. A well tested program, fed improperly typed inputs, will fail in different places depending on it's type discipline. With the weakly typed language, it will fail anywhere within your codebase and won't tell you that it is doing so. With the strong/dynamic types, it will still fail anywhere, and while it will fail loudly, it still leaves you with the work of tracking down why. You won't know that it is due to improperly typed IO until you've traced that data flow all the way back from the point of the failure. For large apps, this can be a nightmare. With a static/strong typing discipline, type errors can only occur in one place: exactly at the point where objects are constructed during IO. And let's say that that upstream API provider (that you have no control over) silently switched to bignum strings instead of floats, your strong type system caught the problem immediately, and now you have to accommodate. You change the object, and get a stream of type errors in your IDE, and you go around fixing them all until they disappear. Almost like magic, your program works again. No new tests, no running multiple times until you track down all the type errors. With a dynamically typed language you'll have no such luck.
      - laurencerowe 5y ago
        Frustratingly many statically typed languages are not very strongly typed since they don't force you to check for null values. Kotlin/Swift/Rust really are a huge improvement over Java/Go/C#. At it's best TypeScript feels very similar to programming in one of these OCaml like languages, though you're still vulnerable to errors in untyped dependencies or people on your project abusing the `any` escape hatch.
  - sesm 5y ago
    (+ 1 “1”) is not a type error for CLJS, because it’s not a type error for JS. Clojure is designed as a hosted language and it shares all the basic types and operations with the host platform.
    - darksaints 5y ago
      That's perfectly fine. In fact, it would probably have too much overhead to try to build a dynamic/strong runtime on top of javascript, so adopting the platform's type discipline was probably a wise choice. However, the type discipline is a fundamental aspect of the definition of a programming language, and you can't exactly call it the same language if it has a completely different type discipline on different platforms. And it would be nice if clojure advocates stopped lying to people about how it is the same language. Be transparent about how different it really is, even if it looks the same on the surface.
  - mbrodersen 5y ago
    That’s quite bad. It basically means that Clojure and clojurescript are different languages. So you can’t blindly share client/server code.
steerablesafe 5y ago
The rise of reversible debugging for C/C++ made debugging much more enjoyable for me. For those tedious to reproduce bugs you can reproduce and record it once and replay it many times with the bonus of reverse-* operations. With rr the execution isn't even slowed that much.
- 2ion 5y ago
  Indeed. I actually quite like hung programs :) Attach a debugger, and now lessee what it is doing...
AlotOfReading 5y ago
At one point, I encountered a low level bug that would only manifest if the computer was physically located in a particular room. It ended up being an undocumented cabling revision, but I revisited a lot of assumptions about the nature of software and reality before getting to that point.
- hutzlibu 5y ago
  "but I revisited a lot of assumptions about the nature of software and reality before getting to that point." I encountered some bugs that felt like this (software only though), where I did not found the all explaining logical solution and ended up rewriting that part. But I saved the whole state of the project and data in a zipped file and keep them locked away - so one day, when I feel like madness (and maybe have better tools avaiable) I will jump into it, either to proof my own stupidity - or the existence of dark magic, poltergeister and demons.
- yummypaint 5y ago
  I have come to love the feeling that the universe is broken when bug hunting. It means the solution is close at hand. If there are still lots of plausible explanations left then it means there is lots of work left to do. Things always seem most insane just before figuring it out.
  - laurent92 5y ago
    And then you find it and you remember the sentence: “Everything that can fail, will fail”.
  - pineconewarrior 5y ago
    Thank you for this positivity! I'll try to remember this next time I am frustrated.
- cortesoft 5y ago
  Sounds like the 500 mile email bug https://web.mit.edu/jemorris/humor/500-miles https://web.mit.edu/jemorris/humor/500-miles
wooly_bully 5y ago
Your last point on the correct code running, ugh. I haven't gotten better at it even thought I'm aware of it more now. Had an incredibly persistent linting config issue just last week, turns out I had the wrong config open for editing in vim.
tyingq 5y ago
The ones I really hate are the ones that disappear when run under a debugger, or with higher logging levels. Usually race conditions.
- rubatuga 5y ago
  Yep, GDB also regularly splits up TCP network packets, so beware of changing stream packet boundaries
- robotnikman 5y ago
  Ah, the dreaded heisenbug
  - amelius 5y ago
    The solution is simple, though: ship with the debugger.
    - laurent92 5y ago
      …with the JVM debug port open, so you can unlock your customers when they are stuck. Even works for software embedded on rockets - After all, network latency to the moon is only 2s, right?
hughrr 5y ago
If anyone is complaining about a few hours here and there, relax. It took 7 years for me to solve an intermittent one once. The issue turned out to be an immeasurably small thread synchronisation issue that went away when we bought new CPUs which were fast enough for it to not happen. In the end this task was rewritten in a single threaded process as the CPUs are now fast enough to complete the work on one thread.
- vmception 5y ago
  This hurts me to read
  - hughrr 5y ago
    That's less than 1% of the pain I experienced there.
- deleted 5y ago
  [deleted]
- ambicapter 5y ago
  This is all well and good but for people like me, I generally need to fix a bug in a week before people start looking at me weird.
opportune 5y ago
It took me some time but I have a good workflow for debugging hangs/deadlocks/etc. If you know what part of the code produces it, you iterate over it indefinitely in a debugger until it hangs, then once you notice the iteration has stopped you “step in” to the debugger. Then you run another script that dumps the current trace back for each existing thread. That should be enough to detect the lock normally. The big problem is if you don’t know how to reproduce the issue in a debugger for some reason.
- TeMPOraL 5y ago
  Last time I had a "silent crash" heisenbug that produced no output, I solved it by system-level tracing (ETW, Event Tracing for Windows). It turned out not to be a crash at all! We had a script in our test harness that cleaned up stray processes, and that script sometimes managed to run long enough that it caught the start of the next test, and killed the tested executables before they managed to output anything.
amalcon 5y ago
>#1 assumption I always make sure to check now is "is the code that's running actually from the source code I'm looking at". Once I got so far as stepping through some malfunctioning code at ASM level on a production machine, only to discover that the ASM had a jump if equal instruction while the source code prescribed exactly the opposite. I compared to the executable binary that was supposed to be there, and it was different from what was actually there. There had been a bit flip somewhere along the line. I now spend more time than I probably should comparing hashes of any executable binary that's acting strangely enough.
- felipemnoa 5y ago
  I've been there a couple of times. Relevant: https://xkcd.com/1316/ https://xkcd.com/1316/
- 13of40 5y ago
  I worked at Microsoft back when they were looking at the Itanium as a Win64 platform. We were making a .Net console application and one day I was in the lab while our daily test automation was running, and the Itanium machines had a bunch of crazy random chars and colors on the console windows. It wasn't causing tests to fail, but I rolled up my sleeves and debugged the issue, thinking it was probably just some buggy test thing. It turned out we were doing something nobody else was doing in .Net at the time, which was to return a structure by value while we were populating the console buffer. Apparently the JIT compiler lost track of which registers it was using to store the struct and overwrote them. In the disassembly it looked like "move some stuff into a register, then move more stuff into the same register". Thought I was losing it for a second. I reckon the moral is don't get into the mindset that the test is always broken. Sometimes it's actually the product.
- lmm 5y ago
  We had a fun one at my first job where the same Java webapp was behaving differently on a couple of servers (this is back when servers were administered manually by sysadmins). We did a recursive md5sum of the homedir on each server and got the same results. Turns out that by default tomcat will use filesystem order when loading jars from its library directory, and some of the servers used different filesystems, so even though each one had the same set of jars in the library directory, they would be loaded in a different order which lead to different behaviour.
lamontcg 5y ago
> Hangs could be so many things, but it is the absence of information. start adding information. you need to know what the code is getting to and/or what its not getting to and progressively narrow it down. spamming debug print statements can be useful if you have no better ideas to get yourself moving. or if you can manage to hook up an interrupt handler to be able to get it to dump a stack trace that'll immediately tell you. or attach a debugger while its spinning in order to get a stack trace. or attach something like strace to dump out system calls or ltrace to dump out libc calls. that can't catch processing spinning in "math" though, but that absence of information is a hint that eliminates lots of possibilities. you may need to add a feature to the software in order to do this kind of debugging, so you write your own USR1 interrupt handler or something. it will be worth it. i think the kids these days use stuff like systemtap?
- MereInterest 5y ago
  One of the best pieces of programming advice I've ever received was from my undergrad advisor. "You'll never win a staring contest with the code." If you're in the middle of debugging and you find yourself staring at the code hoping that it will suddenly reveal answers, that's not a productive position to be in. In those cases, better to figure out what diagnostics can help, and to get more information.
perryizgr8 5y ago
> 20 minutes or way longer One of my coworkers was working on a bug that only manifested itself after ~2 weeks of putting the system under continuous stress. Bad times. Stress tests often seem to unearth strange bugs. Once when we used to sell boxes running a single-threaded real-time OS, the OS vendor gave us a separate machine that would sample the registers and memory of the box 10000 times a second (using jumper cables on the motherboard), in the hope that we would catch the exact state of the system as it crashed after days of stress testing. The sampler machine was a crazy overpowered beast of a machine with really high clock CPUs and oodles of RAM so it could dump as many samples as possible into a ring buffer in memory.
lloeki 5y ago
>> there’s no output at all > Hangs!! I hate hangs. I hate them. Crashes give information. Hangs could be so many things, but it is the absence of information. variants that made me scream include: - misleading information, the kind you can't dismiss as irrelevant, and you go down a thousand rabbit holes that seemingly lead nowhere, but it somehow all ties together into understanding the issue, only after a thousand turns and while keeping the holistic view of it all - useless information. kind of like the one above but every piece of information you get truly leads nowhere Once I had an actual crash that somehow happens before jumping into the program's code (like, before main), but changing your code does toggle the trigger... but when it crashes you have no core and can't attach a debugger, so you're left with debugging on the happy path and constantly think about what could possibly go wrong, and you don't even know what you're looking for. understanding why crt0/ld trips over is not fun.
mateo411 5y ago
I also hate hangs. If you are working with the JVM you can send it a signal kill -3 [pid] which can give you a thread dump. That's a QUIT signal if you are curious. I once found a stuck HTTP request when I did this, so I fixed the issue by adding a timeout to the HTTP connection object. When you are designing a system or a program you should have a way to get diagnostic information to make it easier to fix issues that can arise.
- pixl97 5y ago
  Always design in timeouts. And always make them configurable.
  - mateo411 5y ago
    Yes, I agree.
- bennofs 5y ago
  JVM also has jstack and other tools like jprofd to introspect running vms.
bravetraveler 5y ago
First day of electrical engineering classes we were taught, before troubleshooting a circuit - make sure you're plugged in. Assumptions are evil :)
theshrike79 5y ago
> #1 assumption I always make sure to check now is "is the code that's running actually from the source code I'm looking at". So may "impossible" bugs I run into are in fact impossible... in the codebase I'm looking at. And it turns out what's deployed is some other code where it is very much possible. cough I spent two days doing this when testing different application servers for a Java servlet I was consulting on. Two days of banging my head and wondering why it works on that server and not on that server. I was deploying the new .war to the wrong place and the server autodeploy was picking up the old version every time. I've been doing this for over two decades.

poidos 5y ago

> the code that's running actually from the source code I'm looking at Super common when working with Clojure(script)! Bites me in the ass all the time.

darksaints 5y ago
Speaking of clojurescript and bad assumptions: Clojure and clojurescript are both dynamically typed, but clojure is strongly typed (type errors throw exceptions), and clojurescript is weakly typed (type errors are technically valid code). Too many clojure enthusiasts act like they're both the same language, but that is an absolutely massive difference. Try the following code out in each: (+ "1" 1). In clojure you get an exception, in clojurescript you get "11" (maybe a warning at compile time which won't show it's face at runtime when the data is fed to you from an API call). That bug silently corrupted analytics data for 3 months for a service I worked on. That alone was the reason I stopped using dynamically typed languages.
- laurencerowe 5y ago
  > That alone was the reason I stopped using dynamically typed languages. Nitpick, but weakly typed not dynamically typed is the problem here. Python would throw an exception (though I know mostly prefer JS with TypeScript to Python nowadays.)
  - darksaints 5y ago
    Yes, in this particular instance that would be correct. But tracking down that bug made me realize something. A well tested program, fed improperly typed inputs, will fail in different places depending on it's type discipline. With the weakly typed language, it will fail anywhere within your codebase and won't tell you that it is doing so. With the strong/dynamic types, it will still fail anywhere, and while it will fail loudly, it still leaves you with the work of tracking down why. You won't know that it is due to improperly typed IO until you've traced that data flow all the way back from the point of the failure. For large apps, this can be a nightmare. With a static/strong typing discipline, type errors can only occur in one place: exactly at the point where objects are constructed during IO. And let's say that that upstream API provider (that you have no control over) silently switched to bignum strings instead of floats, your strong type system caught the problem immediately, and now you have to accommodate. You change the object, and get a stream of type errors in your IDE, and you go around fixing them all until they disappear. Almost like magic, your program works again. No new tests, no running multiple times until you track down all the type errors. With a dynamically typed language you'll have no such luck.
    - laurencerowe 5y ago
      Frustratingly many statically typed languages are not very strongly typed since they don't force you to check for null values. Kotlin/Swift/Rust really are a huge improvement over Java/Go/C#. At it's best TypeScript feels very similar to programming in one of these OCaml like languages, though you're still vulnerable to errors in untyped dependencies or people on your project abusing the `any` escape hatch.
- sesm 5y ago
  (+ 1 “1”) is not a type error for CLJS, because it’s not a type error for JS. Clojure is designed as a hosted language and it shares all the basic types and operations with the host platform.
  - darksaints 5y ago
    That's perfectly fine. In fact, it would probably have too much overhead to try to build a dynamic/strong runtime on top of javascript, so adopting the platform's type discipline was probably a wise choice. However, the type discipline is a fundamental aspect of the definition of a programming language, and you can't exactly call it the same language if it has a completely different type discipline on different platforms. And it would be nice if clojure advocates stopped lying to people about how it is the same language. Be transparent about how different it really is, even if it looks the same on the surface.
- mbrodersen 5y ago
  That’s quite bad. It basically means that Clojure and clojurescript are different languages. So you can’t blindly share client/server code.

steerablesafe 5y ago

The rise of reversible debugging for C/C++ made debugging much more enjoyable for me. For those tedious to reproduce bugs you can reproduce and record it once and replay it many times with the bonus of reverse-* operations. With rr the execution isn't even slowed that much.

2ion 5y ago
Indeed. I actually quite like hung programs :) Attach a debugger, and now lessee what it is doing...

AlotOfReading 5y ago

At one point, I encountered a low level bug that would only manifest if the computer was physically located in a particular room. It ended up being an undocumented cabling revision, but I revisited a lot of assumptions about the nature of software and reality before getting to that point.

hutzlibu 5y ago
"but I revisited a lot of assumptions about the nature of software and reality before getting to that point." I encountered some bugs that felt like this (software only though), where I did not found the all explaining logical solution and ended up rewriting that part. But I saved the whole state of the project and data in a zipped file and keep them locked away - so one day, when I feel like madness (and maybe have better tools avaiable) I will jump into it, either to proof my own stupidity - or the existence of dark magic, poltergeister and demons.
yummypaint 5y ago
I have come to love the feeling that the universe is broken when bug hunting. It means the solution is close at hand. If there are still lots of plausible explanations left then it means there is lots of work left to do. Things always seem most insane just before figuring it out.
- laurent92 5y ago
  And then you find it and you remember the sentence: “Everything that can fail, will fail”.
- pineconewarrior 5y ago
  Thank you for this positivity! I'll try to remember this next time I am frustrated.
cortesoft 5y ago
Sounds like the 500 mile email bug https://web.mit.edu/jemorris/humor/500-miles https://web.mit.edu/jemorris/humor/500-miles

wooly_bully 5y ago

Your last point on the correct code running, ugh. I haven't gotten better at it even thought I'm aware of it more now. Had an incredibly persistent linting config issue just last week, turns out I had the wrong config open for editing in vim.

tyingq 5y ago

The ones I really hate are the ones that disappear when run under a debugger, or with higher logging levels. Usually race conditions.

rubatuga 5y ago
Yep, GDB also regularly splits up TCP network packets, so beware of changing stream packet boundaries
robotnikman 5y ago
Ah, the dreaded heisenbug
- amelius 5y ago
  The solution is simple, though: ship with the debugger.
  - laurent92 5y ago
    …with the JVM debug port open, so you can unlock your customers when they are stuck. Even works for software embedded on rockets - After all, network latency to the moon is only 2s, right?

hughrr 5y ago

If anyone is complaining about a few hours here and there, relax. It took 7 years for me to solve an intermittent one once. The issue turned out to be an immeasurably small thread synchronisation issue that went away when we bought new CPUs which were fast enough for it to not happen. In the end this task was rewritten in a single threaded process as the CPUs are now fast enough to complete the work on one thread.

vmception 5y ago
This hurts me to read
- hughrr 5y ago
  That's less than 1% of the pain I experienced there.
deleted 5y ago
[deleted]
ambicapter 5y ago
This is all well and good but for people like me, I generally need to fix a bug in a week before people start looking at me weird.

opportune 5y ago

It took me some time but I have a good workflow for debugging hangs/deadlocks/etc. If you know what part of the code produces it, you iterate over it indefinitely in a debugger until it hangs, then once you notice the iteration has stopped you “step in” to the debugger. Then you run another script that dumps the current trace back for each existing thread. That should be enough to detect the lock normally. The big problem is if you don’t know how to reproduce the issue in a debugger for some reason.

TeMPOraL 5y ago
Last time I had a "silent crash" heisenbug that produced no output, I solved it by system-level tracing (ETW, Event Tracing for Windows). It turned out not to be a crash at all! We had a script in our test harness that cleaned up stray processes, and that script sometimes managed to run long enough that it caught the start of the next test, and killed the tested executables before they managed to output anything.

amalcon 5y ago

>#1 assumption I always make sure to check now is "is the code that's running actually from the source code I'm looking at". Once I got so far as stepping through some malfunctioning code at ASM level on a production machine, only to discover that the ASM had a jump if equal instruction while the source code prescribed exactly the opposite. I compared to the executable binary that was supposed to be there, and it was different from what was actually there. There had been a bit flip somewhere along the line. I now spend more time than I probably should comparing hashes of any executable binary that's acting strangely enough.

felipemnoa 5y ago
I've been there a couple of times. Relevant: https://xkcd.com/1316/ https://xkcd.com/1316/
13of40 5y ago
I worked at Microsoft back when they were looking at the Itanium as a Win64 platform. We were making a .Net console application and one day I was in the lab while our daily test automation was running, and the Itanium machines had a bunch of crazy random chars and colors on the console windows. It wasn't causing tests to fail, but I rolled up my sleeves and debugged the issue, thinking it was probably just some buggy test thing. It turned out we were doing something nobody else was doing in .Net at the time, which was to return a structure by value while we were populating the console buffer. Apparently the JIT compiler lost track of which registers it was using to store the struct and overwrote them. In the disassembly it looked like "move some stuff into a register, then move more stuff into the same register". Thought I was losing it for a second. I reckon the moral is don't get into the mindset that the test is always broken. Sometimes it's actually the product.
lmm 5y ago
We had a fun one at my first job where the same Java webapp was behaving differently on a couple of servers (this is back when servers were administered manually by sysadmins). We did a recursive md5sum of the homedir on each server and got the same results. Turns out that by default tomcat will use filesystem order when loading jars from its library directory, and some of the servers used different filesystems, so even though each one had the same set of jars in the library directory, they would be loaded in a different order which lead to different behaviour.

lamontcg 5y ago

> Hangs could be so many things, but it is the absence of information. start adding information. you need to know what the code is getting to and/or what its not getting to and progressively narrow it down. spamming debug print statements can be useful if you have no better ideas to get yourself moving. or if you can manage to hook up an interrupt handler to be able to get it to dump a stack trace that'll immediately tell you. or attach a debugger while its spinning in order to get a stack trace. or attach something like strace to dump out system calls or ltrace to dump out libc calls. that can't catch processing spinning in "math" though, but that absence of information is a hint that eliminates lots of possibilities. you may need to add a feature to the software in order to do this kind of debugging, so you write your own USR1 interrupt handler or something. it will be worth it. i think the kids these days use stuff like systemtap?

MereInterest 5y ago
One of the best pieces of programming advice I've ever received was from my undergrad advisor. "You'll never win a staring contest with the code." If you're in the middle of debugging and you find yourself staring at the code hoping that it will suddenly reveal answers, that's not a productive position to be in. In those cases, better to figure out what diagnostics can help, and to get more information.

perryizgr8 5y ago

> 20 minutes or way longer One of my coworkers was working on a bug that only manifested itself after ~2 weeks of putting the system under continuous stress. Bad times. Stress tests often seem to unearth strange bugs. Once when we used to sell boxes running a single-threaded real-time OS, the OS vendor gave us a separate machine that would sample the registers and memory of the box 10000 times a second (using jumper cables on the motherboard), in the hope that we would catch the exact state of the system as it crashed after days of stress testing. The sampler machine was a crazy overpowered beast of a machine with really high clock CPUs and oodles of RAM so it could dump as many samples as possible into a ring buffer in memory.

lloeki 5y ago

>> there’s no output at all > Hangs!! I hate hangs. I hate them. Crashes give information. Hangs could be so many things, but it is the absence of information. variants that made me scream include: - misleading information, the kind you can't dismiss as irrelevant, and you go down a thousand rabbit holes that seemingly lead nowhere, but it somehow all ties together into understanding the issue, only after a thousand turns and while keeping the holistic view of it all - useless information. kind of like the one above but every piece of information you get truly leads nowhere Once I had an actual crash that somehow happens before jumping into the program's code (like, before main), but changing your code does toggle the trigger... but when it crashes you have no core and can't attach a debugger, so you're left with debugging on the happy path and constantly think about what could possibly go wrong, and you don't even know what you're looking for. understanding why crt0/ld trips over is not fun.

mateo411 5y ago

I also hate hangs. If you are working with the JVM you can send it a signal kill -3 [pid] which can give you a thread dump. That's a QUIT signal if you are curious. I once found a stuck HTTP request when I did this, so I fixed the issue by adding a timeout to the HTTP connection object. When you are designing a system or a program you should have a way to get diagnostic information to make it easier to fix issues that can arise.

pixl97 5y ago
Always design in timeouts. And always make them configurable.
- mateo411 5y ago
  Yes, I agree.
bennofs 5y ago
JVM also has jstack and other tools like jprofd to introspect running vms.

bravetraveler 5y ago

First day of electrical engineering classes we were taught, before troubleshooting a circuit - make sure you're plugged in. Assumptions are evil :)

theshrike79 5y ago

> #1 assumption I always make sure to check now is "is the code that's running actually from the source code I'm looking at". So may "impossible" bugs I run into are in fact impossible... in the codebase I'm looking at. And it turns out what's deployed is some other code where it is very much possible. cough I spent two days doing this when testing different application servers for a Java servlet I was consulting on. Two days of banging my head and wondering why it works on that server and not on that server. I was deploying the new .war to the wrong place and the server autodeploy was picking up the old version every time. I've been doing this for over two decades.