4 ms·

Everyone has a test environment. Some places also have a production environment.

LinuxBender 4y ago
Everyone has a test environment. Yup, the weird bugs are rarely ever found in the test/dev/staging/qa environments. Sometimes it takes longer to even figure out how to reproduce some odd bug a customer found in Prod. As one example we had one blade server that was causing financial data corruption. The senior java developers and architects debugged java down to the CPU and found 1 CPU core that was making mathematical errors under specific conditions. This would never in a million years have been found in dev, test, staging, QA, pre-prod environments. I was given the instruction by leadership to "drive over the server" in the parking lot as neither upstream OEM's were interesting in debugging it further. Another fun one I ran into was using VLAN tagging on CentOS 5 on specific hardware with a specific NIC of whom shall not be named could get into a situation where partial frames were being transmitted in a loop from the ring buffer at the highest physical packet rate of the NIC which is surprisingly very fast. It brought down the access switches and the very big distribution routers. We debugged it pretty far down but hit a wall when one of the three vendors involved would not sign the 3-way NDA. The NIC was not Intel. Or here is another one. We had a multi-million dollar storage solution that was never needed and in production it caused an IO fencing issue that broke the production databases in a glorious way. That bug would never have been tickled in dev or staging. It required a specific series of failures in the storage that triggered a logic failure on their controller of raid controllers. The vendor should have caught this one. I could probably write a book about all the weird bugs that "should never happen" but only happen under real-world Production use cases.
- toast0 4y ago
  > specific NIC of whom shall not be named Inquiring minds want to know. CentOS 5 was a while ago, so maybe it's old enough you can talk about it? > The NIC was not Intel. I found intel's reputation for stable, reliable NICs to be well deserved. (I've heard their 2.5G nics are an exception, at least early ones) But I run into one where one if the rxqueues didn't work. That's fun to debug, but dropping traffic is way better than writing line rate garbage.
  - LinuxBender 4y ago
    Inquiring minds want to know. CentOS 5 was a while ago, so maybe it's old enough you can talk about it? I would like to but if I go any deeper into that one a certain company will be embarrassed and will respond very poorly. When the day comes that they and their legal entities cease to exist I will share more of the ugly details. I found intel's reputation for stable, reliable NICs to be well deserved. Same here, at least for their server NIC's. I've found that some of their consumer NIC's tend to lean conservative on power management by default and will start having issues negotiating speeds on very long ethernet cable runs to which I have to work around by changing ACPI settings, BIOS settings and sometimes even through a L2 switch at one end. Most of those seem to use the igb driver.
    - toast0 4y ago
      Sorry for this. I guess you can only speak in broad strokes, you can't give us any real details or put anything out into the sunshine. Although you did give us a bit of intel. I guess you'll keep your NDA in force. Not sure if I missed any vendors there. The nvidia one is a reach ;)
- dang 4y ago
  Write that book!
  - LinuxBender 4y ago
    It is on my long-term to-do list but there are some things I have to knock out first. I've never written a book but I am sure there are people here that have written threads on the best ways to go about it and the pitfalls. I should search for those.

Everyone has a test environment. Yup, the weird bugs are rarely ever found in the test/dev/staging/qa environments. Sometimes it takes longer to even figure out how to reproduce some odd bug a customer found in Prod. As one example we had one blade server that was causing financial data corruption. The senior java developers and architects debugged java down to the CPU and found 1 CPU core that was making mathematical errors under specific conditions. This would never in a million years have been found in dev, test, staging, QA, pre-prod environments. I was given the instruction by leadership to "drive over the server" in the parking lot as neither upstream OEM's were interesting in debugging it further. Another fun one I ran into was using VLAN tagging on CentOS 5 on specific hardware with a specific NIC of whom shall not be named could get into a situation where partial frames were being transmitted in a loop from the ring buffer at the highest physical packet rate of the NIC which is surprisingly very fast. It brought down the access switches and the very big distribution routers. We debugged it pretty far down but hit a wall when one of the three vendors involved would not sign the 3-way NDA. The NIC was not Intel. Or here is another one. We had a multi-million dollar storage solution that was never needed and in production it caused an IO fencing issue that broke the production databases in a glorious way. That bug would never have been tickled in dev or staging. It required a specific series of failures in the storage that triggered a logic failure on their controller of raid controllers. The vendor should have caught this one. I could probably write a book about all the weird bugs that "should never happen" but only happen under real-world Production use cases.

toast0 4y ago
> specific NIC of whom shall not be named Inquiring minds want to know. CentOS 5 was a while ago, so maybe it's old enough you can talk about it? > The NIC was not Intel. I found intel's reputation for stable, reliable NICs to be well deserved. (I've heard their 2.5G nics are an exception, at least early ones) But I run into one where one if the rxqueues didn't work. That's fun to debug, but dropping traffic is way better than writing line rate garbage.
- LinuxBender 4y ago
  Inquiring minds want to know. CentOS 5 was a while ago, so maybe it's old enough you can talk about it? I would like to but if I go any deeper into that one a certain company will be embarrassed and will respond very poorly. When the day comes that they and their legal entities cease to exist I will share more of the ugly details. I found intel's reputation for stable, reliable NICs to be well deserved. Same here, at least for their server NIC's. I've found that some of their consumer NIC's tend to lean conservative on power management by default and will start having issues negotiating speeds on very long ethernet cable runs to which I have to work around by changing ACPI settings, BIOS settings and sometimes even through a L2 switch at one end. Most of those seem to use the igb driver.
  - toast0 4y ago
    Sorry for this. I guess you can only speak in broad strokes, you can't give us any real details or put anything out into the sunshine. Although you did give us a bit of intel. I guess you'll keep your NDA in force. Not sure if I missed any vendors there. The nvidia one is a reach ;)
dang 4y ago
Write that book!
- LinuxBender 4y ago
  It is on my long-term to-do list but there are some things I have to knock out first. I've never written a book but I am sure there are people here that have written threads on the best ways to go about it and the pitfalls. I should search for those.