3 ms·

Storytime? What happened?

• rodgerd 12y ago
  Since I don't have deep pockets I shan't name the vendor, but it started out with us taking delivery of a bunch of blades where our blade supplier had changed the NIC from one vendor to another. We discovered we had repeatable problems with J2EE cluster traffic over UDP - we'd see packet loss rates go to absurd levels as we ramped up load on the cluster, leading to a situation where a node dropping out and then rejoining would cuase the cluster to lock up trying to bring the node back into the fold. We could repeat the massive packet loss using UDP test tools. Rather ugly. Coincidentally we had a visit from the CTO of high-performance storage vendor who happen to have a bunch of kernel hackers on their staff. We mentioned our problem in passing and he explained how they'd nearly lost a major contract because a deployment had moved thei customer from being storage-bound to initially untracable data loss. Digging around by their kernel hackers showed the NIC was losing data. After a certain amount of too-ing-and-fro-ing the vendor moved from denying the problem to admitting that their silicon had a defect that would throw away data under load, and their Windows drivers tried to spackle over the problem. They were relying on no-one being able to drive enough load through the card to cause a problem. This dovetailed with our experience, and we found that installing a different manufacturer's card in the blade let us work around the problem. Our blade supplier moved to a new NIC vendor subsequently.