3 ms·

Deployment is where the really scary bugs can happen the easiest. I've been working on a warehouse management software (that was running on mobile barcode scan

by ajuc 8y ago

Deployment is where the really scary bugs can happen the easiest. I've been working on a warehouse management software (that was running on mobile barcode scanners each warehouse worker had, as he moved stuff around the warehouse and confirmed each step with the system by reading barcodes on shelves and products). We had a test mode, running on a test database, and production mode, running on the production database, and you could switch between them in a menu during the startup. During testing/training users were running on the test database, then we intended to switch the devices to production mode permanently, so that the startup menu wouldn't show. A few devices weren't switched for some reason (I suspect they were lost when we did the switch and found later), and on these devices the startup menu remained active. Users were randomly taking devices each day in the morning, and most of them knew to choose "production" when the menu was showing. Some didn't, and were choosing the first option instead. We started getting some small inaccuracies on the production database. People were directed by the system to take 100 units of X from the shelf Y, but there was only 90 units there. We looked at the logs on the (production) database, and on the application server, but everything looked fine. We were suspecting someone might just be stealing, but later we found examples where there was more stuff in reality on some shelves than in the system. At that time we introduced a big change to pathfinding, and we thought the system was directing users to put products in the wrong places. Mostly we were trying to confirm that this was the cause of the bugs. Finally we found the reason by deploying a change to the thin client software running on the mobile devices to gather log files from all the mobile devices and send to server.

hinkley 8y ago
I bet you had one engineer who claimed that the real problem was that the users were stupid and not that the deployment process was error prone. I've heard about this case many times before but somehow in the other renditions they downplayed or neglected to mention that the deployments were manual. As this story was first explained to me, one of the servers was not getting updated code, but I was convinced by the wording that it was a configuration problem with the deployment logic. Performing the same action X times doesn't scale. Somewhere north of 5 you start losing count. Especially if something happens and you restart the process (did I do #4 again? or am I recalling when I did it an hour ago?)
- sucrose 8y ago
  Was the deployment process of your parent post actually error-prone? From what I gathered, the developers were unaware of the lost handheld scanners. I imagine if they did they could've proactively put them out-of-service until found.
  - ajuc 8y ago
    We had automatic updates in the thin clients (that's how we were able to add "logging to server" on all of them at once). The problem was - the startup menu with testing/production choice was enabled independently of the autoupdate mechanism (separate configuration file ignored by autoupdates) for some technical reason (I think to allow a few people to test new processes while most of the warehouse works on the old version on production database).

by ajuc 8y ago

hinkley 8y ago

I bet you had one engineer who claimed that the real problem was that the users were stupid and not that the deployment process was error prone. I've heard about this case many times before but somehow in the other renditions they downplayed or neglected to mention that the deployments were manual. As this story was first explained to me, one of the servers was not getting updated code, but I was convinced by the wording that it was a configuration problem with the deployment logic. Performing the same action X times doesn't scale. Somewhere north of 5 you start losing count. Especially if something happens and you restart the process (did I do #4 again? or am I recalling when I did it an hour ago?)

sucrose 8y ago
Was the deployment process of your parent post actually error-prone? From what I gathered, the developers were unaware of the lost handheld scanners. I imagine if they did they could've proactively put them out-of-service until found.
- ajuc 8y ago
  We had automatic updates in the thin clients (that's how we were able to add "logging to server" on all of them at once). The problem was - the startup menu with testing/production choice was enabled independently of the autoupdate mechanism (separate configuration file ignored by autoupdates) for some technical reason (I think to allow a few people to test new processes while most of the warehouse works on the old version on production database).