This reminds me of putting the LimeWire stub installer into production the first day of the COO/CTO's vacation. We were the most popular Java desktop app at th

This reminds me of putting the LimeWire stub installer into production the first day of the COO/CTO's vacation. We were the most popular Java desktop app at the time, circa 2005, P2P file sharing. At the time, we had approximately 60 million client GUIDs connecting to the network per month (or was that quarterly?, in any case, tons of users). It was a small company, 5 developers, 13 employees total. One of the largest costs was bandwidth for the free version of the software. We already had a mechanism for using the P2P network to save bandwidth on version upgrades of the free version. The hello handshake between peers included the serial number for a signed (1024-bit DSA) XML message containing the SHA-1 hash of the latest free installer. If the serial numbers didn't match, the out-of-date peer would download the latest XML message from the other. Within 60 seconds of a new release coming out, over 99% of the online nodes would have the SHA-1 hash of the latest free installer and start downloading the latest free version installer from each other. Edit: Originally, the XML update messages just had a URL for the installer. We accidentally DDoS'd ourselves the first time we used it, with millions of users flooding in within minutes of the latest release, so we added a randomized delay over 24 hours for actually notifying the user of a new version. After we added the P2P download of the upgraded installer, we still redirected the user to our website for the upsell opportunity. I don't recall how we got the already-downloaded installer to launch after visiting our website. So, one of the other developers and I wrote a proof-of-concept minimal C++ installer. Its only GUI was a progress bar. It had just the DSA verification code from Crypto++, a minimal parser for just the subset of XML we were using, and enough of the P2P protocol to bootstrap into the network and download the latest free installer. It would check the SHA-1 of the downloaded installer and then launch the downloaded installer. It was less than 1/100th the size of the actual application installer, and had the advantage that it would always install the absolute latest release version of the free version of the application. Also, the monthly profit sharing payment was roughly the same as my salary. For more senior developers, it exceeded their salary, so we were all highly incentivized to cut costs. The COO/CTO shot down our proof of concept with just vague worries that we might have broken something, that maybe it only worked from the office and our apartments. He had no concrete criticisms. So, the next time the COO/CTO went on vacation, we went into the webserver and replaced the full sized free version installer with the tiny stub installer. The first time the free version started up after install and successfully bootstrapped onto the P2P network, it would make a tiny HTTP request to our webserver, so we could monitor for problems. We watched carefully the metrics for successful first-time P2P bootstraps, and they were fine. We were all very happy. About an hour later, the COO/CTO called into the office from the beach on a Monday morning. He said we broke something. We asked what we broke, and he said he wasn't sure, but he was checking total badwidth usage, and it dropped off a cliff about an hour earlier. We told him to calm down, than free version successful connection rates were unchanged... but that stub installer he had nixed was saving bandwidth. He wasn't happy, but agreed to let us to continue trialing the stub installer while he was on vacation. We were the only two C++ developers at LW, and we both left for Google within a year. The stub installer stayed in production for a year or so after we left, but eventually they got rid of it due to maintenance concerns. Edit 2: "The Bulgarian Nuclear Option": I noticed a flaw in that we used a Java long for the XML message serial number, not an arbitrary precision integer. When the recording industry started threatening a lawsuit and the owner asked us to start researching options for a censored version of LW, one of the developers claimed he took my observed flaw and signed a message with serial number Long.MAX_VALUE to lock everything at the current version. He claimed he had this final version message on his home machine and had a chatbot mechanism he could activate from his cell phone to be able to lock out the owner from being able to auto-update people to a censored version. The developer in question was from Bulgaria, and hence, this was "The Bulgarian Nuclear Option". News got back to the owner, and I heard that the developer in question came back white as a sheet from being called into the owner's office. The Bulgarian wasn't the other C++ dev.