3 ms·

I am right now on the train home from work quite late (22:40 local time). The reason I spent so long at work today is because we update after 8 o'clock. However

I am right now on the train home from work quite late (22:40 local time). The reason I spent so long at work today is because we update after 8 o'clock. However, we found a critical bug in some open software that we use that we do not know why it triggers. We barely even know how it happens! Of course we haven't been able to solve it yet... After personally spending a lot of time with trying to fix the software I can say that I have gotten a much deeper appreciation of tools that are widely used and with an active community, where most bugs have already been solved. Previously I had believed if you had a tool that solved the exact problems you have, then you should use it. I hope I'll resist such an urge in the future by having a little pragmatic angel on my shoulder reminding me of the danger of underused software. No software is perfect and even tough you might be able to fix software when it is open source, you paint yourself into a corner when you will necessarily have to fix it. I felt my experience was matching some of the authors woes and I urge anyone that chooses technologies to think one step ahead and learn from our mistakes!

quickthrower2 8y ago
To avoid burning out I suggest: 1. Deploy 2. Test deployment 3. If success, great go home at 8:10pm. 4. If fail, rollback, retest, go home at 8:20pm. Come in next day and fix in source code and schedule new deployment. 5. To avoid #4 in 99% of scenarios, use a staging environment.
- opportune 8y ago
  In my experience staging environments only work in a few (very common) use cases. When I do big data work I certainly use a test environment but we can't just replicate all the data in two clusters because that would be absurdly expensive - and this makes testing performance basically only possible in prod. I'm sure basically any database system with a non-trivial amount of data has the exact same issue. But staging definitely works for something like a web server
- catears 8y ago
  Luckily we have "update nights" at most once every other month. But the workflow you describe is what I would want in the end. I work in a small startup tough so change is happening rapidly to make these processes better. It's far from institutionalized! We do however have a staging environment already that is frequently updated and updates available within an hour from starting the update. Thanks for the concern and tips nonetheless!

quickthrower2 8y ago

To avoid burning out I suggest: 1. Deploy 2. Test deployment 3. If success, great go home at 8:10pm. 4. If fail, rollback, retest, go home at 8:20pm. Come in next day and fix in source code and schedule new deployment. 5. To avoid #4 in 99% of scenarios, use a staging environment.

opportune 8y ago
In my experience staging environments only work in a few (very common) use cases. When I do big data work I certainly use a test environment but we can't just replicate all the data in two clusters because that would be absurdly expensive - and this makes testing performance basically only possible in prod. I'm sure basically any database system with a non-trivial amount of data has the exact same issue. But staging definitely works for something like a web server
catears 8y ago
Luckily we have "update nights" at most once every other month. But the workflow you describe is what I would want in the end. I work in a small startup tough so change is happening rapidly to make these processes better. It's far from institutionalized! We do however have a staging environment already that is frequently updated and updates available within an hour from starting the update. Thanks for the concern and tips nonetheless!