4 ms·
I'm a developer who's dealt with heavy operational loads for years. Some random things that help: - Metric-driven design -- Define your metrics and success c
by cle 10y ago
I'm a developer who's dealt with heavy operational loads for years. Some random things that help:
- Metric-driven design
-- Define your metrics and success criteria up front, as graphs on a dashboard, and build your system so that it's clear from the dashboard that the system is meeting its success criteria
- Testing
-- Unit test your business logic
-- For every logic bug, write a test which fails, fix the bug, and verify that the test succeeds (regression test)
-- Integration test all your APIs and the "success criteria"
--- try to keep the implementation-agnostic
-- Load test for latency-sensitive components
--- The results of load tests can be published as metrics in your metrics dashboard, since latency is often a success criteria
-- Continuous Delivery pipeline
--- Not only does this improve productivity and stability, but it is super useful for emergent deployments, so that you can know quickly if your fix is breaking anything, and you can add regression tests quickly
- Logging
-- Be liberal with logging
-- Log internal state changes (cache evictions, refreshes, connections opening/closing, etc.)
-- Add lots of logging in complicated business logic (there will be bugs, and logs will make them easy to find)
- Liberally sprinkle assertions throughout your code (especially if data quality is more important than resilience)
- Be conservative with dependencies
-- Think carefully before using that new sexy library
--- Do the authors make backwards-compatible changes?
--- Is it well-maintained?
--- Is it likely to be abandoned?
-- Think carefully before coupling systems together with code dependencies
- solipsism 10y agoI'm with you but I'd add: -set up automatic notification of failure conditions in those metrics. People shouldn't have to stare at and interpret graphs to know something is wrong.
- spronkey 10y ago-- and make sure, if those notifications are emails, that they are not triggered every single time the error occurs, potentially flooding you with thousands of emails.
- foobarian 10y agoWhat do people use for this kind of thing? I'm aware of Bosun (https://bosun.org/ https://bosun.org/) and Prometheus (https://prometheus.io/ https://prometheus.io/). Both can alert based on aggregated metrics, using rich rules such as values moving away from historical averages by a certain threshold.
- SyneRyder 10y agoA quick plug for Pushover, which has completely changed how I deal with alerts like that. I used to have them sent to me via email, but now the critical / urgent stuff actually buzzes my phone & their API is really easy to use: https://pushover.net https://pushover.net
- marcosdumay 10y ago> Liberally sprinkle assertions throughout your code (especially if data quality is more important than resilience) If you want resilience, make an upper layer that will recover your service on failure, deploy it on redundant hardware, add strict alerts for when recovery is impossible; and still sprinkle assertions through your code. Data quality is never secondary.
- bluejekyll 10y agoBefore any of this: Make sure you can build, deploy, and rollback your software quickly and easily. I find all too often people leave this to the end, when really it should be the first thing you do. Then iterate and add all the stuff mentioned above.