3 ms·
I work at a bank. Currently I am working on a project along with few tech diversified colleagues(mainframe & non-mainframe) to prepare a white paper on SRE. I a
by bobbydreamer 6y ago
I work at a bank. Currently I am working on a project along with few tech diversified colleagues(mainframe & non-mainframe) to prepare a white paper on SRE. I am Db2 DBA mainframes, for me it's normal to support project while implementing change in production and monitor till the first successful run as I worked on it in the development phase and have a good handover to the production dbas and got answers for all their questions. For me what they had said about production monitoring, daily production meetings, oncall support were all normal. In the team already in the name of PI planning we are all working on automating or changing process, updating documentations activities are being carried out.
Scenario is not like developer throwing over the wall to operations. First there is this architect team deciding things then it comes to developers then for structure changes to dbas at this point almost 80% of things are decided. If any perf issues then only logics and things will be changed then comes handovers, peer checks, approval and then to production. So there are lots of walls, outage is a big thing, implementation is a big thing when there is a structure change to a table that's handling 3000+ transactions per second. There are so many things to look out for. So it's not like I can implement anything to production in few minutes or couple of days unless it's a fix.
On the concepts of error budgeting, yeah it's nice. Since error budget is depleted so let's stop feature development. From banking perspective what if there is a regulatory change, well that has to go. I haven't seen any examples that companies saying that they stopped features which has potentials to bring revenue but stopped because we have depleted our error budgets. For example, let's say, a product making company saying we didn't release next version because we depleted our error budget. Even google hasn't showcased anything on the subject saying we didn't release these because we depleted our error budgets thats why new releases or next versions were delayed.
I will agree with half the book on SLO, they are hard, monitoring the performance of customer journey, the request travels many tiers, technologies. That's hard. Each tier or tech has its own monitoring system but tracking customer journey that's hard e-banking or mobile to mainframe in between there are so many things. That's something to think about. Would like to know how big complex organization's handle this as that's the only way one can know if something is broke, it's better to get notified from monitoring team/system than social media or news.
Tooling and automations need to be centralized there would be many duplications.
Currently we have people who are THE EXPERTS of their tech but not aware of other techs. This needs to be improved. But SRE got to be sort of jack of all trades also be expert.
Postmortems are locked up somewhere it's there but that's confidential. There is a difference between a search engine breaking down and financial institution technically/due to bug breaking down. Smaller things can be show cased as there is always an RCA for an incident.
It talks about test automation but at the moment there is growing trend on more test analysts requirements. If every organization is going for test automation, why is this trend.
Overall in adoption, there will be a hybrid. Centralized team is needed for gathering information about all platforms and build centralized monitoring application or team. There need to be a catalog of automations like git repo organization level where anyone can read readme and clone or fork and use a program as it fits their requirement. It would be good know pool of experts and ask for recommendations as currently we know only when we work with them.
Automation which was not a priority can be made priority.
Book has lots of good ideas. It's a slow process to adopt, but its completely useless just to change job roles like there are few companies with too many vice presidents literally.
Hope is not a strategy