9 ms·
Observability is not only for SREs
- xbar 4y agoSeeing Taffy on initial load is pretty much perfect.
- twic 4y agoFlash of Unstyled Cavy.
- dboreham 4y ago> the world of QA Wait...there are companies where there's still a world of QA?
- KaiserPro 4y agoyes, because as we know; SWEs are shit at proper testing. A good QA department will know what the software does, but more importantly how it should behave, and how its changed over time. They should be a good resource for working out how a feature should look and behave. a good QA department is worth its weight in gold
- drewcoo 4y ago> because as we know; SWEs are shit at proper testing Not true. There's nothing magical about testers. They don't have "special" skills or deep voodoo powers. The best ones are just smart people who understand people and software. > They should be a good resource for working out how a feature should look and behave You must have weak designers and PMs. Everyone should be welcome in the process, but those people drive that stuff.
- tremon 4y agoThere's nothing magical about testers Oh, but there is: they have fault-finding in their job description.
- hinkley 4y agoI have worked with a wider range of QA folk than you have, apparently. The best are magic, the worst are leaches. And I think that’s the problem with QA. Those who don’t like it cherry-pick their examples. The thing to remember is that getting rid of QA upsets a triumvirate power dynamic. If it’s the dev team versus project management, the dev team always loses. If it’s sometimes dev and QA against PM, then you can win those fights. But only if you haven’t removed the QA manager from the org chart. We’ve played ourselves in essentially a union-busting manner. If you’ve always been at loggerheads with the QA team, I can tell you that you’ve been missing out. Looking back, defusing the default animosity between QA and Dev has gotten me more promotions than anything else in my career, and the lack of a QA team has put a substantial damper on my upward mobility. It sucks and I really wish we could go back.
- dijit 4y ago> They don't have "special" skills or deep voodoo powers. Some do, seriously. I'm coloured by video games, but some of the QA people I've seen are magic, without them entire products wouldn't ship at all. There's something to be said for having "break shit and report on how you did it" as a job title, regardless of the other gains you get for having someone who holistically understands the product. Some are even lazy, so work to automate themselves out of working, which makes their impact much wider, even when it's domain specific. It's also the case that the people coming out of QA in gamedev and going into other parts of the company tend to do extremely well compared to people going direct into that area. Our old managing director started in QA, as did the executive producer.
- tkiolp4 4y ago> There's nothing magical about testers Yes, there is: they get to do testing full-time. Engineers spend their best “brain time” designing and writing systems. Testers spend their best brain time, well, testing. That can make a huge difference when it comes to find bugs.
- KaiserPro 4y ago> There's nothing magical about testers There is a spectrum, of course. But, a good QA has a "feeling" for finding where things break, an excellent QA know how to fix it. Just as writing software is a skill, so is QA'ing along with Sysadmining.
- xmodem 4y agoHaving worked in orgs both with and without a QA function, they can serve a very important counter-balancing function to an overly aggressive product org, that is determined to ship the features that'll make their growth metrics at any cost. (This is the more modern version of QA, of course, where it stands for Quality Advocate or Quality Assistance)
- pphysch 4y ago"Proper testing" is often introduced, in CS courses, as 100% Test Coverage cargo cultism. Which is really a error-prone and unproductive way of ensuring Correctness in the software beyond the guarantees offered by the programming language. Most SW orgs should not waste time on test coverage metrics. On the other hand, we can look at SW testing as "building a robotic user that will catch errors before real users". This approach is far more intellectually-engaging, productive and gets SWEs in touch with the product. That is to say, SWEs aren't shit at "testing", they are shit at product, because product is virtually omitted in traditional CS curricula.
- tkiolp4 4y agoBefore: software engineers wrote code. Now: they write code, they test it, they understand customer’s requirements, they monitor their code, they are on-call, they mentor others, they challenge the status quo, they give tech talks, sometimes they attend company hackatons… I’m tired of software engineers having to do a bit of everything.
- drewcoo 4y agoYes. https://en.wikipedia.org/wiki/Mirror_Universe https://en.wikipedia.org/wiki/Mirror_Universe But you can read "QA" as "what you'd do to investigate the system if you had time and didn't have people constantly pushing new feature tickets at you."
- KaiserPro 4y agoOne of the only professional achievements I was responsible for is getting the FT to move away from splunk logging, to generating metrics directly. I installed a graphite/grafana cluster[1], and delivered a bunch of tech talks on how to massage raw metrics into useable metrics that related to business goals. I did this because I hated waiting for splunk graphs to generate, and I could never really get any kind of actionable graphs out of it. over a couple of years it went from a couple of thousand metrics to well over a million active metrics. (anything not updated in the last 7 days was deleted) but the biggest "oh its actually useful" was seeing product owners, scrum leaders and buisness analysts create, update use dashboards, and setting alerts on service health _without_ software engineering help. It was brilliant, and a testament to how simple graphite/grafana is to use. [1]well, I installed a graphite cluster, someone twisted my arm to get grafana installed.
- snidane 4y agoI am not very educated in this area, but I wonder why move away from the concept of metrics extracted from logs to discarding logs and storing just the metrics? Don't metrics have to be defined first? I assumed you'd first do a big data log analysis to understand what is going on and then monitor for metrics learned to be useful. Storing just metrics means it'll be hard to investigate when things go wrong when the currently defined metrics don't capture that new issue. In big data circles the consensus these days is to store all raw data since storage is cheap. Useful cleaned aggregations and metrics can be extracted later by processing the full history using the now available big data tools. I suspect this approach is a problem in the traditional software engineering, where focus is not on historical data like it is in data engineering. The solution goes - Splunk is too expensive, let's ditch logs altogether and hope we capture some metrics and plot them in nice dashboards. Instead of dumping the logs highly compressed into cheap s3 and running some Snowflake or Spark on it later. Heck, even dumping the intra-day non historical logs into Kafka or Materialize and extract the same metrics on the fly while preserving all raw data.
- kevindong 4y agoGraphing metrics and doing transformations/comparisons is (much) faster. Yes, metrics do have to be defined first but in my experience that's a non-issue since the things you want to monitor are usually immediately obvious during development (e.g. request-response times, errors returned, new customers acquired, pages loaded, etc.). With that being said, it's not a mutually exclusive situation. You can have both. However, some logs used for plotting metrics have near-zero debugging value (e.g. a log line that just includes the timestamp and the message "event x occurred"). Those kinds of logs should be fully converted over to metrics. Some other logs however are genuinely useful (e.g. an exception occurred, the error count should be incremented, and this is the stack trace).
- hinkley 4y agoFew things are truer in software than this: If you expect people to use a tool for your benefit, you’re going to be disappointed and/or bamboozled every single time. Managers are experts in this. They don’t seem to see why they are always having to harp on people to keep visibility tools up to date because “not getting harped on” is not a motivation to do something. Most of my job is keeping track of my responsibilities. If I’m keeping track of yours, then why are you even here? People use tools for their own benefit (if even then). If that helps out someone else, that’s a bonus feature, not a core one. So we use testing frameworks if it helps us stop our own regressions. We use issue trackers to fight pre-emption and weekend amnesia. If you don’t honor that (by mandating that they be used in a different way that raises the cost/benefit ratio) then you’ll get less compliance, not more. So if the SREs want something, they better have a self-interest angle, otherwise they’ve just got a platform for lecturing, not for progress. Of course, there are a few people in any org who are satisfied with lecturing.
- robluxus 4y agoNot getting harped on is indeed a motivator for a lot of folks. And managers take advantage of this sometimes consciously sometimes unconsciously. Re: SREs and observability, unfortunately sometimes the power imbalance tips the scale in their favor because if as a service owner you let observability out of your control then guess who will report your SLA to your boss? Of course every org and every SRE org is different. Except yours. The Anna Karenina principle applies.
- hinkley 4y agoMy phrasing was poor. I agree it's a motivator, but it's a pretty poor one. Not quite bottom rung, but close enough to it.
- pm90 4y agoThis is an excellent overall comment. One thing: As a sibling has pointed out I do enjoy not getting harped on so usually its either a small enough task so ill just do it or it’s difficult enough that I give reasons why I can’t do it (or need some dedicated time to get it done). Ive had managers who have understood this dynamic and others who have ignored and try to use force by continuing to harp after Ive provided reasons. Generally I haven’t lasted that long under the latter.
- timcavel 4y ago