6 ms·
Tiered support is an anti-pattern (2013)
- latch 2y agoIn my experience, the real problem with any support scheme is that the decision makers don't have enough skin in the game. It's easy to deprioritize tech debt and bug fixing when you aren't facing 3am pages. The obvious solution to this is a distributed team. There can still be some holes (e.g. Jan 1 is, afaik, pretty universally celebrated), but having worked at companies that had a team in the US, Europe and Asia was...really nice with respect to this.
- mrweasel 2y agoThe incentives are frequently wrong as well. The support staff for many companies are rewarded on volume, call-queue length and "resolution time" (not actual resolution time, but how quickly you can get the customer off the phone). The only valuable metrics, in my mind, are "Did we actually solve the issue for the customer" and "Have we effectively prevented this from being an issue in the future".
- the_alchemist 2y agoIsn't this tiered support a necessity when operating at scale? How can you help thousands of customers where faqs and getting starteds don't cut it? You need a filtering mechanism (L1) to help users directly without overloading the product team with a standard reply rtfm.
- benoau 2y agoIt's triaging the support so the people who are best-capable of answering specific things don't waste all their time answering the stuff a new hire can take care of after one week training.
- andrewaylett 2y agoTwo layers is fine, so long as they're allowed to talk to each other. Adding a third causes a lot more problems. Once you get beyond having a single dev team, a layer of support is necessary: you can't expect consumers to know which of your dev teams they ought to be talking to. My employer has an amazing support team, who are effective champions for the user when interacting with development teams. They're fantastic.
- jasonjayr 2y agoThat's an important requirement of an effective support team: they have to have a seat at the project management's table. They have to have the power to request features + changes from developers, at the same priority level as the rest of the product development team. IMHO it's important both for the support team morale, as well as ensuring a positive UX.
- andrewaylett 2y agoAt a higher priority level, IMO. That said, most of the issues I see from support are straightforwardly bugs, even if they're not normally easy ones to fix. But when they have information they think is worth sharing, it's always been worth my time to listen.
- HeatrayEnjoyer 2y agoIf you're operating at scale you hire at scale. How do you suppose Target staffs their "at scale" brick and mortar stores?
- ahtihn 2y agoYes, you hire L1 support staff at scale. You don't hire product developers to handle the scale.
- HeatrayEnjoyer 2y agoNo one said to hire developers. Even L1 phone support would be a great improvement
- magicalhippo 2y agoWe make B2B software, and we have roughly the setup described in the article, where L3 are the devs. I don't recognize what the article describes at all. There are tons of questions answered by L1 which are not even issues with our software. And us devs certainly feel responsible for what we produce, and we still talk to our customers when needed. So I think it's more a culture and possibly management problem than simply layering.
- jonathanlydall 2y agoI don’t necessarily think tiered support is the wrong way to go. But as someone developing a product, sometimes when you’re closer to your users you find that many users get told to RTFM on a particular feature and maybe the actual problem is that the feature is unintuitive.
- magicalhippo 2y agoBut I get feedback from L1/L2 when some feature generates a lot of support requests, and I've many times sat down with support and even called up customers to improve such features. That said, I agree with the sentiment that as devs one shouldn't be far removed from customers. For example I do take time every now and then just looking at support working if they've got a customer showing an issue or similar. Or just hang around to hear what they complain about. This is of course much more difficult if support is in another building and not just down the hall.
- HL33tibCe7 2y agoThere are ways to deal with that without putting devs on the front line though.
- mrweasel 2y agoYears ago I worked for a telco. As part of trying to identify recurring problems and design possible solutions, we would sit in on support calls for a few hours a day, for a week. While we did find a few things that could be improved by adding more software, or reworking solutions, that wasn't what was needed. 90 - 95% of support calls where related to technical issues, which required first level support to create a ticket for technical support, or escalate to them immediately. First level support was pretty much not needed. It would make more sense to reverse the call flow and have technical staff pick up support calls, then transfer to first level support if the issue turned out to be non-technical, which it would be in 5% of the cases. To this day that is still my issue when call ISP or telco support. Why am I trying to have some 19 year old, with access to nothing but my billing information, trying to solve the issue of my router not connecting? The type of first level support most technical companies offers could be solved by better monitoring, better self-service and well written documentation.
- AStonesThrow 2y agoTIL that ".me.uk" is the domain reserved for "personal names" https://en.wikipedia.org/wiki/.uk#Active https://en.wikipedia.org/wiki/.uk#Active Which is pretty cool, considering that when I registered a domain under ".us" in 1993, I used our street address as the leaf node, and I've seen a few random "StonesFamily.org" or similar; it's nice for UK persons to stake a claim here. How long has it been active? How do they resolve disputes? Do they permit subdomains for family groups?
- traceroute66 2y ago> How long has it been active? I think pretty much forever at this point, the only addition to .uk in recent memory is the ability to register .uk TLD itself (before it was all second-level only). I think it might have been around the 2000's when they introduced me.uk. > How do they resolve disputes? AFAIK me.uk falls back to the general rule of "first-come first-serve". The only real requirement for me.uk is that it has to be registered in the name of a natural person and not an agent, trustee, proxy or representative. Any transfer or renewal has to continue to adhere to this rule. There is no UK nationality or residency requirement for me.uk either AFAIK. Disputes fall back to Nominet's standard Dispute Resolution Service. > Do they permit subdomains for family groups? Not AFAIK. Well, of course you can register the base me.uk and DIY DNS subdomains.
- echoangle 2y agoI don’t even get the point, are Facebook developers supposed to deal with 500k support emails per day because grandma forgot her login? That’s unrealistic for any product with more than maybe 1k users.
- andyp-kw 2y agoWe've come to expect zero customer support from the likes of Facebook and Google, but why does that need to be the norm ?
- echoangle 2y agoI didn’t say that you should have zero customer support, but I don’t think the level system is the problem. It’s a problem when the levels don’t properly escalate stuff they don’t understand. But you just can’t have every support case go directly to the developers because 90% of support cases are stuff developers can’t help you with. It’s like saying every person in a hospital should directly go to the operating theater. The 10% people needing urgent care would like it but it’s a very inefficient way to use the time of the staff. That’s why you have a reception (Level 1) that’s triaging cases and checking if you even need a doctor.
- Vampiero 2y agoThe exception being that the triage is a queue-based system, and eventually it actually handles your case: you show up at the hospital, describe your problem to a human being, and you're assigned a priority in the queue pertaining to the specialist that needs to see you. If you're not dying you might have to wait a few hours, but you can rest assured that you will be looked at by an expert -- even if you're a hypochondriac. If your case is an emergency, you can cut through all the queues and go straight to the operating table. Whereas most L1 support systems are either non-existent; a FAQ loop that gets you nowhere; or a Dialogflow/GPT chatbot with 0 reasoning skills, no technical knowledge, and limited domain knowledge (really just a more convoluted version of the FAQ loop). If you're lucky, after some digging you might find a hidden link that lets you talk with a real human being. These systems are deliberately set up to massively increase the effort required to get to talk to a technician, as those are a limited resource and companies don't want to spend money on stuff that would actually improve their internal processes. And if hospitals worked like this then the majority of people showing up for any problem whatsoever would die before receiving care. This is not how triaging works.
- smallerfish 2y agoSupport forums for free tiers are the absolute worst. With some exceptions, the general pattern seems to be power users hanging out and answering questions on the company's behalf. Who are these people? Another support anti-pattern is an untrained offshore support team whose entire mission seems to be to close the case and move on. There is little in life more enraging than a support rep who refuses to engage logical brain and escalate based on clear evidence of a technology issue. At my most successful startup, we had an extremely smart guy in the support seat; he was eager to learn from the dev team how to triage, and so we'd get bug reports complete with network tab screenshots, api paths which had caused an error, etc. He could easily have been a skilled programmer had he chosen to pursue it. When we were acquired, he was merged into a larger team that had much less autonomy, and his morale tanked over time. The zendesk ("here are articles that might be appropriate") approach sucks too, as do chatbots. Perhaps fear of hallucinations is holding people back, but it seems like an LLM based product could take over the "keep it cheap" support vertical. 1) Have people file a report in their own words; 2) synthesize an answer for them, but only if the LLM identifies with high confidence their report as being a known problem that has an answer; 3) cluster reports from users based on salient features and escalate new clusters to the tech team; 4) and if the company can afford it, route everything else to a human team who can categorize, click a few buttons, and send the LLM back with an answer, or escalate.
- grues-dinner 2y ago> power users hanging out and answering questions on the company's behalf. Who are these people? It's genuinely strange. People will even do it for banks and phone network companies. As a company, all you need to do is have XP tiers for posting and someone will be willing to dedicate their life to becoming a Finco Bank Guru and running the forum for free. Usually they have a rather "Boomer" energy, so I get the feeling it's retired people who think the parish council/HOA doesn't meet often enough.
- thih9 2y agoOn one hand it’s unfortunate that people don’t get paid for this kind of help and perhaps act in a short sighted way (giving away too much control to the provider). On the other, a lot of these are people willing to help someone else for free and make their knowledge practically public in the process. Seems aligned with a hacker/wikipedia/oss ethos in many ways.
- makeitdouble 2y agoThe need for devs to have a sense of what's happening on the user side, get support requests and work with the customer support is spot on. It really makes a world of difference. The crucial issue though is to also have competent CS working with the dev team. It's not easy to have two separate teams share info and collaborate efficiently, it requires skills, and these skills need to be on both sides of the table. That makes it harder to retrofit these setups in an existing org when these aspect where completely ignored at hiring time for instance.
- crabbone 2y agoThe article is from the time when CI / CD was relatively new and many felt like it needs promotion / wanted to jump on the bandwagon. Today, after some practical experience, it looks like some conclusions this article makes are unwarranted. For instance, the idea that the development team should be directly involved in addressing customer issues, if implemented, often results in development plans being ruined, QA doing a lot of busy work, release schedule going down the drain. Ultimately, the problem is the lack of expertise necessary to triage and to schedule development necessary to address the problems the customer is facing. Low-tier customer support is usually a low-paid position that doesn't require expertise in the product being supported, or anything at all really. Is there a solution that doesn't involve rising the requirements for the support staff (and rising the pay)? -- I don't know. Doesn't look like it. It seems like asking for having your cake and eating it too.
- mrweasel 2y agoMany of the tiered support offerings from companies also doesn't make sense from a customer perspective. My favorite example is Elatic Co. trying to sell us support. We want one thing, and one thing only: "When this fails, in a way where we can't fix it using documentation, experience or the support forums", we need to be able to call someone. Everything you're offering below that is useless to us. We just need the phone number in the exceedingly rare case that this break down completely, everything else is not something we'd be willing to pay for. But the "call someone, who knows something" is the final tier, and includes all the things below it and is priced accordingly. Understandably being able to call a developer should be expensive, but when you tier it, then customers feel ripped of, because they have to buy something they don't need, to get that one thing they really do need and are willing to pay for. In our case we where willing to pay some fixed price per year for access to phone support, and then an hourly rate if we called, but that wasn't the pricing structure. In fairness the support structure and pricing from Elastic has improved since then, but the issue is the same, if you only need one thing from the higher tiers, then the pricing doesn't make much sense to you as a customer.
- portaouflop 2y agoWould you prefer they only offer the call option without all the other stuff for the same price? Then you don’t have the problem of paying for things you think you don’t need. Your problem is that you want the highest value (direct on-call team with actual skills is expensive AF) for the budget price - as someone who helped design support tiers this is super common but obviously not economically viable. edit: in any case on that price level the price structure should be more flexible I agree
- mrweasel 2y agoThis is not exactly how it works, but as a custom you look at the price of the tier below what you want, and the price of the tier which contains what you want. The price difference must be what the feature you want costs. So if the tier with which provides on-call is $100.000 and the tier below that is $75.000, then the cost appears as $25.000 to me as a customer. That not true, when you dig into it, some of the cost is covered by "unused" benefits of the tiers below, so that one benefit on it's own might be closer to $50.000. If you truly don't need the stuff below, then you'd want more flexibility in the price structure and be able to spend the $50.000, rather than the $100.000. Instead the provider now ends up with no sale. > direct on-call team with actual skills is expensive AF More than most realise. My issue is that there are occasions where you don't mind paying for that service, but you're not allowed to, because the tiers are designed so that it becomes more expensive than it should be. I've seen a number of support contracts, where they only tier that actual provides much value is the one where you can call an expert, but that's hidden behind "Call Us" or "Super Premium Platinum" priced such that only governments and VC funded companies can/will pay for it.
- veggieroll 2y agoAs a developer, doing one or two support cases per day has been one of the best things I have done. It really helps me understand what they’re doing better. It short circuits a lot of bureaucracy. And it’s great when you can push out a change for someone an hour after they wanted it for simple things. So they know that you can make changes fast to address their problems. And it’s just nice to get to know the users and be friendly. We get to chatting. And it just makes everyone more chill knowing that the developers care.
- deleted 2y ago[deleted]
- thom 2y agoSmall world: my first job out of university was working for Pete at the first company described here. It definitely was the kind of place you could add "_old.asp" to any URL and probably hit something. Ultimately I'm not sure there was much value we could have salvaged in our position however we handled support or our wider roadmap: this was a regional news group who fundamentally rejected the idea that the internet was going to destroy their classifieds business, and later couldn't cover the liabilities of its enormous gold-plated pension fund. Every business must decide how to optimise flows of value, but it's just as bad doing a good job at the wrong ones, as it is doing a bad job at the right ones.
- BobbyTables2 2y agoIt’s okay, tired support has been replaced with zero support in the form of community-only forums…
- Havoc 2y agoWithout some sort of tiering the devs would eventually get swamped with how do I turn on my computer level stuff as the biz grows Thinking about it in degrees of separation is fine from an abstract view but collides with reality
- patrakov 2y agoThere is another pitfall in the layered support structure: how would you hire, train, or otherwise produce L2 support engineers? You cannot hire someone directly as L2, as there is a learning curve for company- and product-specific issues, and a freshly hired "L2" would have to learn. However, they are shielded from the most frequent issues by a hoard of L1s and thus cannot learn about the typical problem areas or troubleshooting approaches. So they have to either graduate from L1s or accept a downgrade from being a developer.
- Galanwe 2y agoThere are usually a number of people in L1 roles looking to transition to more development. You can have them be L2 by having 1/3 of their time spent contributing to bug fixes and small features of the software so they get to know it better. I dont think they would have to come from L1 from the same company / project though. Experience does allow one to learn faster, communicate more efficiently, etc. I think it's also good practice to have devs rotates on some small shifts of L1/2 so they don't get out of touch with the reality of maintaining their stack.
- magicalhippo 2y ago> You cannot hire someone directly as L2 We've managed to do that by recruiting from our customers. During support cases and such we'll notice someone is more technical, and then at some point approach them with an offer. This way they already know our software well and can get fairly straight into a L2 position. Of course in the start they'll get some more L1 stuff and such, but it significantly reduces the learning curve. Customers aren't too miffed as the alternative would almost always be that they'd go to a competitor.
- cwbriscoe 2y agoBusiness Analysts can fit the L2 role. Where I work, they are the interface between the end users and the developers. They usually come from the either the business side or the support center. And if they came from the support center, they usually have prior experience as an end user. However, I can see how that wouldn't work out for all industries.
- 2y ago