9 ms·
I'd be so curious to get a window into the profiling, back-room discussions, etc. that happens at orgs like Slack and Teams. What's going on here? What all did
by brundolf 4y ago
I'd be so curious to get a window into the profiling, back-room discussions, etc. that happens at orgs like Slack and Teams. What's going on here? What all did Teams just improve? What are the barriers they're still running into?
I don't want to be that person who says "mrahhh a new CS graduate could have built Teams in 2002 and it would have been faster!" But I'm also like, whatever is stopping it from being faster, I can't imagine what that is from where I'm sitting!
I've built high-complexity, high-performance web apps, and while there are challenges (the smooth infinite scroll without any placeholders in the video was actually pretty slick), there are other things like 900ms to switch tabs and 9.1s to boot that are just really hard to believe (and those are the improved numbers!)
Some of it's probably back-end; Slack for sure has back-end performance problems, where the UI is snappy but then sits there and loads the conversation for several seconds (or has synchronization problems, or loads an old version of history, or...). And I can understand having some performance challenge here- these apps deal with huge amounts of data per-workspace. But also... it's very simple data. Mainly a series of messages, displayed in chronological order. That's one of the most straightforward possible things to optimize.
I'm just so fascinated. The HN commentary around this topic is usually lazy, but there is a genuine mystery here.
Edit: A light overview of the new version was posted below (https://news.ycombinator.com/item?id=35352521 https://news.ycombinator.com/item?id=35352521) but I'm linking it again here for visibility: https://techcommunity.microsoft.com/t5/microsoft-teams-blog/microsoft-teams-advantages-of-the-new-architecture/ba-p/3775704 https://techcommunity.microsoft.com/t5/microsoft-teams-blog/...
- saagarjha 4y agoMost engineers are unfamiliar with how to improve their software’s performance, unfortunately. This has nothing to do with any individual technology stack or product, really, it’s more of a general idea of how you can figure out how things are slow and make them faster. It’s just not something most people ever learn. The way performance blog posts are written on the internet doesn’t really help, either, because most of them are tired arguments about specific things being slow or one-offs that don’t apply elsewhere. Learning how to use a profiler, or have an intuitive sense for how long something “should” take, is critical here, and it takes time to develop. For example, speaking completely independently from the problem here but putting on my performance hat, I would expect that if we take all things as a given (“no major architectural changes allowed”) Teams should not take 10 seconds to start up to show UI. We know that the tech stack they’re using isn’t fundamentally that bad. This suggests that there is probably a handful of things they are doing that are low-hanging fruit but haven’t been identified yet. Once they find those and bring down the time to something more reasonable like a second things will get much harder for them to fix.
- Jtsummers 4y ago> Most engineers are unfamiliar with how to improve their software’s performance, unfortunately. Then they aren't "engineers" yet. Engineers can figure out how to use a profiler and how to improve performance of their software. Calling themselves "engineers" while being unable to do so should be a signal to everyone in their circle that they're incompetent hacks. They should at least be honest and give themselves an appropriate title, like "apprentice" or "intern" or "novice" or "student".
- saagarjha 4y agoHeinlein has a quote in one of his works that goes like this: > A human being should be able to change a diaper, plan an invasion, butcher a hog, conn a ship, design a building, write a sonnet, balance accounts, build a wall, set a bone, comfort the dying, take orders, give orders, cooperate, act alone, solve equations, analyse a new problem, pitch manure, program a computer, cook a tasty meal, fight efficiently, die gallantly. I think all software engineers should be taught to be able to handle a profiler, poke around in a debugger, write a short correctness proof, and consider how an attacker might abuse the APIs they expose. I know a lot of people who are working to make this dream a reality right now. Until this happens, though, gatekeeping the definition of an engineer is inappropriate and does nothing to help improve the situation, because people are generally resistant to learning these things if you call them "incompetent hacks".
- Jtsummers 4y agosoftware profiler : software engineer :: ruler : <pretty much every other> engineer Profilers are simple tools that every software engineer should be familiar with. They are literally just a way to measure performance of a software system. The basic ones do this by tracking time, or counting the number of times a block of code are entered. People have recreated them a million times (probably more) because they're so damned easy to create. I'll take my hit for this one: If someone were to come up to me and say, "I'm an engineer, but I don't know how to measure the software I make." I'd think they were a hack, yes. If they instead were honest and said, "I'm a student studying to become an engineer ..." or "I'm a novice trying to become better..." then at least they're honest individuals and not liars or hacks. If that's gatekeeping, then so be it. They are not engineers yet if they can't use the software equivalent of a ruler.
- muglug 4y ago> I don't want to be that person who says "mrahhh a new CS graduate could have built Teams in 2002 and it would have been faster!" But I'm also like, whatever is stopping it from being faster, I can't imagine what that is from where I'm sitting! > it's very simple data. Mainly a series of messages, displayed in chronological order. That's one of the most straightforward possible things to optimize. When a company grows and add features that customers need, the data model has to grow with it. Queries necessarily become more complex. I used to work at a big video sharing site. I wasn't there in the beginning, but I imagine the initial profile pages had code like SELECT * FROM user_videos WHERE user_id = ? Then some users said "hey, we want some of our videos to be private". So then the simple query became SELECT * FROM user_videos WHERE user_id = ? AND privacy = ? Then they said "ok, but I want people who are mutual friends to be able to see some just-for-friends videos". And then the query suddenly became much more convoluted. We started loading all of a users' videos in memory, then in-memory filtering, returning only the videos the viewer had access to (slow for users with thousands of videos). Then some users came along with 30,000 videos and we had to abandon the in-memory filtering because we were getting OOMs, so we developed a hybrid system that could run queries either in-memory or in the DB depending on volume. This is a trivial example (there are much more painful examples of data model bloat) but it illustrates the point: if you think it should be easy, you probably have an innacurate mental model of what needs to happen behind the scenes.
- fxtentacle 4y agoWe're talking about Teams here, the modern UI for what used to be IRC chat. 10k users on a single server used to be a solved problem for this type of application.
- bigtunacan 4y agoYes, that's the real elephant in the room. Slack and Teams both packaged up IRC into something that looks nicer, but runs 1000x slower and now sell enterprise licenses for this garbage.
- muglug 4y ago> We're talking about Teams here, the modern UI for what used to be IRC chat. That's like saying that Chrome is a modern UI for what used to be Lynx. Those tools offer very different features, even if they share a rudimentary set of behaviours.
- paulddraper 4y agoAs software features evolve over years, they gradually incur more and more incidental complexity. Architecture suffers, code health, and performance suffers. Silly things are done, but it's hard to see they're silly because it's hard to see the entire picture Could you build it better from scratch rather then organically? Of course. And often that's part of the solution to these situations. tl;dr Building software is hard, and its difficulty does not scale linearly with complexity. (And yes, Teams/Slack are complex, even if they "shouldn't be")
- fxtentacle 4y agoFor me, the only possible explanation is that Microsoft had nobody with actual experience left. The entire Teams team seems to be cheap junior developers together with CoPilot. It's not just that Teams is failing at solved problems, it's also that they aren't using Microsoft's own SDKs and APIs themselves anymore. Windows has good fast UI frameworks and all that cool stuff like SuperBoost which can make app startup appear instantly. Why is nobody using it? And there is no useful integration to speak of. Why do I need to re-upload an OneDrive file into Teams? The exact same file is already stored on the exact same server. Does nobody there know about the magic of SHA512? And how come if the internet has a hiccup while I upload a file, I have to re-upload the whole damn thing? In 1980 we already had resumable uploads via FTP. Oh and who's the vendor creating one of the best performance tracking and optimization tools? Microsoft themselves. Did nobody ever dare to run this in a proper Visual Studio?
- jtwaleson 4y agoThey need to support Mac, Linux, iOS, Android too. It’s nice if you can do that on the same codebase.
- NovaDudely 4y agoIt is possible but then, for example, Windows has always had some degree of amateurish stuff in it. Rumour was, so take this with a grain of salt, that the CD loader in Windows 95/98 was developed by a high school intern over a summer holiday. Would make sense how it was stall out the entire system while when you put a CD in until it has been verified.
- nitwit005 4y agoBlog post on the UI re-architecture: https://techcommunity.microsoft.com/t5/microsoft-teams-blog/microsoft-teams-advantages-of-the-new-architecture/ba-p/3775704 https://techcommunity.microsoft.com/t5/microsoft-teams-blog/... Sounds like they just rewrote the whole thing from scratch, down to replacing Electron.
- fxtentacle 4y agoThey replaced Electron, but without getting rid of the actual problems. It's still basically a React website wrapped in a web browser.
- stathibus 4y agoIt’s a skill issue. This is what you get when your developer team has only ever known programming through the lens of terrible tools and practices. They’ve probably never worked on good software before.
- bin_bash 4y ago> from scratch they replaced electron with a MS Edge Web View which is still running V8. I wouldn't say they rewrote it "from scratch"—that would be more like rewriting it in C#
- brundolf 4y agoOh this is great, thanks for sharing!
- millimeterman 4y agoI think the boring but true answer is simply that no one cares. Users don't care because Teams still basically functions and the performance problems are at most a minor inconvenience. As a result, organizations don't care because Teams is cheap and does everything they need it to. As a result, Microsoft doesn't care because dedicating dev resources to something which neither saves them money nor attracts more customers is a waste. Microsoft employs many smart people and I'm sure Teams' performance could be improved by orders of magnitude if they wanted to. But why would they want to? (And to reinforce my point, when the performance _did_ get so bad that people apparently actually started caring, Microsoft not only improved it but made a video advertising it. It's just that there's no incentive to optimize any further than is necessary.)
- illiarian 4y ago> Users don't care because Teams still basically functions People definitely care. If you actually listen to them. They will tell you what a hot pile of slow garbage it is.
- millimeterman 4y agoTalk is cheap - actions are what reveal true preferences. The fact that Teams is so widely used despite it being slow garbage indicates that people actually don't care (enough to overcome whatever benefits they see in Teams).
- brundolf 4y agoThat's part of it, but also, they just published a whole marketing video to advertise a performance improvement
- millimeterman 4y agoSorry, I imagine you posted this before my edit. I don't think this marketing video contradicts my point at all. For Microsoft to commit a large amount of dev time to rewriting Teams and improving its performance, there must have been an actual business reason. That's not something you do because some nerds on HN think your app is too slow. Presumably, Teams was so unbelievably slow that it crossed over to being an actual problem for customers and not just a mild nuisance. So Microsoft improved performance just enough to stop affecting sales. Is the performance _good_? No, it's still comically slow. But there's simply no incentive to improve the performance further.
- tshaddox 4y ago> But I'm also like, whatever is stopping it from being faster, I can't imagine what that is from where I'm sitting! Probably a few thousand features you have never heard of and will never see or use. And the vast majority of their customers will never see or use. But each one has one or two clients for whom someone decided it was vital at the time to implement that feature to please that client. You can probably imagine one small feature that only one or two clients use that adds a very tiny, almost imperceptible amount of loading time. Now just multiply that by several thousand.
- brewdad 4y agoEasily solved by introducing Teams Lite and Teams Professional. Maybe Teams Lite Plus and Teams Professional Lite. Eventually everyone can get exactly what they need.
- tshaddox 4y agoUntil you need just one of those thousands of features that are only in Professional.
- azemetre 4y agoI mean isn't this the actual problem? That teams are incentivized to pump out features and not delivery quality software? I remember a time in my career where I had to spend a good 2 months coming up with ways to reduce our build time in like 2014/2015. I was lucky to work at a company where most of leadership were engineers with engineering backgrounds but I can only imagine how bad my environment would have been if the room was full of MBAs. Seems like when you hire bean counters and MBAs, the incentive/value structure of your team/org/company changes for the worse.
- pixl97 4y agoCustomers buy features and not speed in most cases. When your customers do that, the MBA's are making you money.
- mike_hearn 4y ago
- WinLychee 4y agoI'd like to see a real study done on this, but if I were to hazard a guess, I think it boils down to: organization size, feature prioritization, development process, and finally lack of knowledge or care. It's a symptom of the way software gets written in the modern day, and isn't unique to Slack or Teams at all. Companies that don't do this tend to care obsessively about it, but it's not common at all IME. When you work in orgs with hundreds or thousands of developers working at a breakneck pace churning out feature after feature for years, on a perpetual "sprint", your architecture tends to resemble a multi-layered bowl of spaghetti. A single button click may result in hundreds of database calls (maybe concurrent, maybe cached), or at a minimum invoke dozens of javascript packages in a Rube Goldberg-esque fashion to perform an action which should take a modern computer a single digit number of milliseconds, but instead results in 10 second load times. Sometimes people just completely forget about entire code paths, and then it becomes dangerous to delete since the persons(s) who wrote it either no longer work there, or got reorged and moved on. There are absolutely people who care about software performance, but it isn't a priority because it usually doesn't make you more money, won't get you or your boss promoted (unless it saves money), and is difficult to maintain in the long-term given the pace of change in large codebases. When you do tackle performance problems, it's because you were absolutely forced to due to necessity. In certain domains it is also more important e.g. games, embedded systems, ML applications, etc. One other thought, is that many developers today cannot build software from first principals, as they never learned how. Many only know how to build within the context of a browser environment, and think a "minimal" app is https://create-react-app.dev/ https://create-react-app.dev/ . That's not just me grumbling, it's how our industry works. You forget the layers below you, and build new ones, each generation forgetting most of what the prior knew. The result is giant bloated applications, and a normalization of slowness in software. I still have hope for small teams of dedicated devs. Fast, coherent software gets written by tiny teams or single individuals.
- brundolf 4y agoYeah, the organizational issue is my guess too. Though I'm still amazed Teams managed to spiral down that path so quickly
- 4y ago
- intelVISA 4y agoAlways amusing to see the tech equivalent of fast food workers (web dev) being grilled because they can't cook gourmet steaks... well duh? ...I can't write CSS and nobody moans so why should web devs be expected to engineer software?
- prometheus76 4y agoBecause most of them don't know the difference between web dev and software engineering. Hence the joke of claiming that a 10-second startup time for what is essentially a web app is touted as advertising-worthy.
- NovaDudely 4y agoReminds me of how GTA Online was crawling on initial load because a single thread was parsing a 10MB JSON file. Maybe modern software is like that. Just a hundred small inefficiencies that don't look like much when benchmarked on an individual part of the stack but combine into a monster.
- muhehe 4y agoIt was about 25mb, but that would be still nothing for current computers. The problem was it was done by custom-made super inefficient parser.
- v8xi 4y agoI remember reading how MS Word formatting is such an utter PITA because of required backward compatibility going back to the first version(s) that were just done terribly. I wonder if it might be this way in all their apps
- runnerup 4y ago> I don't want to be that person who says "mrahhh a new CS graduate could have built Teams in 2002 and it would have been faster!" But I'm also like, whatever is stopping it from being faster, I can't imagine what that is from where I'm sitting! One way to test this...MSFT could open the Teams API and some 16 year old could try building a client.
- phendrenad2 4y agoHere's the deal: Making software is not challenging. A CS graduate from 2002 could build Teams. And it would be faster! You can hire dozens of engineers based on whiteboard problems, throw them into a disjointed cacophonous open-plan work environment, give them arbitrary deadlines and 10 layers of middle management, and somehow, somehow, software will still be built! There's no reason to streamline the architecture, or the workflow of the team. If the software is going to be made regardless, why bother? Just do whatever feels normal. Contrast this with the engineers who built the Saturn V. I'm sure a lot of brainpower went into thinking about the overall architecture of the rocket, and how the teams would be structured, and how they would communicate. Tons of process. Anyway, as a result of this lack of organization, software teams usually self-structure into a network of arbitrary mini-siloes. The way this works at FAANGs is you'll have the Foobar Core team, the Foobar Presence team, the Foobar Video team, the Foobar Audio team, the Foobar Notifications team, etc. etc. It's like microservices for teams. And they barely communicate (unless two of these teams are lucky enough to share the same corner of the same open-plan floorspace!) Anyway, these mini-siloes ensure that there are lots of seams between modules, and no one can optimize across boundaries easily, without stepping on the toes of someone else's team. And unless customers are actually switching to other products, there's no political willpower to do that toe-stepping in the name of saving the product.