5 ms·

TL;DR: There're few global incentives for a LinkedIn app developer to make an overall good app, as opposed to making their PM happy. In detail: This is how dea

by YouWhy 3y ago

TL;DR: There're few global incentives for a LinkedIn app developer to make an overall good app, as opposed to making their PM happy. In detail: This is how death by a thousand papercuts looks like. I've had the (dubious) pleasure of observing a roughly similar BigTech app project. There are likely 100s-1000s of app developers involved in the LinkedIn app, and they are likely under immense pressure to Just Ship Already (tm). Some of them consider it conscionable to do weird things like statically linking to an asset library - whether through unawareness/incompetence or just by wanting to make it through the next performance review. In smaller projects, there's some kind of a feedback that can make it through the system to make sure local and global incentives are aligned, but at the LinkedIn scale that would have required the people in charge to be engineers and not suit types. One interesting solution to this very problem that I actually saw in action was employing a team of top-tier systems hackers (in this case: people with a security research background) to hack through the build process. This solution naturally premises on such individuals wanting to work for an app project, incentivizing them along the right global metrics and giving them the go ahead to push back on egregious violations.

gertop 3y ago
> There are likely 100s-1000s of app developers involved in the LinkedIn app I don't understand why you'd need 1000 or even 100 developers to produce such app. It's a CRUD frontend for a backend that already exists. What am I missing? Are you counting the developers of all the third party libs they integrate?
- emj 3y ago
  Management might have been too interested in features to fix deep technical issues. > I had conversations with that manager where I was told in literally so many words, ‘You’re too idealistic. You don’t care enough about the bottom line. You should change your values.’ And I was like, no, nope, that’s not how this is going to work, man. Uh, outside I did the politic. https://news.ycombinator.com/item?id=39612443 https://news.ycombinator.com/item?id=39612443
- rob74 3y ago
  Furthermore, I would like to see some stats how much of their traffic is via the app and how much is via the website. Of course many people have LinkedIn profiles, but are there really so many "hardcore" users who install the app? So maybe the app just isn't that important for them in the grand scheme of things, and the bloat is more the result of neglect than of hundreds of developers working (specifically) on the app?
  - ungreased0675 3y ago
    The mobile web version is intentionally feature degraded, with occasional pop-up nags pushing the app. For that reason, I’d guess a lot of people are using the mobile app.
    - rob74 3y ago
      Ah! I know that from Yahoo Mail (the email address I still use for historical reasons). I even tried the app once, saw that it was actually worse than the website (which is not feature degraded as far as I can tell, at least not the features I use), and since then I just curse and ignore the nags...
    - RyanHamilton 3y ago
      If you use firefox mobile, there's a toggle switch under the menu to set "Desktop Version", I have found it prevents these intentional mobile degraded sites.
      - iggldiggl 3y ago
        Although "Desktop version" is also tied to a desktop viewport, so you have to deal with either too tiny font sizes or lots of scrolling.
        mrguyorama 3y ago
        Because god forbid tech companies actually build compliant and responsive web sites!
- LudwigNagasena 3y ago
  You need at least 100 developers to produce such an app. If there were only 5 or 10 devs, it would be much more performant and less bloated.
- phillipcarter 3y ago
  > It's a CRUD frontend for a backend that already exists. What am I missing? Probably a million features that are all relevant to different people who use the app? As a developer you probably use maybe 1% of LinkedIn, but you're not the primary user. It's a professional social network. It's going to have a lot to do.
- eddd-ddde 3y ago
  You are missing: - Every interaction needs to be recorded (metrics!) - Every interaction needs bot protection - A/B testing, show some users new stuff - Gather all that good user data for tracking and selling - Probably at some point they refactored the UI and now they maintain old and new stuff, they still ship everything of course If you in the web app and open the network tab you will ve surprised how much junk they are moving around for a seemingly simple app.
- jamesfinlayson 3y ago
  I kind of agree but I've worked for two companies in the same industry - one had an app team of five people, the other had an app team of 50 people. The apps were roughly identical in the services that they provided (they both sold the same products but maybe some of the payment methods were different). I'm still not sure why one company needed 10x the people to maintain the same thing.
rvba 3y ago
1) where are the architects? Arent they supposed to deal with such issues? 2) nobody measures such metrics: size, bloat, speed... 3) where are the code reviews? 4) nobody checks if the same library, same icon pack are not attached twice? 5) 1000 people? What do they even do? I suspect a lot of slides, politics and "high visibility projects" Meanwhile the product suffers. Companies sure talk a lot about ecology, then we get apps like that. Not related to Libkedin, but what is the carbon footprint of all electron-based apps that take hundreds of megabytes and gigantic amount of processor time?
- ninkendo 3y ago
  Speaking from experience in a similarly unhealthily large org: > 1) where are the architects? Arent they supposed to deal with such issues? The architects are the ones being told “we need these features, figure out how to do it”, and they build little box charts and sequence diagrams to sort out which systems are responsible for which. Never mind if a “feature” could just be a few functions somewhere. Architects think in terms of boxes and what does what. If the current boxes don’t have a role for a new feature, a new box is created. Boxes are assigned to engineers and interfaces are designed. Nobody cares if things can be done more simply, because the architects are looking to climb the ladder too (more boxes means your job is more justified.) These same architects often don’t understand what each box actually does in implementation either, so they often are blind to radically simpler approaches to problems (I’ve seen cases where people just have no idea that a box already does the thing they want to do, and instead spend an entire quarter trying to build the functionality into the edges of the system, plumbing huge amounts of stuff around, when the thing they wanted could have been a 2-line change. These are fun meetings when that realization happens.) > 2) nobody measures such metrics: size, bloat, speed... They might be measuring speed, but with a kind of perverted reverse ratchet, where they say “this is within X% of production, so the regression is acceptable” and it ships. Once a perf regression ships it becomes the new baseline, so the next change can then be within X% of this one, and in turn the product gets exponentially slower. Nobody ever looks more than one release back. Nobody ever sets out to improve speed, unless by happy accident. If you stumble upon some horribly slow thing and fix it in happenstance, you can get a nice bonus. It’s never the plan though. > 3) where are the code reviews? There are code reviews but they’re completely toothless because the person making the change can easily say “this is high priority for making $DEADLINE” and push it through while filing a ticket to fix it properly in a follow up. The follow up never happens, and in turn any bad code becomes the new baseline and used as an excuse to make the same bad choice in other areas. If you actually reject a PR and push back against the deadline, you are considered not a team player, and, guess what? You will get routed around anyway. They will find a way to get this change into another part of the system you don’t control (because code owner’s files mean there’s probably some other place they can make the change at some cost to the system complexity. This is another factor in (1), because the code base has fiefdoms and often architectural boxes are created purely because nobody wants to make PR’s into box A because of that one grumpy engineer, so functionality is built into box B instead.) > 4) nobody checks if the same library, same icon pack are not attached twice? Yes but there’s always an excuse, see 3. > 5) 1000 people? What do they even do? I suspect a lot of slides, politics and "high visibility projects" The bureaucracy expands to meet the needs of the expanding bureaucracy. All this complexity needs tooling, all this tooling causes complexity. Existing behavior is used to justify bad decisions, bad decisions become existing behavior, the cycle repeats. The real problem with these organizations is that nobody dare admit (or even attempt to understand) how few engineers these kinds of products could require if things were done carefully from the start. And after years of such an org, shrinking becomes impossible: the weight of the complexity of the product means you need all the engineers you have to manage it. The mistakes were already made and there’s no way to close Pandora’s box here. You have to carry on until some day it just collapses (or gets disrupted, etc.) > Not related to Libkedin, but what is the carbon footprint of all electron-based apps that take hundreds of megabytes and gigantic amount of processor time? Trust me I’d love to hate on electron and JavaScript as much as the next person, but all you read above is from an org where we write bare metal code using as many system frameworks as possible, using everything we can that’s native to the platform. No JS, no GC’d languages, no electron/etc. As native of an app as you can get. This stuff has basically nothing to do with tech stack, it’s all organizational. I’ve heard it described as “building software at large” but it really means “letting the software grow unchecked”, and it’s the attitude that’s the problem, not the technology.

by YouWhy 3y ago

gertop 3y ago

> There are likely 100s-1000s of app developers involved in the LinkedIn app I don't understand why you'd need 1000 or even 100 developers to produce such app. It's a CRUD frontend for a backend that already exists. What am I missing? Are you counting the developers of all the third party libs they integrate?

emj 3y ago
Management might have been too interested in features to fix deep technical issues. > I had conversations with that manager where I was told in literally so many words, ‘You’re too idealistic. You don’t care enough about the bottom line. You should change your values.’ And I was like, no, nope, that’s not how this is going to work, man. Uh, outside I did the politic. https://news.ycombinator.com/item?id=39612443 https://news.ycombinator.com/item?id=39612443
rob74 3y ago
Furthermore, I would like to see some stats how much of their traffic is via the app and how much is via the website. Of course many people have LinkedIn profiles, but are there really so many "hardcore" users who install the app? So maybe the app just isn't that important for them in the grand scheme of things, and the bloat is more the result of neglect than of hundreds of developers working (specifically) on the app?
- ungreased0675 3y ago
  The mobile web version is intentionally feature degraded, with occasional pop-up nags pushing the app. For that reason, I’d guess a lot of people are using the mobile app.
  - rob74 3y ago
    Ah! I know that from Yahoo Mail (the email address I still use for historical reasons). I even tried the app once, saw that it was actually worse than the website (which is not feature degraded as far as I can tell, at least not the features I use), and since then I just curse and ignore the nags...
  - RyanHamilton 3y ago
    If you use firefox mobile, there's a toggle switch under the menu to set "Desktop Version", I have found it prevents these intentional mobile degraded sites.
    - iggldiggl 3y ago
      Although "Desktop version" is also tied to a desktop viewport, so you have to deal with either too tiny font sizes or lots of scrolling.
      - mrguyorama 3y ago
        Because god forbid tech companies actually build compliant and responsive web sites!
LudwigNagasena 3y ago
You need at least 100 developers to produce such an app. If there were only 5 or 10 devs, it would be much more performant and less bloated.
phillipcarter 3y ago
> It's a CRUD frontend for a backend that already exists. What am I missing? Probably a million features that are all relevant to different people who use the app? As a developer you probably use maybe 1% of LinkedIn, but you're not the primary user. It's a professional social network. It's going to have a lot to do.
eddd-ddde 3y ago
You are missing: - Every interaction needs to be recorded (metrics!) - Every interaction needs bot protection - A/B testing, show some users new stuff - Gather all that good user data for tracking and selling - Probably at some point they refactored the UI and now they maintain old and new stuff, they still ship everything of course If you in the web app and open the network tab you will ve surprised how much junk they are moving around for a seemingly simple app.
jamesfinlayson 3y ago
I kind of agree but I've worked for two companies in the same industry - one had an app team of five people, the other had an app team of 50 people. The apps were roughly identical in the services that they provided (they both sold the same products but maybe some of the payment methods were different). I'm still not sure why one company needed 10x the people to maintain the same thing.

rvba 3y ago

1) where are the architects? Arent they supposed to deal with such issues? 2) nobody measures such metrics: size, bloat, speed... 3) where are the code reviews? 4) nobody checks if the same library, same icon pack are not attached twice? 5) 1000 people? What do they even do? I suspect a lot of slides, politics and "high visibility projects" Meanwhile the product suffers. Companies sure talk a lot about ecology, then we get apps like that. Not related to Libkedin, but what is the carbon footprint of all electron-based apps that take hundreds of megabytes and gigantic amount of processor time?

ninkendo 3y ago
Speaking from experience in a similarly unhealthily large org: > 1) where are the architects? Arent they supposed to deal with such issues? The architects are the ones being told “we need these features, figure out how to do it”, and they build little box charts and sequence diagrams to sort out which systems are responsible for which. Never mind if a “feature” could just be a few functions somewhere. Architects think in terms of boxes and what does what. If the current boxes don’t have a role for a new feature, a new box is created. Boxes are assigned to engineers and interfaces are designed. Nobody cares if things can be done more simply, because the architects are looking to climb the ladder too (more boxes means your job is more justified.) These same architects often don’t understand what each box actually does in implementation either, so they often are blind to radically simpler approaches to problems (I’ve seen cases where people just have no idea that a box already does the thing they want to do, and instead spend an entire quarter trying to build the functionality into the edges of the system, plumbing huge amounts of stuff around, when the thing they wanted could have been a 2-line change. These are fun meetings when that realization happens.) > 2) nobody measures such metrics: size, bloat, speed... They might be measuring speed, but with a kind of perverted reverse ratchet, where they say “this is within X% of production, so the regression is acceptable” and it ships. Once a perf regression ships it becomes the new baseline, so the next change can then be within X% of this one, and in turn the product gets exponentially slower. Nobody ever looks more than one release back. Nobody ever sets out to improve speed, unless by happy accident. If you stumble upon some horribly slow thing and fix it in happenstance, you can get a nice bonus. It’s never the plan though. > 3) where are the code reviews? There are code reviews but they’re completely toothless because the person making the change can easily say “this is high priority for making $DEADLINE” and push it through while filing a ticket to fix it properly in a follow up. The follow up never happens, and in turn any bad code becomes the new baseline and used as an excuse to make the same bad choice in other areas. If you actually reject a PR and push back against the deadline, you are considered not a team player, and, guess what? You will get routed around anyway. They will find a way to get this change into another part of the system you don’t control (because code owner’s files mean there’s probably some other place they can make the change at some cost to the system complexity. This is another factor in (1), because the code base has fiefdoms and often architectural boxes are created purely because nobody wants to make PR’s into box A because of that one grumpy engineer, so functionality is built into box B instead.) > 4) nobody checks if the same library, same icon pack are not attached twice? Yes but there’s always an excuse, see 3. > 5) 1000 people? What do they even do? I suspect a lot of slides, politics and "high visibility projects" The bureaucracy expands to meet the needs of the expanding bureaucracy. All this complexity needs tooling, all this tooling causes complexity. Existing behavior is used to justify bad decisions, bad decisions become existing behavior, the cycle repeats. The real problem with these organizations is that nobody dare admit (or even attempt to understand) how few engineers these kinds of products could require if things were done carefully from the start. And after years of such an org, shrinking becomes impossible: the weight of the complexity of the product means you need all the engineers you have to manage it. The mistakes were already made and there’s no way to close Pandora’s box here. You have to carry on until some day it just collapses (or gets disrupted, etc.) > Not related to Libkedin, but what is the carbon footprint of all electron-based apps that take hundreds of megabytes and gigantic amount of processor time? Trust me I’d love to hate on electron and JavaScript as much as the next person, but all you read above is from an org where we write bare metal code using as many system frameworks as possible, using everything we can that’s native to the platform. No JS, no GC’d languages, no electron/etc. As native of an app as you can get. This stuff has basically nothing to do with tech stack, it’s all organizational. I’ve heard it described as “building software at large” but it really means “letting the software grow unchecked”, and it’s the attitude that’s the problem, not the technology.