9 ms·

A story from a physicist friend working in a mid-size high-tech company: There was a director-level organization of about 50 people trying to solve a technical

by alok-g 3y ago

A story from a physicist friend working in a mid-size high-tech company: There was a director-level organization of about 50 people trying to solve a technical research-level problem. The physicist found an innovative solution that was too simple and just worked. When he demonstrated it to everyone, he thought everyone would be very happy. On the contrary, his manager started having unexplained issues with him and he was shortly let go. He has ultimately given up on corporate politics and is currently a physics teacher at a usual local college. As the physics teacher, he found that the students in his class barely knew basic stuff (vector addition, etc.) from the pre-requisite courses. He tried to teach to cover up. He diligently and fairly checked the answers/solutions from the students in the mid-term exams, and found that practically no one passed. He was shortly called by the college dean and heard back. The dean's primary concern was that irrespective of whether students actually learn or not, if so many fail the exams, the college would get shut down. He is now forced to pass the students even while he realizes that by passing them now, he's setting them up for failure in the future if they actually pursue physics. I surmise that if the Shirky Principle were to stop acting, say 80% of projects/institutions may just go away and should. ---- In some cases, it's even worse -- Institutions invent fake problems that they allege to be the solution for and then of course never solve them.

matthewdgreen 3y ago
On the one hand, your friend could just be experiencing a streak of bad luck or a series of bad organizations. On the other hand, after many years of life experience, when I hear a (first or second hand) account that takes the form “of course I was the reasonable one in every case and these N organizations were clearly behaving like idiots”, I’ve learned to consider the possibility that the speaker is just unusually difficult. The larger the value of N, the more seriously I consider it. And I’ve learned this the hard way.
- mturmon 3y ago
  Yes. Besides “N”, another indicator you alluded to is the degree to which the aggrieved person is willing to take any responsibility for the outcome. Sometimes gifted people can negotiate problems with a nervous college dean or engineering manager, so that their solution gets adopted. These problems exist in the world, and successful engineers will hopefully learn to cope with them.
- deleted 3y ago
  [deleted]
- ip26 3y ago
  Not to mention the unrecognized genius trope. In more familiar terms, the “I could do that in a weekend” guy (https://blog.codinghorror.com/code-its-trivial/amp/ https://blog.codinghorror.com/code-its-trivial/amp/)
  - armchairhacker 3y ago
    Ironically Stack Overflow is one of the best-functioning sites I know of. Community issues, sure, but the pages load really quickly and the UI highlights exactly what it should. But then there are some big sites which truly suck: massive load time and basic UI issues. Like outdated government/university sites, job application sites, Kroger’s online store (https://joshstrange.com/2024/02/11/krogers-digital-struggle/ https://joshstrange.com/2024/02/11/krogers-digital-struggle/ | https://news.ycombinator.com/item?id=39345309 https://news.ycombinator.com/item?id=39345309). And I know some of these organizations suffer from inadequate funding, legacy systems, and/or compliance rules which make their sites’ issues legitimate, and I’m certain no developer could recreate them in a weekend. But I’m also convinced many of them have issues because of carelessness and/or inefficiency, and a few may be actively trying to emulate bad UX (probably to deter people from using them).
- markerz 3y ago
  I think both sides are true. On the one hand, dysfunction exists in all organizations. On the other hand, it takes effort to maintain a high functioning organization and most people don’t want to put in the real hard work. Not only are most people resistant to change, but the gratitude/reward probably won’t be there. It’s far easier to complain and wallow in a stuck situation than to sit through the discomfort of pushing for change. Schools, non-profits, government programs, corporations, they all experience it. While I can’t say I support this way of life, I can definitely understand why it exists.
  - ClumsyPilot 3y ago
    > gratitude/reward probably won’t be there. It’s far easier to complain and wallow in a stuck situation than to sit through the discomfort of pushing for change. That's the job of management, that's why they get rewarded for that, and I have a feeling we have a lot of bad management.
    - Xylakant 3y ago
      “managing” their workspace and surroundings is the jobs of every senior person in a workplace. Having a line manager that cares and is good at their job helps, but if technical folks believe they can just care about the technical part of a job, they’re setting themselves up for failure and frustration. Getting a solution adopted needs much more than technical excellence, it needs communication, an eye for human factors, an idea of costs and tradeoffs. Complaining about “bad management” is a symptom of a lack of awareness. There’s a lot of bad management, but there’s about as many technical people that just ignore all of those non-technical factors and then constantly complain that their solutions never make it anywhere.
      - generic92034 3y ago
        In my eyes there is a good deal of victim blaming in that view. Time and again engineers pushing for changes, ignoring borders of teams or (sub-)orgs and the management hierarchy are met with highly toxic reactions, usually by managers who are seeing some kind of dangerous insubordination.
- Dylan16807 3y ago
  N here is only 2. It's pretty plausible. "Streak" and "series" are overstating 2 by a lot.
  - coldtea 3y ago
    You can go to N=1000 schools and see the same attitude. It's an "unspoken understanding" that a big chunk of the students shall graduate even if most know shit, else there will be trouble from parents, the state, management, and so on.
- dclowd9901 3y ago
  Literally every organization I've worked (probably 10 or 12 so far) for has had some glaring example of this principle.
- alok-g 3y ago
  I align. We note that N is only an indicator. To actually answer, one needs to seek and analyze a detail after detail of facts. >> And I’ve learned this the hard way. I would like to hear more to learn from your experience. For the said physicist, I understand the situation well already.
- jiggawatts 3y ago
  > the speaker is just unusually difficult. "Your actions are making it unusually difficult for us to continue rent seeking!" I bet the Mafia considers that young police officer that actually wants to stop crime "unusually difficult" as well. Generally when everyone is corrupt, non-corruption is seen as bad and evil.
- atoav 3y ago
  As someone who (by the words of a collegue) put a lot of skill points into diplomacy: there are many clever people who just suck at communicating their solution to the right people in the right way. And then when things don't work out they think it is about the facts (and granted: often it is) and not about the communications. Institutional politics means it matters what you propose in front of whom in which order and in which way. This is especially true if you are not surrounded by idealists but by oportunists.
  - SamPatt 3y ago
    Agreed. The mistake I made when I was younger was thinking that good communication was about being able to clearly and concisely convey the information in your head into another head. I eventually learned that this was necessary but not sufficient, because "the right way" to communicate varies significantly based on social / political / cultural / emotional / institutional dynamics. As a young man I simply didn't understand enough of those dynamics to be able to model the when and the why of communication (meta-communication?), even if I was good at the idea transmission part. Life experience helps immensely here.
    - navane 3y ago
      Or you became "one of them", and now you too have an agenda.
      - catlifeonmars 3y ago
        No it’s not about that. I’ve seen this failure case over and over again. When you communicate, it’s not enough to explain a solution. You have to painstakingly help others internalize the problem. Once you’ve done that, and gained enough critical mass, you find that the solution almost communicates itself. It’s almost like magic when you see an organization shift its thinking like this
        Xcelerate 3y ago
        > When you communicate, it’s not enough to explain a solution. You have to painstakingly help others internalize the problem. This is why I think startups or small groups of people are still able to outcompete bigger incumbents. So many people worry “if I share my great idea, someone will steal it and implement it!” Far more often I’ve found it to be the case that I catch myself wondering if it would be less work to start a competitor to my current company based on my idea rather than continue attempting to translate it, broadcast it, and massage it into some form that convinces a critical mass of coworkers of its value such that it eventually turns into action. At least by starting my own company, I’ll know pretty quickly whether it was actually a good idea or not.
- underdeserver 3y ago
  I completely agree with your sentiment, but I think whether or not you did vector arithmetic correctly at a first-year undergraduate course level is not subjective.
  - deleted 3y ago
    [deleted]
- jokethrowaway 3y ago
  I matured a different, if not opposite viewpoint: every entity acts in it own self interest, so if the incentives are not aligned, things will tend go south. It may work for a long period of time because of sheer will power and commitment to a greater good from the people involved in the organisation, but over time, everything will follow market incentives. Public health care seems to be a primary example (and my family has been in the sector for more than 40 years); HC was good for a long period of time then it got progressively worse and nowadays it's just an item in your tax bill. If you literally don't want to die you better know someone on the inside which can push the right buttons or go private. The money spent on law enforcement had a reverse effect on crime rate, the police actually got less efficient the more money they manage to extract from public funds. They got really good at getting more money from ridiculous speed limits and automated systems though. Looking in the private tech sector my experience has been similar: the larger the organisation the more it resemble a government and the more inefficiency is tolerated. Within layers and layers of middle management it's easy to waste plenty of investors' money - albeit not indefinitely, like a government would.
- coldtea 3y ago
  >On the one hand, your friend could just be experiencing a streak of bad luck or a series of bad organizations. I've seen the same play out in several organizations, including the public sector. You don't even need to be cynical to know this is the case... >when I hear a (first or second hand) account that takes the form “of course I was the reasonable one in every case and these N organizations were clearly behaving like idiots”, I’ve learned to consider the possibility that the speaker is just unusually difficult. Probably based on different experiences, with different context. Because in the context described (e.g. the educational example) this is par for the course - and any number of teachers and professors can attest that if you do your work too well and apply rigid standards, you're out of there in no time, or at least make enemies very fast. >the speaker is just unusually difficult. Of course, if the baseline is complacency, profiteering, and lack of standards, anybody who stirs those waters is "unusually difficult.".
  - simonh 3y ago
    The college example doesn’t seem likely though. Exams are set by central authorities and independently adjudicated. It’s not up to teachers whether students pass or not, and parents and education authorities look at exam results to judge colleges, not teacher assessments.
    - coldtea 3y ago
      Not sure if you have in mind entrance exams or something that's not set by professors (he mentions mid-terms). Or perhaps you have your local example in mind. This however is absolutely the case in my country, exam questions are set by the teachers, and are graded by them (well, often also by postgraduates given this task by the teacher so they can slack). As for the US, I did a quick search now, and found this: "Unlike final exams, which are scheduled by the university’s Registrar, midterm exams are typically scheduled during class time by the professor. Some classes may have two midterm exams, in which case they are spread further throughout the semester. Professors outline these exams in the course syllabus, so they will not come as a surprise. The weight of each midterm exam on the final grade is also usually provided in the course syllabus. Many instructors are open to telling students about the format of the midterm exam, as well as the topics or themes that the exam will cover." So looks like it's on the instructors, as the grandparent says there too.
- itronitron 3y ago
  In the case of the college professor and the dean, who is being the reasonable one?
- whatshisface 3y ago
  Two is not a large value of N for the phenomenon you're talking about. Also... is it really that hard to believe that a manager would defend their headcount and a community college would let standards slide in their hardest courses? You also have to consider the plausibility of the stories at hand before writing off the person who's complaining. I don't think your skepticism is at all warranted.
thaumasiotes 3y ago
> In some cases, it's even worse -- Institutions invent fake problems that they allege to be the solution for and then of course never solve them. Why is that worse? In scenario A, you have an institution that fights to prevent a problem from being solved. If you ignore them and their politics, you'll have the problem that they protect. In scenario B, you have an institution that fights to convince you that you have a problem. If you ignore them and their politics, you won't have the problem. But scenario A is better?
- alok-g 3y ago
  Consider how the reasoning proceeds if we do not take solving a problem as a Boolean. Also let's measure not just by whether the problem exists or not (or to what degree) but also by how much overheads or missed opportunity the society is having with the institutions even just existing. In the scenario A, knowingly or unknowingly, they would tend to solve the problem partially. The others around would see their contributions so far and foresee their value for the future. If others discover or perceive that the institution is not holding any promise for the future, the institution could struggle to survive. In the scenario B, the institution existing is entirely a lost opportunity for the society. Hence the scenario A is typically better (for making progress with given amount of resources).
  - thaumasiotes 3y ago
    In scenario A, the problem is already solved, and the institution exists to prevent that solution from being implemented. That's the whole point of this comment thread. Surely that is not better than lying about an imaginary problem?
    - alok-g 3y ago
      >> In scenario A, the problem is already solved, and the institution exists to prevent that solution from being implemented. In my understanding, that's one way. Another may be to keep making slow progress to solving so that those around see the contributions and the relevance of the institution but not fully solve. Another way may be to show their relevance even if the problem is not getting solved by convincing that the situation would be worse without them. And so on.
jongjong 3y ago
This sounds similar to my own story in the blockchain sector. I joined a major crypto project which was under-delivering on their promises to investors. In my own time, I built a prototype which would have allowed them to meet their promises and would have appeased regulators whom I presumed were breathing down their necks. Win-win right? Perfect plan! What a genius move it was to use my skills in the most optimal way possible to wedge myself between this very cash-rich company and a large group of dissatisfied investors... with the backing of regulators of one of the wealthiest, most trusted nations on earth to add even more pressure! Surely, the company would be overjoyed, adopt my project, offer me a salary increase and a bonus package... WRONG. They tried to cancel the project. I had to quit my job in order to pursue the project outside of the company... Then they spent years gaslighting their own employees and community to keep everyone away from my project to ruin my prospects... The tech worked perfectly; that was several years ago and it's still running. Never encountered any bug or hack which is very unusual for this kind of tech. That company I used to work for essentially ended up abandoning their own project (after spending many years wasting millions of dollars on like 30 engineers). With 15x the engineering capacity, they couldn't deliver in 3 years what I and a friend built in 1 year. Now I'm demoralized from the entire tech industry and basically gave up on my career (until there is a complete political system change?). If this epic plan which was executed almost perfectly didn't work and there is no legal recourse for me due to institutional corruption, then what chance do I have in the future in such system? I will never get such opportunity again... And even if I do and it's executed perfectly, it's not going to work out in my favor because bad actors can essentially get away with everything.
- x-complexity 3y ago
  Can you share a link to your project? It sounds interesting.
Buttons840 3y ago
Your physicist friend solving the "technical research-level problem" should have evaluated whether it was a real problem, or a problem the company artificially created for themselves. If it was a real problem, and the company ignored his solution, then take the solution to market and become a competitor. If it was an artificial problem only found at that one company, then be very stingy about any effort you put into solving it. Give no free thought to the problem outside of work. Find some real problems to work on instead.
- alok-g 3y ago
  >> Your physicist friend ... should have evaluated whether it was a real problem ... I do not have the facts around this, however knowing him, I would guess that it was a real problem. >> ... and the company ignored his solution, then take the solution to market and become a competitor. As you know, this is not so straightforward in practice: * Not everyone is a entrepreneur, for whatever reason that is. * Not everyone has a brand to be able to raise money. * It may not be wise for investors to fund a project completing with an established/powerful mid-size company that owns a wide range in that space. * Some projects may require large investments and teams especially when the other side is a big company. * The big companies deploy all kind of schemes if they see a threat. * One of the threats is the existing intellectual property rights in that space that the company would already have. * As the invention was already made when the person was employed at the company, the invention as such likely already belongs to the company (as per typical employment agreements that most do not care to read), and this already forms a threat falling in the above bullet point. * The corporate politics would start at the newly started company itself becoming an internal threat. * And so on. Hopefully you do not see all above as unreal, a fake problem. :-) Not to say there aren't successes -- We hear about those so often. [Edit:] Well, someone cited a story like that right here in this thread. https://news.ycombinator.com/item?id=39500316 https://news.ycombinator.com/item?id=39500316
  - Buttons840 3y ago
    Yes. I said he should do this and that. That's more forceful and judgemental than I intended. I meant it only as a philosophy to consider, but didn't know how to soften my message while keeping it succinct.
aunty_helen 3y ago
Addressing the second situation because it’s easier to bike shed about, I feel like your friend went to the dean with a problem rather than a solution. You said it yourself, the dean’s primary concern was keeping the school running, which in contrast to letting some loser kids pass a physics class seems very reasonable. It’s also the reason why trusting straight-A students is using a flawed metric, there are A students and then there are students with A grades. We have more complex inspection processes to deal with this. A better solution (full bike shed mode now), would be to help address the deficiencies in the class at an earlier level. Talk with previous teachers, get them feedback about what it is that kids are missing when they get to his class. Help them address the root cause of the problem. “Dean, I’ve found X, fed it back to teachers Y and began to see a yoy improvement in the pass rate of class Z.” Beautiful.
- alok-g 3y ago
  I do not have all the facts of the situation at hand. However, I have no reason to believe that my friend would not have done all that. I do know for a fact that he is actively trying to teach the students those pre-requisite concepts himself. The dean should have talked about digging into the problem as needed (my friend also could have been doing something wrong; knowing him, I know he would have readily accepted his mistakes if that were the case) and help solve, instead of forcing him to pass the students. I do recognize that this story is not a good example of the Shirky principle. An educational institute is likely genuinely trying to solve the problem, which is to educate the students. That problem continues because fresh students keep coming in, not because the institute itself is preventing the solution. An educational institution cannot be expected to pursue inventing a solution that disrupts the education sector altogether.
datavirtue 3y ago
Yep, as an engineer I always put on my business cap before opening my mouth or setting priorities. The systems' inertia will steamroll an individual without thinking. I have end-users that I think about every day when building my products, but then I have my customers that are ultimately prioritized.
specialist 3y ago
> ...found an innovative solution that was too simple and just worked. Yup. For years, the big boss exalted us to find the next generation solution. Mr big boss was original programmer, founder, and majority owner. We had an Illustrator/CAD style program (ScenicSoft's Preps) for designing print production plans. My office was right next to big boss, to facilitate my efforts. I kept him in the loop on my progress. It took a while, but I did exactly that. Reduced the majority of the design (image positioning) work to a simple form. Two views; fields on left and live preview on the right. What could be easier? (These kinds of things almost write themselves once you find the correct mental model, which is typically the really hard part.) I started demoing my solution to my peers and SMEs. Wow, Bravo, Amazing. Then I demo'd for big boss. Big boss said nothing. Walked out. Never spoke to or even acknowledged me again. Ghosted. -- I've been noodling on a novel SQL JDBC wrapper thing. For ages now. Input stock SQL and output typesafe wrappers. I use it for all my own work. It's stupid simple and easy. Every single person whose seen a demo just doesn't "get it". They expect an ORM or a template or a fluent API. The reactions have really unsettled me. From my prior innovations, I'm used to pushback, debate, rancor, battle lines. I just don't know what to do with blank befuddlement. (Yes, I'm slowly working towards a FOSS release, as able.)
- armchairhacker 3y ago
  > I've been noodling on a novel SQL JDBC wrapper thing. For ages now. Input stock SQL and output typesafe wrappers. I use it for all my own work. It's stupid simple and easy. Just wanted to say this seems like how I imagine using SQL. It’s well-known that ORMs are a bad abstraction. Although given that it’s well-known, there’s already a lot of alternate solutions, so I’m interested what yours is and why people may be “befuddled”.
  - specialist 3y ago
    Converts normal SQL DML statements into prepared statements and type-safe wrappers. For Java, that means subclassing PreparedStatement and ResultSet. No templates, maps, annotations, DSL. Just normal SQL. Apologies, a few weeks ago, I started yak shaving while implementing the build stuff (eg maven plugin). I'll try to focus, ship something beta sooner than later.
lippihom 3y ago
Have a few engineering professor friends that have said the same thing regarding have to pass a certain % of their students, regardless of if they know the material or not. Generally it seems like colleges are just kicking this lack of learning / understanding down the line to prospective employers to vet.
dustingetz 3y ago
tell him the world’s eigenstates are discrete and aligned to vectors that maximize local cashflow
silexia 3y ago
"I surmise that if the Shirky Principle were to stop acting, say 80% of projects/institutions may just go away and should." This describes almost all government institutions.
phendrenad2 3y ago
> Institutions invent fake problems that they allege to be the solution for and then of course never solve them I think lost companies that fail to find product/market fit go through this phase before they collapse. They try to live on pure willpower and reality-shifting.
- pas 3y ago
  Or when the market shifts and they survive on inertia, for a while.
a_gnostic 3y ago
> In some cases, it's even worse -- Institutions invent fake problems that they allege to be the solution for and then of course never solve them. Plenty of corporate and government institutions stand ready to "solve" those problems…
matttproud 3y ago
Small question: I recall there being a named term for organizations that outlive their usefulness and sometimes become parasitic on their original mission. Is this the same as the Shirky Principle? I was introduced to this term many years ago, so I think it's older than Shirky. Any idea what this might have been?
amenhotep 3y ago
> In some cases, it's even worse -- Institutions invent fake problems that they allege to be the solution for and then of course never solve them. Rather frequently, and worst of all, I think, when this isn't done by an institution but by the symbiotic organism that arises between a media interest and a political party.

by alok-g 3y ago

matthewdgreen 3y ago

On the one hand, your friend could just be experiencing a streak of bad luck or a series of bad organizations. On the other hand, after many years of life experience, when I hear a (first or second hand) account that takes the form “of course I was the reasonable one in every case and these N organizations were clearly behaving like idiots”, I’ve learned to consider the possibility that the speaker is just unusually difficult. The larger the value of N, the more seriously I consider it. And I’ve learned this the hard way.

mturmon 3y ago
Yes. Besides “N”, another indicator you alluded to is the degree to which the aggrieved person is willing to take any responsibility for the outcome. Sometimes gifted people can negotiate problems with a nervous college dean or engineering manager, so that their solution gets adopted. These problems exist in the world, and successful engineers will hopefully learn to cope with them.
deleted 3y ago
[deleted]
ip26 3y ago
Not to mention the unrecognized genius trope. In more familiar terms, the “I could do that in a weekend” guy (https://blog.codinghorror.com/code-its-trivial/amp/ https://blog.codinghorror.com/code-its-trivial/amp/)
- armchairhacker 3y ago
  Ironically Stack Overflow is one of the best-functioning sites I know of. Community issues, sure, but the pages load really quickly and the UI highlights exactly what it should. But then there are some big sites which truly suck: massive load time and basic UI issues. Like outdated government/university sites, job application sites, Kroger’s online store (https://joshstrange.com/2024/02/11/krogers-digital-struggle/ https://joshstrange.com/2024/02/11/krogers-digital-struggle/ | https://news.ycombinator.com/item?id=39345309 https://news.ycombinator.com/item?id=39345309). And I know some of these organizations suffer from inadequate funding, legacy systems, and/or compliance rules which make their sites’ issues legitimate, and I’m certain no developer could recreate them in a weekend. But I’m also convinced many of them have issues because of carelessness and/or inefficiency, and a few may be actively trying to emulate bad UX (probably to deter people from using them).
markerz 3y ago
I think both sides are true. On the one hand, dysfunction exists in all organizations. On the other hand, it takes effort to maintain a high functioning organization and most people don’t want to put in the real hard work. Not only are most people resistant to change, but the gratitude/reward probably won’t be there. It’s far easier to complain and wallow in a stuck situation than to sit through the discomfort of pushing for change. Schools, non-profits, government programs, corporations, they all experience it. While I can’t say I support this way of life, I can definitely understand why it exists.
- ClumsyPilot 3y ago
  > gratitude/reward probably won’t be there. It’s far easier to complain and wallow in a stuck situation than to sit through the discomfort of pushing for change. That's the job of management, that's why they get rewarded for that, and I have a feeling we have a lot of bad management.
  - Xylakant 3y ago
    “managing” their workspace and surroundings is the jobs of every senior person in a workplace. Having a line manager that cares and is good at their job helps, but if technical folks believe they can just care about the technical part of a job, they’re setting themselves up for failure and frustration. Getting a solution adopted needs much more than technical excellence, it needs communication, an eye for human factors, an idea of costs and tradeoffs. Complaining about “bad management” is a symptom of a lack of awareness. There’s a lot of bad management, but there’s about as many technical people that just ignore all of those non-technical factors and then constantly complain that their solutions never make it anywhere.
    - generic92034 3y ago
      In my eyes there is a good deal of victim blaming in that view. Time and again engineers pushing for changes, ignoring borders of teams or (sub-)orgs and the management hierarchy are met with highly toxic reactions, usually by managers who are seeing some kind of dangerous insubordination.
Dylan16807 3y ago
N here is only 2. It's pretty plausible. "Streak" and "series" are overstating 2 by a lot.
- coldtea 3y ago
  You can go to N=1000 schools and see the same attitude. It's an "unspoken understanding" that a big chunk of the students shall graduate even if most know shit, else there will be trouble from parents, the state, management, and so on.
dclowd9901 3y ago
Literally every organization I've worked (probably 10 or 12 so far) for has had some glaring example of this principle.
alok-g 3y ago
I align. We note that N is only an indicator. To actually answer, one needs to seek and analyze a detail after detail of facts. >> And I’ve learned this the hard way. I would like to hear more to learn from your experience. For the said physicist, I understand the situation well already.
jiggawatts 3y ago
> the speaker is just unusually difficult. "Your actions are making it unusually difficult for us to continue rent seeking!" I bet the Mafia considers that young police officer that actually wants to stop crime "unusually difficult" as well. Generally when everyone is corrupt, non-corruption is seen as bad and evil.
atoav 3y ago
As someone who (by the words of a collegue) put a lot of skill points into diplomacy: there are many clever people who just suck at communicating their solution to the right people in the right way. And then when things don't work out they think it is about the facts (and granted: often it is) and not about the communications. Institutional politics means it matters what you propose in front of whom in which order and in which way. This is especially true if you are not surrounded by idealists but by oportunists.
- SamPatt 3y ago
  Agreed. The mistake I made when I was younger was thinking that good communication was about being able to clearly and concisely convey the information in your head into another head. I eventually learned that this was necessary but not sufficient, because "the right way" to communicate varies significantly based on social / political / cultural / emotional / institutional dynamics. As a young man I simply didn't understand enough of those dynamics to be able to model the when and the why of communication (meta-communication?), even if I was good at the idea transmission part. Life experience helps immensely here.
  - navane 3y ago
    Or you became "one of them", and now you too have an agenda.
    - catlifeonmars 3y ago
      No it’s not about that. I’ve seen this failure case over and over again. When you communicate, it’s not enough to explain a solution. You have to painstakingly help others internalize the problem. Once you’ve done that, and gained enough critical mass, you find that the solution almost communicates itself. It’s almost like magic when you see an organization shift its thinking like this
      - Xcelerate 3y ago
        > When you communicate, it’s not enough to explain a solution. You have to painstakingly help others internalize the problem. This is why I think startups or small groups of people are still able to outcompete bigger incumbents. So many people worry “if I share my great idea, someone will steal it and implement it!” Far more often I’ve found it to be the case that I catch myself wondering if it would be less work to start a competitor to my current company based on my idea rather than continue attempting to translate it, broadcast it, and massage it into some form that convinces a critical mass of coworkers of its value such that it eventually turns into action. At least by starting my own company, I’ll know pretty quickly whether it was actually a good idea or not.
underdeserver 3y ago
I completely agree with your sentiment, but I think whether or not you did vector arithmetic correctly at a first-year undergraduate course level is not subjective.
- deleted 3y ago
  [deleted]
jokethrowaway 3y ago
I matured a different, if not opposite viewpoint: every entity acts in it own self interest, so if the incentives are not aligned, things will tend go south. It may work for a long period of time because of sheer will power and commitment to a greater good from the people involved in the organisation, but over time, everything will follow market incentives. Public health care seems to be a primary example (and my family has been in the sector for more than 40 years); HC was good for a long period of time then it got progressively worse and nowadays it's just an item in your tax bill. If you literally don't want to die you better know someone on the inside which can push the right buttons or go private. The money spent on law enforcement had a reverse effect on crime rate, the police actually got less efficient the more money they manage to extract from public funds. They got really good at getting more money from ridiculous speed limits and automated systems though. Looking in the private tech sector my experience has been similar: the larger the organisation the more it resemble a government and the more inefficiency is tolerated. Within layers and layers of middle management it's easy to waste plenty of investors' money - albeit not indefinitely, like a government would.
coldtea 3y ago
>On the one hand, your friend could just be experiencing a streak of bad luck or a series of bad organizations. I've seen the same play out in several organizations, including the public sector. You don't even need to be cynical to know this is the case... >when I hear a (first or second hand) account that takes the form “of course I was the reasonable one in every case and these N organizations were clearly behaving like idiots”, I’ve learned to consider the possibility that the speaker is just unusually difficult. Probably based on different experiences, with different context. Because in the context described (e.g. the educational example) this is par for the course - and any number of teachers and professors can attest that if you do your work too well and apply rigid standards, you're out of there in no time, or at least make enemies very fast. >the speaker is just unusually difficult. Of course, if the baseline is complacency, profiteering, and lack of standards, anybody who stirs those waters is "unusually difficult.".
- simonh 3y ago
  The college example doesn’t seem likely though. Exams are set by central authorities and independently adjudicated. It’s not up to teachers whether students pass or not, and parents and education authorities look at exam results to judge colleges, not teacher assessments.
  - coldtea 3y ago
    Not sure if you have in mind entrance exams or something that's not set by professors (he mentions mid-terms). Or perhaps you have your local example in mind. This however is absolutely the case in my country, exam questions are set by the teachers, and are graded by them (well, often also by postgraduates given this task by the teacher so they can slack). As for the US, I did a quick search now, and found this: "Unlike final exams, which are scheduled by the university’s Registrar, midterm exams are typically scheduled during class time by the professor. Some classes may have two midterm exams, in which case they are spread further throughout the semester. Professors outline these exams in the course syllabus, so they will not come as a surprise. The weight of each midterm exam on the final grade is also usually provided in the course syllabus. Many instructors are open to telling students about the format of the midterm exam, as well as the topics or themes that the exam will cover." So looks like it's on the instructors, as the grandparent says there too.
itronitron 3y ago
In the case of the college professor and the dean, who is being the reasonable one?
whatshisface 3y ago
Two is not a large value of N for the phenomenon you're talking about. Also... is it really that hard to believe that a manager would defend their headcount and a community college would let standards slide in their hardest courses? You also have to consider the plausibility of the stories at hand before writing off the person who's complaining. I don't think your skepticism is at all warranted.

thaumasiotes 3y ago

> In some cases, it's even worse -- Institutions invent fake problems that they allege to be the solution for and then of course never solve them. Why is that worse? In scenario A, you have an institution that fights to prevent a problem from being solved. If you ignore them and their politics, you'll have the problem that they protect. In scenario B, you have an institution that fights to convince you that you have a problem. If you ignore them and their politics, you won't have the problem. But scenario A is better?

alok-g 3y ago
Consider how the reasoning proceeds if we do not take solving a problem as a Boolean. Also let's measure not just by whether the problem exists or not (or to what degree) but also by how much overheads or missed opportunity the society is having with the institutions even just existing. In the scenario A, knowingly or unknowingly, they would tend to solve the problem partially. The others around would see their contributions so far and foresee their value for the future. If others discover or perceive that the institution is not holding any promise for the future, the institution could struggle to survive. In the scenario B, the institution existing is entirely a lost opportunity for the society. Hence the scenario A is typically better (for making progress with given amount of resources).
- thaumasiotes 3y ago
  In scenario A, the problem is already solved, and the institution exists to prevent that solution from being implemented. That's the whole point of this comment thread. Surely that is not better than lying about an imaginary problem?
  - alok-g 3y ago
    >> In scenario A, the problem is already solved, and the institution exists to prevent that solution from being implemented. In my understanding, that's one way. Another may be to keep making slow progress to solving so that those around see the contributions and the relevance of the institution but not fully solve. Another way may be to show their relevance even if the problem is not getting solved by convincing that the situation would be worse without them. And so on.

jongjong 3y ago

This sounds similar to my own story in the blockchain sector. I joined a major crypto project which was under-delivering on their promises to investors. In my own time, I built a prototype which would have allowed them to meet their promises and would have appeased regulators whom I presumed were breathing down their necks. Win-win right? Perfect plan! What a genius move it was to use my skills in the most optimal way possible to wedge myself between this very cash-rich company and a large group of dissatisfied investors... with the backing of regulators of one of the wealthiest, most trusted nations on earth to add even more pressure! Surely, the company would be overjoyed, adopt my project, offer me a salary increase and a bonus package... WRONG. They tried to cancel the project. I had to quit my job in order to pursue the project outside of the company... Then they spent years gaslighting their own employees and community to keep everyone away from my project to ruin my prospects... The tech worked perfectly; that was several years ago and it's still running. Never encountered any bug or hack which is very unusual for this kind of tech. That company I used to work for essentially ended up abandoning their own project (after spending many years wasting millions of dollars on like 30 engineers). With 15x the engineering capacity, they couldn't deliver in 3 years what I and a friend built in 1 year. Now I'm demoralized from the entire tech industry and basically gave up on my career (until there is a complete political system change?). If this epic plan which was executed almost perfectly didn't work and there is no legal recourse for me due to institutional corruption, then what chance do I have in the future in such system? I will never get such opportunity again... And even if I do and it's executed perfectly, it's not going to work out in my favor because bad actors can essentially get away with everything.

x-complexity 3y ago
Can you share a link to your project? It sounds interesting.

Buttons840 3y ago

Your physicist friend solving the "technical research-level problem" should have evaluated whether it was a real problem, or a problem the company artificially created for themselves. If it was a real problem, and the company ignored his solution, then take the solution to market and become a competitor. If it was an artificial problem only found at that one company, then be very stingy about any effort you put into solving it. Give no free thought to the problem outside of work. Find some real problems to work on instead.

alok-g 3y ago
>> Your physicist friend ... should have evaluated whether it was a real problem ... I do not have the facts around this, however knowing him, I would guess that it was a real problem. >> ... and the company ignored his solution, then take the solution to market and become a competitor. As you know, this is not so straightforward in practice: * Not everyone is a entrepreneur, for whatever reason that is. * Not everyone has a brand to be able to raise money. * It may not be wise for investors to fund a project completing with an established/powerful mid-size company that owns a wide range in that space. * Some projects may require large investments and teams especially when the other side is a big company. * The big companies deploy all kind of schemes if they see a threat. * One of the threats is the existing intellectual property rights in that space that the company would already have. * As the invention was already made when the person was employed at the company, the invention as such likely already belongs to the company (as per typical employment agreements that most do not care to read), and this already forms a threat falling in the above bullet point. * The corporate politics would start at the newly started company itself becoming an internal threat. * And so on. Hopefully you do not see all above as unreal, a fake problem. :-) Not to say there aren't successes -- We hear about those so often. [Edit:] Well, someone cited a story like that right here in this thread. https://news.ycombinator.com/item?id=39500316 https://news.ycombinator.com/item?id=39500316
- Buttons840 3y ago
  Yes. I said he should do this and that. That's more forceful and judgemental than I intended. I meant it only as a philosophy to consider, but didn't know how to soften my message while keeping it succinct.

aunty_helen 3y ago

Addressing the second situation because it’s easier to bike shed about, I feel like your friend went to the dean with a problem rather than a solution. You said it yourself, the dean’s primary concern was keeping the school running, which in contrast to letting some loser kids pass a physics class seems very reasonable. It’s also the reason why trusting straight-A students is using a flawed metric, there are A students and then there are students with A grades. We have more complex inspection processes to deal with this. A better solution (full bike shed mode now), would be to help address the deficiencies in the class at an earlier level. Talk with previous teachers, get them feedback about what it is that kids are missing when they get to his class. Help them address the root cause of the problem. “Dean, I’ve found X, fed it back to teachers Y and began to see a yoy improvement in the pass rate of class Z.” Beautiful.

alok-g 3y ago
I do not have all the facts of the situation at hand. However, I have no reason to believe that my friend would not have done all that. I do know for a fact that he is actively trying to teach the students those pre-requisite concepts himself. The dean should have talked about digging into the problem as needed (my friend also could have been doing something wrong; knowing him, I know he would have readily accepted his mistakes if that were the case) and help solve, instead of forcing him to pass the students. I do recognize that this story is not a good example of the Shirky principle. An educational institute is likely genuinely trying to solve the problem, which is to educate the students. That problem continues because fresh students keep coming in, not because the institute itself is preventing the solution. An educational institution cannot be expected to pursue inventing a solution that disrupts the education sector altogether.

datavirtue 3y ago

Yep, as an engineer I always put on my business cap before opening my mouth or setting priorities. The systems' inertia will steamroll an individual without thinking. I have end-users that I think about every day when building my products, but then I have my customers that are ultimately prioritized.

specialist 3y ago

> ...found an innovative solution that was too simple and just worked. Yup. For years, the big boss exalted us to find the next generation solution. Mr big boss was original programmer, founder, and majority owner. We had an Illustrator/CAD style program (ScenicSoft's Preps) for designing print production plans. My office was right next to big boss, to facilitate my efforts. I kept him in the loop on my progress. It took a while, but I did exactly that. Reduced the majority of the design (image positioning) work to a simple form. Two views; fields on left and live preview on the right. What could be easier? (These kinds of things almost write themselves once you find the correct mental model, which is typically the really hard part.) I started demoing my solution to my peers and SMEs. Wow, Bravo, Amazing. Then I demo'd for big boss. Big boss said nothing. Walked out. Never spoke to or even acknowledged me again. Ghosted. -- I've been noodling on a novel SQL JDBC wrapper thing. For ages now. Input stock SQL and output typesafe wrappers. I use it for all my own work. It's stupid simple and easy. Every single person whose seen a demo just doesn't "get it". They expect an ORM or a template or a fluent API. The reactions have really unsettled me. From my prior innovations, I'm used to pushback, debate, rancor, battle lines. I just don't know what to do with blank befuddlement. (Yes, I'm slowly working towards a FOSS release, as able.)

armchairhacker 3y ago
> I've been noodling on a novel SQL JDBC wrapper thing. For ages now. Input stock SQL and output typesafe wrappers. I use it for all my own work. It's stupid simple and easy. Just wanted to say this seems like how I imagine using SQL. It’s well-known that ORMs are a bad abstraction. Although given that it’s well-known, there’s already a lot of alternate solutions, so I’m interested what yours is and why people may be “befuddled”.
- specialist 3y ago
  Converts normal SQL DML statements into prepared statements and type-safe wrappers. For Java, that means subclassing PreparedStatement and ResultSet. No templates, maps, annotations, DSL. Just normal SQL. Apologies, a few weeks ago, I started yak shaving while implementing the build stuff (eg maven plugin). I'll try to focus, ship something beta sooner than later.

lippihom 3y ago

Have a few engineering professor friends that have said the same thing regarding have to pass a certain % of their students, regardless of if they know the material or not. Generally it seems like colleges are just kicking this lack of learning / understanding down the line to prospective employers to vet.

dustingetz 3y ago

tell him the world’s eigenstates are discrete and aligned to vectors that maximize local cashflow

silexia 3y ago

"I surmise that if the Shirky Principle were to stop acting, say 80% of projects/institutions may just go away and should." This describes almost all government institutions.

phendrenad2 3y ago

> Institutions invent fake problems that they allege to be the solution for and then of course never solve them I think lost companies that fail to find product/market fit go through this phase before they collapse. They try to live on pure willpower and reality-shifting.

pas 3y ago
Or when the market shifts and they survive on inertia, for a while.

a_gnostic 3y ago

> In some cases, it's even worse -- Institutions invent fake problems that they allege to be the solution for and then of course never solve them. Plenty of corporate and government institutions stand ready to "solve" those problems…

matttproud 3y ago

Small question: I recall there being a named term for organizations that outlive their usefulness and sometimes become parasitic on their original mission. Is this the same as the Shirky Principle? I was introduced to this term many years ago, so I think it's older than Shirky. Any idea what this might have been?

amenhotep 3y ago

> In some cases, it's even worse -- Institutions invent fake problems that they allege to be the solution for and then of course never solve them. Rather frequently, and worst of all, I think, when this isn't done by an institution but by the symbiotic organism that arises between a media interest and a political party.