11 ms·

The overarching message is that builders should deeply consider the impact of what they're building on civilization. "Technology is never neutral, because it t

• Xenoamorphous 4mo ago
  > Scientists are actually preoccupied with accomplishment. So they are focused on whether they can do something. They never stop to ask if they should do something. From Crichton's book Jurassic Park, which like most of his books is about the perils of technological advancements. They used the quote in the movie, slightly tweaked.
  • throw4847285 4mo ago
    I would replace the word scientists with engineers in that quote. People often conflate the two, but in my experience, scientists tend to be more cautious and there are built in checks and balances in the process (however flawed). Engineers/technologists tend to have no such guardrails, and are also usually embedded into entirely profit motivated environments, whatever their own values might be.
    • velcrovan 4mo ago
      This sounds like you don’t have much exposure to actual professional engineering disciplines. I’m sure civil, electrical, structural and mechanical PEs would be quite surprised to hear there are no guardrails on their professions.
      • bombcar 4mo ago
        In my experience “engineers” and builders are often quite “conservative” and really don’t like pushing the envelope, and they often only do it under protest. The most famous example may be the perpetual war between architects and engineers/builders.
      • mulr00ney 4mo ago
        Engineers versus "engineers". I have fond memories of a boss who was an actual, licensed engineer while the rest of us were very much normal software devs. Boss was pretty chill except when someone someone suggested we should be called "engineers" rather than "developers", at which point they said "if you guys were building bridges, people would be dead." (I don't think all software needs to be built to rigorous engineering standards but man... I think about that line a lot.)
        • rectang 4mo ago
          I prefer the term "software developer" and that's what I use when I don't need the prestige of the term "software engineer". It's disadvantageous for organizations to do that with actual job titles, though. Absent US government intervention to codify the term "engineer", probably the only way out of the "engineer" trap is through further title inflation, where the developers all become "vice presidents". :)
          • mulr00ney 4mo ago
            Yeah, it's 100% the better term. We've got rules against using engineer here in Canada though several companies I've worked for have called me an engineer. Apparently Professional Engineers Ontario sometimes goes after people for calling themselves engineers but I've never heard of it actually happening, and I don't know that they have any real teeth given that the places I worked that called me an engineer were Canadian-owned. (In fact, the only place where they checked if I could use the title was the one multi-national. Go figure.)
          • jimbokun 4mo ago
            People meeting the definition of Software Engineer while having that title may be rare, but we certainly need more of them.
    • Otterly99 4mo ago
      Honestly, scientists too. I did 10 years of research after deciding that it was not worth it. There is a vast amount of research with no direct application that gets published under the assumption that more knowledge is always better, but in my experience scientists rarely question the usefulness of their research (because most of the time, they find it interesting, which is motivating enough). Granted, I am talking about harmless subjects, but there is also the dimension of resource usage that almost no scientist considers (the amount of plastics and chemicals used in biochemistry and cell biology is... concerning).
  • redsocksfan45 4mo ago
    [dead]
  • tom2026hn 4mo ago
    I worry that companies like Anthropic, which are moving toward RSI, may prioritize speed over the timely identification of irreversible risks.
    • dbalatero 4mo ago
      Did you mean AGI? I'd think whatever else they are doing, LLMs are reducing RSI...
      • moogly 4mo ago
        Maybe. I am not convinced I type fewer keystrokes being a Markdown Monkey compared to writing code largely via autocomplete. Fewer Tab keystrokes, for sure.
        • dbalatero 4mo ago
          I was mostly making a joke.
      • ascorbic 4mo ago
        Recursive self-improvement
      • FarmerPotato 4mo ago
        Yup. "I wonder how long the codebot will be combobulating over that prompt?" I'll just rest my hands for a sec..."
    • sdenton4 4mo ago
      Every time I prioritize speed over risks, I, too, end up with repetitive strain injuries.
    • jimbokun 4mo ago
      We need to force the executives and engineers working at Anthropic and other AI companies to read this encyclical and pass an exam demonstrating that they read it.
• cma 4mo ago
  I would also recommend God and Golem, Inc., A Comment on Certain Points Where Cybernetics Impinges on Religion: https://monoskop.org/images/1/1f/Wiener_Norbert_God_and_Golem_A_Comment_on_Certain_Points_where_Cybernetics_Impinges_on_Religion.pdf https://monoskop.org/images/1/1f/Wiener_Norbert_God_and_Gole... Norbert Weiner was an atheist but he talks about three areas religion is the only thing to have really examined that relate to capable AI: omniscience, omnipotence, and worship (gadget worship). It has very prescient stuff on blackbox learning/distillation, reinforcement learning/reward hacking, alignment through human feedback. His The Human Use of Human beings and Cybernetics are extremely good too and have more of a mash of the themes between Rerum Novarum and Magnifca Humanitas, and more near-term automation.
  • runlaszlorun 4mo ago
    Awesome, thx for posting. I now have my new next book to read. Been wanting to read more of the original cybernetics stuff. Would you also recommend "The Human Use of Human Beings"?
    • cma 4mo ago
      Yes, it starts slow with a lot of history of Cybernetics stuff through Leibniz and stuff (kind of prescient given chain rule -> back propagation, and control theory's relevance to optimizers). It is about twice as long as God & Golem and covers a lot of the same plus more examination of automation and human augmentation. I think either it or Cybernetics also goes into mass communication with some relevance to how social media played out. God & Golem is the most succinct and up to date though, probably a 2hr read. The book Cybernetics is a lot of math and ergodic theory stuff that went beyond me, but is the longest and you still get a lot out of it skimming over that stuff if you don't have the background for it. The last revision of it in the 50s added some of the same blackbox function copying/imitation learning/distillation stuff, reinforcement learning with reward hacking concerns, and superintelligence as genie/monkey's paw. I would read the three in reverse order of publication. He also foresaw another big area of potential existential danger, Wiener filter for guidance and control of missiles (later superceded with Kalman filter bringing the nuclear hard targets era with 15min retaliation windows) and refused to work on it or share prior work, and he also had bioweapons delivery concerns before the bioweapons treaties, publishing this open letter in The Atlantic in 1947: https://archive.ph/D7BPt https://archive.ph/D7BPt
• bem94 4mo ago
  I have only read a few passages (and some of the excellent quotes others have shared here), but I find the underlying message here so much more compelling than those found in the various "manifestos" which come out of Silicon Valley. I think reading this helps me imagine a version of the future I'd actually like to live in. A version where technology is used well (rather than preaching for abstinence from technology) and where values other than "intelligence" (in whatever guise) are on an equal footing. Even writing that makes me feel naive (and to an extent I know it is) but I think it would be inconsistent for someone who cheers for humanity's efforts to solve/chip away at "impossible" problems (like LLMs were thought to be not so long ago) to shirk from the challenge of making the world better for _everyone_.
  • bluesign 4mo ago
    The thing is why that this feels so good future is; it is a system with no constraints. A bit like Star Trek universe in Roddenberry's imagination. This kind of utopia can only be achieved with all honest actors, but in reality systems are usually designed around bad actors. Even with all morally good actors locally, there is no guarantees for external forces. Thinking it hypothetically, even with global coordination ( all good actors ) there is not a proven path that would lead us to better place from any starting point from past.
    • mapt 4mo ago
      To get from here to Roddenberry's communism, according to Roddenberry's lore, we passed through the Eugenics Wars, the Second Civil War, and then fifty years of World War Three and the 'post-atomic horror' before coming to our senses.
    • energy123 4mo ago
      It's probably more predictive to model actors as being neither good nor bad but constrained by various collective dilemmas, such as prisoners dilemma, the security spiral, tragedy of the commons, race dynamics, collective action or first mover problems, information asymmetries, the commitment problem, among others. Those are the hardest problems to solve because they're pathologies that result from the global, largely amoral structure rather than consequences of the individual exercise of morality. In the AI case, each firm is in an arms race, and nobody can slow down without effectively collapsing due to positive gross margins only being viable with a frontier model that attracts marginal demand. An appeal to morality might have an impact but more effective action would be to address the structure that the AI companies are situated in that causes this dynamic in the first place. In practice, thats going to be a global agreement to slow down, and global regulations.
• zplo68 4mo ago
  Science and Engineering is morally detached. Ask a scientist or engineer what philosophy or theology has taught us about the source of morality and their education, training and experiences havent prepared them to answer that question. This didnt matter so much in the past because their activities never had the scale it does today. For basic training in philosophy if you are mid or upper level exec whose decisions are going to effect a whole lot of people, go to open yale courses and take the intro to Philosophy classes. It will help develop your answer to the question - why do you do what you do if you are going to die anyway tomorrow.
  • slfnflctd 4mo ago
    Far too many of us (particularly younger people, but not only them) undervalue or are dismissive of philosophy. I once was like this, partly because at the time I'd been brought up - like most humans - to believe my parents' religion held all the answers philosophy might address. I quickly learned as an adult that whether you're a person of faith or not, it's not pointless at all. It's the foundation of everything. Philosophy is how you explain the deeper reasons behind why you follow whatever religion you do, or adhere more meaningfully to whatever kind of agnosticism, atheism and/or 'spirituality' (with or without woo) you espouse.
    • lo_zamoyski 4mo ago
      [dead]
  • moron4hire 4mo ago
    This can only be true if you believe in a certain mad-scientist version of these things. We as a society have systems (however flawed) to hold scientists and engineers to ethical standards: e.g. peer review and in many cases--e.g. civil engineering, architecture, medicine--even legal frameworks to enforce ethical standards. Science and engineering are human endeavors, they are not divorced from the human condition, they cannot be separated from humanity and human rights. Maybe you mean mathematics is amoral and are committing the common conflation of "engineering is just applied science and science is just applied mathematics," which is a really bad case of missing the forest for the trees.
• Xmd5a 4mo ago
  > Therefore builders "bear a particular ethical and spiritual responsibility" > This is a message we need right now. Feels good man. The solution found by the private parties driving technological change is sainthood. Or aiming for it. At least, better than you. They have the vision of what's good for the herd, but the more time I spend as a sheep, the more it seems the "herd" is just a way to recycle the story of their own exceptionalism stripped of any mark of individuality. A simple visit to fiftyyears.com will greet you with "We back the indispensable". I guess it's the same "we".
• vermilingua 4mo ago
  Not only builders, the greatest takeaway for me is that everyone has a responsibility in shaping the discourse, culture, and usage of transformative technology. This "the builders will do the right thing" mentality is even (in my interpretation) explicitly called out in several places: > It is the pursuit of the common good that gives life to a people, understood not as a mere collection of individuals, but as a living reality in which people learn to recognize that they themselves are interconnected and jointly responsible for the res publica. In this sense, every person contributes to the building up of one’s people... > When it comes to decisions regarding economic flows and digital platforms, as well as the governance of data and algorithms, we cannot allow a handful of actors to dictate these processes on their own; instead, we must build forms of cooperation that respect the various levels of the global community and make them jointly responsible for the common good. > We cannot be satisfied with merely calling for the moralization of machines — the so-called “alignment” of AI with human values — without also having the courage to insist on a further condition: the possibility of openly discussing the ethical frameworks involved and subjecting them to shared standards of social justice.... What is needed is a more active political involvement that is capable of slowing things down when everything is accelerating, and of protecting the opportunities for communities still to be able to participate and ask questions.
  • ako 4mo ago
    Agreed, I think the builders are the wrong people to ask for this self reflection. Anything can be used for good and bad. A knife can be used to prepare food, or to stab someone. A drone can be used to attack a country, or to defend a country. A wooden beam can be used to build a house, or to build a cross to crucify someone. Same way AI can be used for good and bad. And it’s up to the person using the tool to act moral. Unfortunately this world is full of people that consider power and money more important than morals. For me, religion falls in this bucket, lots of “do as I say, not do as I do”. And on top of this, there’s very little agreement on what is good, and what is bad. Morals are quite fuzzy, and flexible.
    • jimbokun 4mo ago
      You are welcome to your opinion, but human civilization was even more biased towards instrumentality before the broad spread of Christianity. I am well aware of the abuses and scandals of Christian churches throughout history. I am also aware that the Romans and other civilizations pre-dating Christianity had even less regard for the poor and powerless.
• GoToRO 4mo ago
  hard disagree. No matter what gets built, it will be used for bad things if the society is not properly constructed. We need to fix our government and then you can build anything you want. I don't like putting the blame on the builders, building is already difficult as it is. Also you can't built something in isolation, word will get out anyway. That's why this speech is either naive or he just wanted to tick a todo list: say something about AI.
  • xandrius 4mo ago
    I get where you are coming from but this is the common "reduction to politics" that anyone who doesn't want to address a problem uses: think of any societal or human problem and you can have your comment with different nouns. Sure, IF we could just go and fix our governments in some magical way then the problem would disappear. That goes from hunger, climate change, videogame addiction and AI. The problem is that what you value in life in different than what others do, so we now have a system in which sometimes you get what you want and sometimes you don't. But back to the topic, I do think that how OpenAI and Anthropic handled the government and them asking to drop guardrails is something a company can actually and actively do without having to reinvent the universe.
  • jamesinmn 4mo ago
    > I don’t like putting blame on the builders Why does it have to be either/or? From a high-level view, there are powerful cultural/political forces that nudge us towards building harmful, wasteful things without us being aware of said forces. The government, corporate greed, and billion-dollar marketing budgets are to blame. But on the ground, from an in-the-trenches perspective, I think an engineer at Meta, for example, shares in some level of blame, too. Mainly because there is plenty of evidence showing how much harm Meta products have brought upon society. Yet Meta couldn’t be built without engineers opting in to work there in exchange for $300k salaries. We desperately need moral clarity and courage at both the policy-level and the individual level.
    • bluesign 4mo ago
      Because there will always be bad actors. You cannot design a system on all actors' full cooperation. Ofc they share the blame, but it is not solving the problem. You can say for example: loan sharks are bad for society; so government gives anyone 0 percent credit. You just removed one problem, created another. Just and sustaining system with individual morality is destined to fail. Only option is social regulation. Which is at government level.
• exabrial 4mo ago
  [flagged]
• Curosinono 4mo ago
  [flagged]
  • sdsd 4mo ago
    >for the last 15 years. The scandal broke out in worldwide media starting in 2002 (not coincidentally, the same year South Park released Red Hot Catholic Love). This reminds me of when Contrapoints off-hand commented about how things were different in the 70s but that was "30 years ago".
• ThrowawayR2 4mo ago
  Not new for software and hardware industry though, practitioners have just chosen to ignore it. From the Association for Computing Machinery, which encompasses all forms of software development, the very first principle is the public good: "Software engineers shall act consistently with the public interest. In particular, software engineers shall, as appropriate: 1.01. Accept full responsibility for their own work. 1.02. Moderate the interests of the software engineer, the employer, the client and the users with the public good. 1.03. Approve software only if they have a well-founded belief that it is safe, meets specifications, passes appropriate tests, and does not diminish quality of life, diminish privacy or harm the environment. The ultimate effect of the work should be to the public good. ..." From the IEEE, which also encompasses computer engineering, their first principle and its first few sub-items are: "To uphold the highest standards of integrity, responsible behavior, and ethical conduct in professional activities. 1. to hold paramount the safety, health, and welfare of the public, to strive to comply with ethical design and sustainable development practices, to protect the privacy of others, and to disclose promptly factors that might endanger the public or the environment; 2. to improve the understanding by individuals and society of the capabilities and societal implications of conventional and emerging technologies, including intelligent systems; ..."
  • hnthrow0287345 4mo ago
    This would be more meaningful if, perhaps, we had to swear an oath to it before being able to practice. And practitioners would be treated more seriously if everyone knew we swore that oath. And the legal utility as accountability and defense would also be useful. Of course people are going to ignore it if there's no force behind it.
    • lukan 4mo ago
      It seems to work somewhat with medical doctors and the Hippocratic Oath. But I would argue it is way easier there. Building software has way more grey areas.
      • trolleski 4mo ago
• doug_durham 4mo ago
  Yes but... No one living today knows what direction AI technology will take humanity. If we have an algorithm breakthrough then we may avoid building new data centers. If the abilities of the technology plateau then there might not be large impacts on employment. Builders need to focus on the impact of their next steps. Don't put polluting natural gas generators in neighborhoods today. Don't make unemploying folks the goal of your tool, today. Don't make decisions that harm people and the environment today.
  • lelandfe 4mo ago
    https://www.theguardian.com/us-news/2026/may/13/utah-approves-datacenter-backlash https://www.theguardian.com/us-news/2026/may/13/utah-approve... > The Stratos artificial intelligence datacenter footprint will cover more than 40,000 acres (62 sq miles) over three sites in Box Elder county in north-western Utah. The facility will require about 9GW of power, which is more than the entire state of Utah currently consumes Sure hope your rosy inflection point happens real soon. Otherwise, the direction this is taking us is pretty obvious.
    • FabHK 4mo ago
      9 GW is massive. About half a percent of world electricity consumption, half of what stupid Bitcoin burns.
  • moron4hire 4mo ago
    As I keep telling my project owner at work, "the AI will get better" is not a plan.
• ncr100 4mo ago
  Cool, inspires me to think, "should we regulate away the toxic side effects which we realize COULD have come, if we built it less considerately?"
• pj_mukh 4mo ago
  Under this logic, would the Pope have built a Hammer? a Knife? I am hoping the Pope has a cleaner view of AI quietly automating the drudgery in the back offices and not just robot dogs with machine guns. And with that view, should we build it?
  • PantaloonFlames 4mo ago
    It's too simplistic to imagine the tension is between robot patrol dogs vs automating drudgery. If automating drudgery suddenly puts 30% of people out of work, it has huge broad negative impact on people who are currently alive and working in the current system. Innovate, but do it with awareness.
    • pj_mukh 4mo ago
      Than an even better question for the pope is: If a technology existed that reduced the cost of producing a critical thing (think food, housing, medical care) down to near zero, however, it made the humans currently building the thing redundant, should we build it? Would it be okay to use the hyper-optimization power of Capitalism to build such a technology faster?
      • 3form 4mo ago
        Likely the answer would be: yes, we should build it, and take appropriate care of the people being made redundant. These two are not mutually exclusive, it's just that no one wants to pay for it.
• ctoth 4mo ago
  The strange part is how moral responsibility somehow always lands on the builders... the people with the least leverage... while the funders get to ask the ethical questions. Weird!
  • erikerikson 4mo ago
    I agree that holding only the concrete implementers responsible would be inappropriate. However, I don't believe that distinction is made. One says "I am building a house" even if they are completely contracting the job out. I'd suggest the greatest responsibility lays with the funders and that the Pope would agree.
  • ThrowawayR2 4mo ago
    No, we don't have to take the funders' money; that's what having professional standards means. Nobody would excuse a doctor performing unsafe procedures because they "needed the money". Engineers were jailed for the Volkswagen emissions tampering scandal and nobody would excuse them for needing to take funders' money.
    • deleted 4mo ago
      [deleted]
  • jeremyjh 4mo ago
    The funders are among the principal builders in this context. This is addressing the people who have a say in what is built, and how it is built. Much of that belongs to the executive and ownership class, but not all of it.
  • hodgesrm 4mo ago
    Mark Zuckerberg is a builder in the context of the encyclical.
  • arter45 4mo ago
    It lands on both. "It was just a job" or "I was just following orders" doesn't excuse you from doing unethical stuff.
• theshindesahil 4mo ago
  [flagged]
• hackboyfly 4mo ago
  Thanks for this!
• devilbunny 4mo ago
  The Roman Catholic Church has a lot of things wrong with it, now and in the past, but it’s a human institution older than almost any other, and it’s composed of a lot of very intelligent people. Agree or disagree with them, but a papal encyclical is almost always worth reading and understanding.
  • ycombinete 4mo ago
    Indeed. And because of it’s age and outlook its view is very far back and very far forward. The casual reference to a 135 year old encyclical that dealt with the seismic shift of industrialisation took me quite aback for a number of reasons.
  • michaelsbradley 4mo ago
    There are so many encyclicals, apostolic letters, etc. One could spend years reading just a fraction of them, depending on reading and comprehension speed, of course, which varies by person. Two I recommend, from the last 40 years: Veritatis splendor, John Paul II, 1993 Argues that Christian freedom is fulfilled, not limited, by objective moral truth: some acts are intrinsically evil regardless of intention or circumstance, conscience must be formed by divine law rather than self-authorization, and the Church must faithfully teach this moral truth as the path to authentic human flourishing in Christ. https://www.vatican.va/content/john-paul-ii/en/encyclicals/documents/hf_jp-ii_enc_06081993_veritatis-splendor.html https://www.vatican.va/content/john-paul-ii/en/encyclicals/d... Fides et Ratio, John Paul II, 1998 Argues that faith and reason are complementary paths to truth: reason needs faith to avoid skepticism, relativism, and reductionism, while faith needs reason to express, defend, and deepen its understanding of divine revelation and the human search for meaning. https://www.vatican.va/content/john-paul-ii/en/encyclicals/documents/hf_jp-ii_enc_14091998_fides-et-ratio.html https://www.vatican.va/content/john-paul-ii/en/encyclicals/d...
    • thuridas 4mo ago
      Note that, as anybody, the Catholic Church may have is bias. For example. Their dualist view of the world makes them see AI as something very different from human intelligence. So, without having read it, the church may negate that it could be at human level. A few years ago that would negate that computers could be creative because they do not have a soul. Anyway, kudos for focussing on the important issues and the impact on human.(And less on sex)
• pickledish 4mo ago
  This is also the message I got from "the wind rises"! Though from talking with other people that takeaway doesn't seem universal -- which IMO is one of the ways to tell it's a great film :)
• kombine 4mo ago
  "Technology is never neutral, because it takes on the characteristics of those who devise, finance, regulate and use it." My comment under the post on Omarchy here got downvoted and flagged by the "technology is neutral" crowd because I dared to say that it is unethical to use software produced by a white supremacist.
• oersted 4mo ago
  I'd argue that the problem is exactly that the responsibility has been left to the builders, who have a hard enough job actually building and have little room for competing incentives. The concept of a self-regulating industry is utter nonsense. I am a builder, and I try to do things right, but if I am honest with myself I do not have the mental capacity or willpower spare, or the worldly knowledge of everyone's perspectives, to manage these massive challenges. I need other smart people to also focus on that, people that are actual representatives of the population, and yes, force me to stop and do it better occasionally. Achieving a good balance sometimes does require having opposing forces fight it out, it can be healthy.
• ngcazz 4mo ago
  Like glibly synthesizing a reimplementation of NextJS
• dubcanada 4mo ago
  I frankly disagree, how do you know "will this make humanity better?" until the product is done? AI wasn't what it is today because of a specific companies innovations. It stemmed from decades of research built on top of other research. How did any of those builders know what they were getting into? Every technology should be done to the fullest potential, how are we ever supposed to explore the stars or cure cancer if everyone is scared of accidentally building Skynet?
  • alok-g 4mo ago
    Just like we can incrementally innovate, we should be able to incrementally figure too.
• slashdave 4mo ago
  > We cannot condone naïve enthusiasms, nor fuel unfounded fears