10 ms·
Enforcement by Software
- kfarr 5y agoMoney quote: > People seem to think that if the software requires some document, then surely the rules require the document in question. That is, human beings believe that the software must be an accurate embodiment of the law.
- bob1029 5y ago> Of course, you can achieve the same result with bureaucracy. You just change the forms and the rules. But it takes much effort. Changing software is comparatively easier. There is no need to document the change very much. There is no need to train the staff. Herein lies the central value proposition of our B2B product stack. With enough validations and required inputs throughout, it is possible to put very complex business processes 100% on-rails. The value to our customers come from the fact that they can basically hire people directly out of high school and have them accurately manage complex financial transactions for customers on the front line while having virtually zero domain knowledge. If you are stuck wondering "well how far could we take this?", look to semiconductor manufacturing. I've worked in facilities that are nearly 100% automated. If we can make an entire building that keeps people from doing the wrong thing, there isn't a whole lot that is off the table. I do see dark patterns that can emerge from this style of software, but at the same time it is impossible to argue with the standardization of outcomes you can achieve if you are willing to go to the ends of the earth in detailing your particular theme park rides. The biggest complaint is usually from power users, but they typically get to retain some access to more powerful legacy tools anyways.
- cwillu 5y agoReminds me of the retail management system from the story https://marshallbrain.com/manna https://marshallbrain.com/manna
- morelisp 5y agoBroader ethical issues of deskilling aside, it makes no sense to do this for university professors (virtually the definition of an extensively-trained professional) creating course material (a key task they were hired to do).
- autoexec 5y agoAll the variations of "Click yes if you are over 18" were probably enough to get most people used to the idea that enforcement by software is a joke and that lying to software in order to get what you want is easy and profitable.
- teddyh 5y ago“Computer says no.”
- smoyer 5y agoIf any here are unaware of the origin of this phrase, one of my former bosses worked in the U.K. and said this anytime someone asked for something unreasonable. He was also nice enough to introduce me to Little Britain - here's one to get you started: https://www.youtube.com/watch?v=qNDS4kVwA68 https://www.youtube.com/watch?v=qNDS4kVwA68
- Hokusai 5y ago"Learned helplessness is behavior exhibited by a subject after enduring repeated aversive stimuli beyond their control." The key part is beyond their control. As a society we are being trained to see software beyond our control. "The algorithm" decides. Many non software engineers fail to see that behind each software decision there is a company that can change it. Learned helplessness is dangerous even just because improvement becomes impossible when people just accepts software like if it was the physical laws of the universe.
- sumtechguy 5y agoIt is amazing how this sort of thing sticks too. One cell company I used to work for has terrible GUI's. I mean they are actually infuriating to use. It has been that way for at least a decade. I was reminded of how bad they were yesterday when I had to use that GUI again to fiddle something in my account. I told my wife 'no wonder the line is 20 people deep at the stores no one wants to mess with it'. Does no one do usability testing anymore? Sit down and make someone use the software. Not 'do x and y happens, call it a day' I mean end to end 'hey here is a task figure out how to do it' then sit on your hands and take notes while they try to do it? Bad interface design makes people feel powerless. That powerless feeling makes people give up and just accept that 'computers are complex'. Then once you are in that mode little changes become a huge deal.
- nerdponx 5y ago> Does no one do usability testing anymore? Sit down and make someone use the software. Not 'do x and y happens, call it a day' I mean end to end 'hey here is a task figure out how to do it' then sit on your hands and take notes while they try to do it? Bad interface design makes people feel powerless. That powerless feeling makes people give up and just accept that 'computers are complex'. Then once you are in that mode little changes become a huge deal. The company probably sees usability testing as an unnecessary expense that doesn't improve their bottom line.
- j-pb 5y agoThis reminds me of some of Startrek's greatest quotes: There can be no justice so long as law is absolute. Even life itself is an excercise in exceptions. - Jean Luc Picard and When has justice ever been as simple as a rule-book? - Riker
- silon42 5y agoI love the TOS episodes where Kirk turns off the central AI.
- skohan 5y agoThis is a concept I think about often in an engineering management context, and I think it applies not only to software, but to processes in general. I think there is often an impulse to systematize workflows, because it makes work quantifiable, and it helps us to justify to others that we're working in the right way. In other words, if you fail without a system in place, it's your fault for not planning enough. If you fail with a system in place, there's always something outside of yourself to blame it on (as long as you were following the agreed upon process). But sometimes rigid processes create their own problems. At various points you have to decide if something goes in column A or column B, either of which conclusion can have knock-on effects in terms of how it has to be handled by the system. Sometimes this creates problems down the line, because the system wasn't designed to handle something that isn't really A or B. Or else you end up adding columns C, D, E... to handle all these unfit cases until you have a system which is so complex and cumbersome it ceases to be useful, or you are spending all your time dealing with the system rather than solving the actual problem or creating value. In the absence of a process, it's usually up to common sense and human discretion to solve a problem. People can see what the situation is, what the goals are, and decide the best course of action to take. And the best thing about that is, human beings can take things into account like empathy or external factors, or creative opportunities which a system could never be designed to account for. That's why it's a really good thing to find places within the system to create space for people to figure things out for themselves. So one problem with systems and processes designed around using software is that it can make it difficult to include that human element. If the process is not designed correctly, it may be impossible to progress unless the software is satisfied.
- klodolph 5y ago> Surprisingly, it appears that no other department has been routing around the damage that is this new tool. I recommend a different strategy when you encounter annoying processes like this—insist upon following these processes, to the letter, and going through appropriate channels until the system collapses under its own weight. For example—a place I used to work had an internet filter that was too restrictive. Rather than admit that the internet filter was poorly designed and never should have been put in place, the bureaucracy provided us with a form to fill out to unblock web sites when we were blocked from visiting something necessary for work. You could obviously try to work around the filter, and some did. If they were blocked, they tried a different site or filled out the form, and just tried to get work done the best they could. This was, in my opinion, a mistake. Fight bureaucracy with bureaucracy. People started filling out forms every single time they tried to access a website for work that was blocked. Not just once per site, but every time they wanted to access a site they’d fill out the form. The IT department got a thick stack of forms to sort through every single day, and the entire web filter was scrapped not too long after. “Silently routing around bureaucracy” is a good plan if what you’re doing has some real importance to it. But if you want to solve bureaucracy long-term, make it painful for everyone. The other danger with routing around bureaucracy is that when something goes wrong, you end up dangerously exposed because you can’t CYA properly if you’re routing around bureaucracy. If you route around bureaucracy and shit goes south, bureaucracy wins and you take the blame.
- voiper1 5y agoSounds like malicious compliance. But electronic forms can be easily ignored. If so, I'd follow up with printing it out and handing it to the person responsible for handling it. If it's not handled immediately, then also deliver it to their boss. And so on. It has to actually be painful to the bureaucracy so they want to change it. Applied to this example of course changes, you could do something simple like changing the title or intro and then physically going to each other person and telling them they need to approve the change. Then innocently change it again right after approval. ("What? It's a WIP! I have a responsibility to make updates to teach the course in the best way.") Bonus points to get other co-workers in on it so you're not the problem-person.
- etothepii 5y ago"There is no negotiating with a computer." --Simon Biggs. At its core I think it's the main reason executives are so bad with technology. Nothing else in life is like a computer. You can not argue, you can not infuriate. It is pure logic. Unfortunately those of us who know about computers know that the logic is almost always flawed.
- 3pt14159 5y agoIt cuts both ways. The very system that forces people to use four digit years and unambiguous date formats[0] is the same system that leads us all into one giant Year 2038 problem. I mostly agree about executives, though that's fading. The world got the message that being good at technology is something effective people increasingly need to know. I'm an expert in Excel (Solver, UDF, arrays, custom formats, etc) and even I've picked up some tricks from CEOs. [0] What fresh hell is the date "12/11/10"?
- pjc50 5y agohttps://twitter.com/swiftonsecurity/status/1385565737167724545?lang=en-GB https://twitter.com/swiftonsecurity/status/13855657371677245... "A COMPUTER CAN NEVER BE HELD ACCOUNTABLE THEREFORE A COMPUTER MUST NEVER MAKE A MANAGEMENT DECISION" (IBM slide, 1979)
- moritonal 5y agoI generally believe that all business logic such as approvals and what-not should be implemented in the user-land portion of an engine. If it's hardcoded that x-process should be followed, then you've now attached the price of a developer to changing that process. I'm getting to the point where when friends ask me for a project, I implement it in something like Node-Red (a "not-much-code" solution) then give that to them as a dockerfile. I know it's not perfect, but at least they "could" change it with just a mouse.
- WhyNotHugo 5y agoThere's a lot of this in Argentina (we call it "Legislation through it system"). For example, for many years it was legal to invoice in foreign currency, but you MUST inform the tax agency of all invoiced emitted in real time. Their API required that you specify the currency, and only the value for local currency was accepted -- nothing else. So while it was technically legal to do something, the IT system would block you from doing it.
- WaxedChewbacca 5y ago
- undecisive 5y agoI've had my first run-in with the UK's test-and-trace system yesterday. I did an LFT on Tuesday, and registered it - but hit the wrong box, said I was positive, clicked send, panicked, didn't notice that there was a link to fix it, tried to use the back button, etc. Long story short, I ended up registering another LFT just so that I could mark myself as non-contagious - but that wasn't enough for the Test and Trace system. The next day I was hounded by text messages, and then called by an operative at the end of the day. I told them the mistake, and they told me there's nothing their system could do - I needed to call 119 (they didn't tell me which phone system options I needed - It's a horrendous phone system) and get them to mark it as a mistake. But before I abruptly ended the call with this operative, she told me that until I do this, I'm not allowed out of my house because my phone number was being monitored. I was not polite about the end of that call (I was not rude, mind - not her fault they she has a script to follow - but I was a bit angry) And once I finaly got bounced around 119 and found the right person to talk to, she said: "Well, self isolation rules end tomorrow - so don't worry about it". I picked the right time to make this mistake, apparently. But yeah, software has far too much control and extra-legal real-life consequences.
- dijonman2 5y agoWhat an awful system! What happens if you refuse to participate? I would give up my phone and go back to landlines.
- _dain_ 5y ago>What happens if you refuse to participate? Luckily, nothing.
- undecisive 5y agoYeah. So given the choice, I'd never register a single one. Unfortunately, the agency that runs the scheme won't allow me to visit my bed-bound mother without "evidence" of a clear test, despite the fact that the system only checks that a new test id is given, not whether you actually used the test, or what the result was (you're expected to be truthful on that form - you're not allowed to just be truthful when you enter the building). Which completely defeats the point.
- smoyer 5y ago"I for one welcome our new robot overlords" [0] More seriously, does this article mean that the societal decline caused by misinformation is stronger because it's showing up on a computer screen (e.g. Facebook?) 0. https://www.youtube.com/watch?v=OGxdgNJ_lZM https://www.youtube.com/watch?v=OGxdgNJ_lZM
- bsedlm 5y agowhat about enforcement's softer sibling, herding? I think most institutions are doing this knowing full well how it works and with a clear intention of where they want us to end up I recall reading about how google would arrange the food in the cafeteria to softly nudge people towards healthy food, hence I assume that when they make software that everybody uses it they carefully and intentionally consider which things to make convenient and which to make pain in the ass. That way they can always plausibly claim to not have forbidden anything you're just being lazy by not dealing with all the 'incidental' inconvenience. In then end, if you wanna do something which they don't want you to do then you must really really want to do it for it'll be a pain in the butt to do it. On the other hand, if it's something they want you to do then click and done. I wish I could make more people see why this makes it important that software (and technology in general) to be free and open source, because the alternative of closed and proprietary software systems (which by being inscrutable) works as tools for nudging, restraining, limiting, and in more extreme scenarios, enslaving people who must use this systems without making them.
- userbinator 5y agoAnd there lies the scary point. People are easily manipulated by computing. I've seen "tools turning people into tools" happen with developers too, and I think that's even scarier --- when it happens to the people who should know more about how computers work than anyone else, it is really disturbing. I suppose some of it might have to do with the general erosion of freedoms and individualism that's happening to society in general; it seems governments and corporations have slowly discouraged thinking for oneself and rewarded docile obedience.
- ryandrake 5y agoReminds me of an article by Venkatesh Rao, who points out we're moving toward a world where you are either above the API or below the API. You either tell robots what to do, or are told by robots what to do. 1: https://www.ribbonfarm.com/2017/08/17/the-premium-mediocre-life-of-maya-millennial/ https://www.ribbonfarm.com/2017/08/17/the-premium-mediocre-l...
- RajT88 5y agoThe flip side of this is when software intentionally does not implement adherence to the rules in play. As an example, in what I call the "License Compliance Shakedown", the DRM software used does not prevent usage outside the license usage terms. A year, two, three go by, and an audit is conducted and "true up" fees are demanded at threat of lawsuit. Flexera pioneered this model, and works with their customers to adopt it. I would say it is a small but not insignificant part of what is pushing companies towards open source.
- drewg123 5y agoAt a previous employer, we had an expense reporting system that required that every expense have an image of a receipt. So lots of people would loose receipts, and would then just include a photo of a napkin with "I lost the receipt" scrawled on it..
- 908B64B197 5y ago> However, the staff that built our course revision system decided to make it so that every single change should go through all layers of approvals. So if I want to change the title of an assignment, according to this tool, I need the department to approve. I have to wonder if the rule wasn’t put in place a long time ago after someone (tenured) tried sneaking in something truly controversial in the curriculum he planned to teach. Dr. Lemire doesn’t teach a controversial subject -at least having an opinion on vectorized instruction sets won’t get you on the news- and sounds like the type of researcher that values teaching a lot so I assume he frequently updates his course material and must be annoyed by this. Keeping in mind that recently a university close by (lower caliber but still) fired a professor for saying the N-Word in a classroom as part of a required reading during a race and literature class [0]. Maybe it’s the king of backlash they want to avoid. > An actual policeman can be intimidating. However, it is a human being. If they ask something that does not make sense, you are likely to question them. You are also maybe more likely to think that a policeman could be mistaken. Software is like a deaf policeman. And people want software to be correct. It’s ironic because the Province he’s writing this from had policeman enforcing a mandatory curfew (because of Covid of course) and would slap fines on people being outside after the designated hours. No matter if they were simply walking alone or their dog. To anyone with high school level biology that should sound a little ridiculous. Walking alone you can’t really transmit or catch an airborne disease. Interacting with a policeman sure. > I think that there is great danger in some of the recent "digit ID" initiatives that various governments are pushing. Suppose, for example, that your driver's license is on your mobile phone. It seems reasonable, at first, for the government to be able to activate it and deactivate it remotely. You no longer need to go to a government office to get your new driver's license. However, it now makes it possible for a civil servant to decide that you cannot drive your car on Tuesdays. They do not need a new law, they do not need your consent, they can just switch a flag inside a server. Again, the author’s country just demonstrated the slippery slope by arbitrarily freezing assets and bank accounts on suspicion of being friendly to a group of (first amendment protected) protesters. Instead of following the legal pathway of demonstrating a reasonable reason to a (non-political) judge to get an account frozen, the government just gave themselves emergency powers to do as they please. No oversight. No reasonable doubt? Doesn’t matter. What’s next, someone has a black sounding name and too much money in his account? Freeze it, most likely crime money. [0] https://globalnews.ca/news/8450947/quebec-university-academic-freedom/ https://globalnews.ca/news/8450947/quebec-university-academi...