3 ms·

> people trying to game those rules for their own advantage once they figure them out I mean, good for them. They deserve my money for obeying the letter of my

by derefr 5y ago

> people trying to game those rules for their own advantage once they figure them out I mean, good for them. They deserve my money for obeying the letter of my requirements. That's exactly what I want them to do. If the letter of the requirements don't accurately and precisely represent the spirit of the requirements, that's on me. IMHO that sort of adversarial relationship is extremely helpful at developing a good formal spec for the requirements in the first place. It's TDD, but with a monetary incentive on the part of the developer to "code to make the test pass" and nothing more. You don't get to rely on the developer's charitability—the only knob you get to turn, is what test you write! Of course, in the end, the "tests" might need to be Sufficiently Advanced for this to work, e.g. a spec that specifies something as "what a layman would understand X to be" — where actually evaluating that in an automated way would involve automating a "jury of your peers" type setup to act as the "layman."

crazygringo 5y ago
I think you're misunderstanding, so here's an example: The rules might be set up to allow 4 excuses of "building couldn't be accessed" per week before a warning, and 3 warnings in any 60-day period before firing. If you don't know this, then you simply do your job, and report when a building can't be accessed. If you do know this, then sometimes you're feeling lazy, want to go home a bit earlier and not do your job, and you lie and report the building can't be accessed, because you know you're under the threshold and so won't be fired. This isn't about requirements. It's about setting thresholds to catch people who are intentionally not performing their job.
- gjs278 5y ago
  the rules might also say that excuse only works when a system that checks the doors confirms that the building wasn’t accessible. saying I couldn’t access the building with zero proof (a picture) could lead to a review later on where the termination happens for lying. it doesn’t have to be 100% automated and even if it was, they could automate out liars eventually.
- derefr 5y ago
  That is about requirements, though. The failure in this example reflects a gap in the spec where there should have been a requirement for proof of building inaccessibility, in the form of e.g. a video of the worker trying and failing to open the door. At it happens, this exact thing is 99% of the reason Amazon requires its delivery people to take+submit a photo proof of a delivery when they mark the package as "left at door." Without that requirement, they'd be able to just toss the packages out and mark them as "left at door." Amazon doesn't show those pictures to customers to show them where the package was left — that part should be obvious. They show them to customers so the customer can drive the complaint process when the picture shows that the driver did something stupid! The fact that customers are handed this evidence, in turn disincentivizes drivers from doing something stupid with customers' packages in the first place. (This personally happened to me just last week: I ordered an Amazon package, and the driver left it at — and took a picture of — the door of a condo within a building that, per the picture, was a completely different building from the one I live in. That picture was very helpful in getting [Amazon's customer service to contact the driver to get] me my package. That was an accident on the driver's part, but it should be clear to the drivers that this is a system that would track — and automatically punish — any attempt at intentional malfeasance at this step, on the basis of this same evidence.)
  - crazygringo 5y ago
    Not everything has proof, or has proof that is cost-effective to generate. Amazon requiring its workers to photograph packages is a great idea, and one I applaud them for. But most instances of employees running into difficulties don't have such elegant solutions. Unless you want employees to be wearing bodycams their whole shift... and then you're going to have to pay other people to review the footage which can take serious time...
    - derefr 5y ago
      > Unless you want employees to be wearing bodycams their whole shift... and then you're going to have to pay other people to review the footage which can take serious time... Proof can (and should) be positive, rather than adversarial. You could have a bodycam that's recording for the whole shift, but — like in the case of the Amazon driver's photograph — the onus is on the worker to (hopefully with app assistance) trim their bodycam footage down to a short clip that proves the building was inaccessible, and then submit that clip as part of their report. If they choose "building was inaccessible" as an option, then this clip would be a required field on the form; they can't "complete" their job without it. Yes, there's a bit of video evidence to review. But it's very small amounts of evidence at a time, and evidence only generated in an edge-case. (At least, at game-theoretic equilibrium. There might be more evidence generated at first as workers test the water to see what they can get away with.)
raxxorrax 5y ago
It is not TDD, it is bad management.
- derefr 5y ago
  You'd be right if we were talking about an iterated prisoners' dilemma (i.e. hiring the same contractors over and over.) It'd be bad management to be in an adversarial relationship with people you maintain a consistent relationship with. But the point of Mechanical Turk is to gain a scaling advantage by designing your task as a non-iterated prisoners' dilemma (i.e. so that arbitrary new untrained people can come in and do one of your tasks once, and then not necessarily ever interact with you / your tasks again.) This requires a very different mindset; one that not many people possess. Even most of the people who are using Mechanical Turk "successfully" don't design their tasks to actually be reliably accomplished by the firehose of arbitrary new untrained people that come through the service. Most employers on the service just use it as a temporary-worker recruiting platform — using work-sample test tasks to cull down to a set of reliable workers, and then feeding real tasks to only that pool of pre-tested workers. But this throws away the key advantage of Mechanical Turk — the scale and elasticity to demand you get by being able to rely on "literally any human" to complete your tasks, rather than needing to be filter them down further and thus expose yourself to winnowing (which is especially harsh in the MTurk world — no worker on the service stays with it for very long, as it's one of the least marginally profitable things you can do with your time, only worth it if you have literally no better options. And most people find another option fairly quickly.) It's not every task that can take advantage of "literally any human" — but the workloads that can are far more numerous than these tasks' designers think. They just don't understand how to create specs that stand up to an adversarial implementation. Designing tasks for MTurk is, essentially, writing a wish for a malevolent genie, such that you can automatically verify whether you got what you wished for, and such that you can prove to the malevolent genie that that verification step is what's going to happen, such that they have no reason to defect. It's the baby brother to the AI control problem. A good example of a well-designed MTurk task (if it were implemented as one) would be Google's older implementation of reCaptcha, the one that shows the "worker" text from books / street-view street-facing signage, and gets them to write what they see. The whole "present one known and one unknown" and "cross check inputs across several workers" parts of the design are there to allow the system to work in a non-iterated prisoners' dilemma context.

by derefr 5y ago

crazygringo 5y ago

I think you're misunderstanding, so here's an example: The rules might be set up to allow 4 excuses of "building couldn't be accessed" per week before a warning, and 3 warnings in any 60-day period before firing. If you don't know this, then you simply do your job, and report when a building can't be accessed. If you do know this, then sometimes you're feeling lazy, want to go home a bit earlier and not do your job, and you lie and report the building can't be accessed, because you know you're under the threshold and so won't be fired. This isn't about requirements. It's about setting thresholds to catch people who are intentionally not performing their job.

gjs278 5y ago
the rules might also say that excuse only works when a system that checks the doors confirms that the building wasn’t accessible. saying I couldn’t access the building with zero proof (a picture) could lead to a review later on where the termination happens for lying. it doesn’t have to be 100% automated and even if it was, they could automate out liars eventually.
derefr 5y ago
That is about requirements, though. The failure in this example reflects a gap in the spec where there should have been a requirement for proof of building inaccessibility, in the form of e.g. a video of the worker trying and failing to open the door. At it happens, this exact thing is 99% of the reason Amazon requires its delivery people to take+submit a photo proof of a delivery when they mark the package as "left at door." Without that requirement, they'd be able to just toss the packages out and mark them as "left at door." Amazon doesn't show those pictures to customers to show them where the package was left — that part should be obvious. They show them to customers so the customer can drive the complaint process when the picture shows that the driver did something stupid! The fact that customers are handed this evidence, in turn disincentivizes drivers from doing something stupid with customers' packages in the first place. (This personally happened to me just last week: I ordered an Amazon package, and the driver left it at — and took a picture of — the door of a condo within a building that, per the picture, was a completely different building from the one I live in. That picture was very helpful in getting [Amazon's customer service to contact the driver to get] me my package. That was an accident on the driver's part, but it should be clear to the drivers that this is a system that would track — and automatically punish — any attempt at intentional malfeasance at this step, on the basis of this same evidence.)
- crazygringo 5y ago
  Not everything has proof, or has proof that is cost-effective to generate. Amazon requiring its workers to photograph packages is a great idea, and one I applaud them for. But most instances of employees running into difficulties don't have such elegant solutions. Unless you want employees to be wearing bodycams their whole shift... and then you're going to have to pay other people to review the footage which can take serious time...
  - derefr 5y ago
    > Unless you want employees to be wearing bodycams their whole shift... and then you're going to have to pay other people to review the footage which can take serious time... Proof can (and should) be positive, rather than adversarial. You could have a bodycam that's recording for the whole shift, but — like in the case of the Amazon driver's photograph — the onus is on the worker to (hopefully with app assistance) trim their bodycam footage down to a short clip that proves the building was inaccessible, and then submit that clip as part of their report. If they choose "building was inaccessible" as an option, then this clip would be a required field on the form; they can't "complete" their job without it. Yes, there's a bit of video evidence to review. But it's very small amounts of evidence at a time, and evidence only generated in an edge-case. (At least, at game-theoretic equilibrium. There might be more evidence generated at first as workers test the water to see what they can get away with.)

raxxorrax 5y ago

It is not TDD, it is bad management.

derefr 5y ago
You'd be right if we were talking about an iterated prisoners' dilemma (i.e. hiring the same contractors over and over.) It'd be bad management to be in an adversarial relationship with people you maintain a consistent relationship with. But the point of Mechanical Turk is to gain a scaling advantage by designing your task as a non-iterated prisoners' dilemma (i.e. so that arbitrary new untrained people can come in and do one of your tasks once, and then not necessarily ever interact with you / your tasks again.) This requires a very different mindset; one that not many people possess. Even most of the people who are using Mechanical Turk "successfully" don't design their tasks to actually be reliably accomplished by the firehose of arbitrary new untrained people that come through the service. Most employers on the service just use it as a temporary-worker recruiting platform — using work-sample test tasks to cull down to a set of reliable workers, and then feeding real tasks to only that pool of pre-tested workers. But this throws away the key advantage of Mechanical Turk — the scale and elasticity to demand you get by being able to rely on "literally any human" to complete your tasks, rather than needing to be filter them down further and thus expose yourself to winnowing (which is especially harsh in the MTurk world — no worker on the service stays with it for very long, as it's one of the least marginally profitable things you can do with your time, only worth it if you have literally no better options. And most people find another option fairly quickly.) It's not every task that can take advantage of "literally any human" — but the workloads that can are far more numerous than these tasks' designers think. They just don't understand how to create specs that stand up to an adversarial implementation. Designing tasks for MTurk is, essentially, writing a wish for a malevolent genie, such that you can automatically verify whether you got what you wished for, and such that you can prove to the malevolent genie that that verification step is what's going to happen, such that they have no reason to defect. It's the baby brother to the AI control problem. A good example of a well-designed MTurk task (if it were implemented as one) would be Google's older implementation of reCaptcha, the one that shows the "worker" text from books / street-view street-facing signage, and gets them to write what they see. The whole "present one known and one unknown" and "cross check inputs across several workers" parts of the design are there to allow the system to work in a non-iterated prisoners' dilemma context.