4 ms·

A better way to buy code. Use tests to buy code

3 points by cashmonkey85 11y ago

ColinWright 11y ago
This is a lovely idea, but I'd like to share with readers an experience I had a while ago. I was looking to outsource some coding, so I advertised and chose someone that seemed reasonable. I asked for some sample code doing similar things to what I wanted, they responded with some snippets that looked right, so we entered an agreement. I would provide some sample input and output, they would provide code that produced the right results. Once I verified that it did what was required, I would pay. It was a problem domain that wasn't overly complex, and I thought it would be an ideal task to outsource. It was a lot more complex that the example I'll give here, but I want to give you a flavour of how it all went wrong. This is not the task, just something to explain how it went wrong. I asked for code that would return "True" if the 2nd and 4th characters in a supplied string were the same, but otherwise all characters were different. So here are some examples of input and output: severity True vagabonds True unknowable True unsnarled True telegraphs True robot True faunas False flimsily False freighting False rummaging False mosaic False shushes False conflicting False Remember, the actual task was described in full, was much more complex than this, and we were simply providing some example test cases. What we got back was the equivalent of this: def test_string(x): if x == 'severity' return True if x == 'vagabonds' return True if x == 'unknowable' return True if x == 'unsnarled' return True if x == 'telegraphs' return True if x == 'robot' return True if x == 'faunas' return False if x == 'flimsily' return False if x == 'freighting' return False if x == 'rummaging' return False if x == 'mosaic' return False if x == 'shushes' return False if x == 'conflicting' return False So to anyone who tries to specify their coding tasks by using tests - beware. There are coders who will strive to meet the exact letter of the requirements, and then demand payment. For what it's worth, we never resolved this and eventually paid an agreed amount to make them go away. We took this as a reasonably expensive lesson about some of the potential pitfalls of out-sourcing.
- cashmonkey85 11y ago
  Hey, thanks for the example. This is definitely something we want to avoid with this platform and have thought of ways to mitigate it. One way is we will allow the buyer to further test submissions as black boxes to see if the output matches their expectations. The second way is with a rating system as extra information for the buyer and an incentive for the coders to write good code.
  - ColinWright 11y ago
    How do you avoid this scenario: * Customer creates job spec * Customer creates white examples * Customer creates black examples * Coder bids * Customer accepts * Customer sends spec * Customer sends white examples * Coder delivers code * Code fails black examples * Customer refuses to pay * Coder claims code meets spec ??
    - codeswap 11y ago
      The platform is weighted in favour of the buyers. What I mean by that is the coders compete to win the payout. The buyer doesn't see or own the source code until they pay. Many coders can submit solutions and the buyer can pick the best one to purchase. It's up to the buyer to query the submission fully as a black box and to trust the rating system.

3 points by cashmonkey85 11y ago

This is a lovely idea, but I'd like to share with readers an experience I had a while ago. I was looking to outsource some coding, so I advertised and chose someone that seemed reasonable. I asked for some sample code doing similar things to what I wanted, they responded with some snippets that looked right, so we entered an agreement. I would provide some sample input and output, they would provide code that produced the right results. Once I verified that it did what was required, I would pay. It was a problem domain that wasn't overly complex, and I thought it would be an ideal task to outsource. It was a lot more complex that the example I'll give here, but I want to give you a flavour of how it all went wrong. This is not the task, just something to explain how it went wrong. I asked for code that would return "True" if the 2nd and 4th characters in a supplied string were the same, but otherwise all characters were different. So here are some examples of input and output: severity True vagabonds True unknowable True unsnarled True telegraphs True robot True faunas False flimsily False freighting False rummaging False mosaic False shushes False conflicting False Remember, the actual task was described in full, was much more complex than this, and we were simply providing some example test cases. What we got back was the equivalent of this: def test_string(x): if x == 'severity' return True if x == 'vagabonds' return True if x == 'unknowable' return True if x == 'unsnarled' return True if x == 'telegraphs' return True if x == 'robot' return True if x == 'faunas' return False if x == 'flimsily' return False if x == 'freighting' return False if x == 'rummaging' return False if x == 'mosaic' return False if x == 'shushes' return False if x == 'conflicting' return False So to anyone who tries to specify their coding tasks by using tests - beware. There are coders who will strive to meet the exact letter of the requirements, and then demand payment. For what it's worth, we never resolved this and eventually paid an agreed amount to make them go away. We took this as a reasonably expensive lesson about some of the potential pitfalls of out-sourcing.

cashmonkey85 11y ago
Hey, thanks for the example. This is definitely something we want to avoid with this platform and have thought of ways to mitigate it. One way is we will allow the buyer to further test submissions as black boxes to see if the output matches their expectations. The second way is with a rating system as extra information for the buyer and an incentive for the coders to write good code.
- ColinWright 11y ago
  How do you avoid this scenario: * Customer creates job spec * Customer creates white examples * Customer creates black examples * Coder bids * Customer accepts * Customer sends spec * Customer sends white examples * Coder delivers code * Code fails black examples * Customer refuses to pay * Coder claims code meets spec ??
  - codeswap 11y ago
    The platform is weighted in favour of the buyers. What I mean by that is the coders compete to win the payout. The buyer doesn't see or own the source code until they pay. Many coders can submit solutions and the buyer can pick the best one to purchase. It's up to the buyer to query the submission fully as a black box and to trust the rating system.