5 ms·
Before you ask, because I was curious, from the paper: "For MTurk, we aimed to select the best available crowd-workers, notably by filtering for workers who ar
by PostOnce 4y ago
Before you ask, because I was curious, from the paper:
"For MTurk, we aimed to select the best available crowd-workers, notably by filtering for workers who are classified as “MTurk Masters” by Amazon, who have an approval rate of over 90%, and who are located in the US."
- swyx 4y agohonestly i wonder if @dang will approve an auto summarizer bot on HN since it helps improves the quality of discussions. finetune on HN comments, anticipate the top few questions, and then answer from the source doc
- theamk 4y agoI hope not, as such bot will inevitably introduce factual errors every once in a while. When people don't read articles others can see that and correct misconceptions. Officialy approved bots will not be able to receive or act on this feedback.
- codetrotter 4y agoDo like the mods in some subreddits have done. 1. The comment with the summary is made like any other comment. (In the case of Reddit, that’s perhaps mainly because the mods have no other option, but the point still stands.) Because of this, other users can downvote the bot comment if it is incorrect, as well as respond to it with corrections. Exactly the same way as you’d interact with a person in the past if their summary comment was incorrect; downvote them and/or correct them. 2. Include a disclaimer stating that the summary may be inaccurate. Encourage people to correct and/or downvote bad summaries in the disclaimer
- 93po 4y agoi think this would better as a browser extension for those who want it, the simplicity of HN is nice
- 8organicbits 4y agoOne benefit of the summary bots on Reddit is that people who don't read the article, but read the summary, tend to reply to the summary. I can mostly ignore those comments as they tend to be lower quality. Unfortunately, the people who only read the headline tend to top post (with their comments being lowest quality).
- 93po 4y agoI think ignoring comments and news stories in general on reddit is the best strategy
- 8organicbits 4y agoDepends on the subreddit, but generally extremely true.
- nirui 4y agoI'm genuinely curious on which part of the comment above triggered such down votes, and why. Not that I support the ideas mentioned, I'm just curious.
- ftxbro 4y agoAlso "the per-annotation cost of ChatGPT is less than $0.003 -- about twenty times cheaper than MTurk." It's interesting that the best available MTurk Master crowd-workers located in the US are paid about six cents per task.
- 8organicbits 4y agoGiven the low quality results of the humans, I suspect they were paying too low and the workers were giving low quality answers. I wonder if they should have paid more per task to get a better comparison. It doesn't feel surprising that GPT is cheaper and more accurate than a human who is randomly clicking buttons. Custom software can already do that.
- arbol 4y agoThe per annotation cost of simple image tasks in MTurk can be as low as $0.0003
- calvinmorrison 4y agoI guess my surprise is that the machine is only 20x cheaper than the cheapest human available.
- sebzim4500 4y agoGiven how badly the humans performed, I suspect a lot of them are either not bothering to read the prompt or using a worse bot than ChatGPT. Not surprsing they are so cheap. If you tried to pay people to do it properly then presumably it would cost much more.
- ummonk 4y agoIt’s worse because a US-based MTurker is far from the cheapest human available.
- taberiand 4y agoHey, with any luck, running one of these bots with the capability to replace me will cost just as much as hiring me. (Ha! As if. Costs are only going to come down)