3 ms·

you have experience with this, so you're probably the perfect person to ask this to- why didn't you just use the old (inaccurate) labels, perform some clusterin

you have experience with this, so you're probably the perfect person to ask this to- why didn't you just use the old (inaccurate) labels, perform some clustering based op and re-label then clusters? does that even make sense?

theK 4y ago
Doesn't this bias the labels?
- shreyshnaccount 4y ago
  probably does, but idk how much.. if its mostly okay it's gonna be easier to correct it id assume
stingraycharles 4y ago
The old (inaccurate) labels were so completely utter shit (pardon my French) that it may as wel have been random. I honestly believe most MTurkers just clicked random BS in order to complete the tasks as soon as possible. What I ended up doing was make some Python CLI-based took that made it extremely fast for us to classify messages; after seeding it with about 1000 classifications, we would then focus on the messages based on certain dimensions: eg “contradictions” (“positive” and “negative” being closest as possible, or “angry” and “happy”), “least” (it was surprisingly difficult to find positive and uplifting tweets, and you don’t want a dataset with 90% negative messages!), etc. That way we worked our way through the dataset and were able to get a pretty decent dataset in about a week time. No idea how others approach this type of problem, but it’s what I came up with.
- bragr 4y ago
  >I honestly believe most MTurkers just clicked random BS in order to complete the tasks as soon as possible. This is correct based on my (limited) mechanical turk experience. Most tasks pay peanuts (minimum payout can be as low as $0.01) so the only reasonable way to make an income is to complete as many tasks as humanly possible, and doing anything but clicking random buttons would slow them down. I doubt paying more could overcome that because so many people engage with the platform in bad faith.
  - codedokode 4y ago
    You can filter out bad workers by preparing an additional well-labeled dataset and removing those who made even a single mistake in it. Also you can give the same task to several workers and check if they give the same label. However this won't protect against a bot using multiple accounts and giving answers based on a hash of a question so that the same question gets the same answer in every account.
    - stjohnswarts 4y ago
      "removing those who made even a single mistake" means you don't want humans working on this lol. Humans will always make mistakes.

theK 4y ago

Doesn't this bias the labels?

shreyshnaccount 4y ago
probably does, but idk how much.. if its mostly okay it's gonna be easier to correct it id assume

stingraycharles 4y ago

The old (inaccurate) labels were so completely utter shit (pardon my French) that it may as wel have been random. I honestly believe most MTurkers just clicked random BS in order to complete the tasks as soon as possible. What I ended up doing was make some Python CLI-based took that made it extremely fast for us to classify messages; after seeding it with about 1000 classifications, we would then focus on the messages based on certain dimensions: eg “contradictions” (“positive” and “negative” being closest as possible, or “angry” and “happy”), “least” (it was surprisingly difficult to find positive and uplifting tweets, and you don’t want a dataset with 90% negative messages!), etc. That way we worked our way through the dataset and were able to get a pretty decent dataset in about a week time. No idea how others approach this type of problem, but it’s what I came up with.

bragr 4y ago
>I honestly believe most MTurkers just clicked random BS in order to complete the tasks as soon as possible. This is correct based on my (limited) mechanical turk experience. Most tasks pay peanuts (minimum payout can be as low as $0.01) so the only reasonable way to make an income is to complete as many tasks as humanly possible, and doing anything but clicking random buttons would slow them down. I doubt paying more could overcome that because so many people engage with the platform in bad faith.
- codedokode 4y ago
  You can filter out bad workers by preparing an additional well-labeled dataset and removing those who made even a single mistake in it. Also you can give the same task to several workers and check if they give the same label. However this won't protect against a bot using multiple accounts and giving answers based on a hash of a question so that the same question gets the same answer in every account.
  - stjohnswarts 4y ago
    "removing those who made even a single mistake" means you don't want humans working on this lol. Humans will always make mistakes.