3 ms·

this is a genuinely great read. Author does a great job providing examples where context is critical, and explains how the dataset not only has labeling errors,

this is a genuinely great read. Author does a great job providing examples where context is critical, and explains how the dataset not only has labeling errors, an even deeper problem is how it models language in general. Since words only have meaning within a context, your model should reflect that somehow. What wasn't really explored I'm this article was to what quantitative degree context sensitivity matters. The counter examples are great, but how can we measure the relationship between amount of context and labelling accuracy?

rkagerer 4y ago
Context is key. My dog has a better understanding of context than any AI I've ever met. (I mean that sincerely - since becoming a pet owner it's something I've really marveled at).
- kortex 4y ago
  Seriously. Half the time, I feel like the command I'm giving is just being interpreted as "do the needful", and she just figures it out, based mostly on tone, context, and memory.
echen 4y ago
Great question! I'd love to measure that more rigorously too. Although from what we've seen, the amount context sensitivity matters really depends on the labeling task / application. For example, when you're trying to label a tweet that's a reply, context matters even more than when you're labeling a parent tweet: it's often hard to understand what the reply tweet is talking about when you can't see the full thread, it can be hard to tell whether something is a joke or an insult when you can't tell whether the replier and original tweeter follow each other or not, etc. This is important because sometimes our customers don't realize this, and will send us tweet text by itself instead of a full tweet link. It's also important because even if your models are using text alone (and not a richer set of context/features), there may be patterns in the text itself that an ML could pick up on that a human wouldn't without that extra context. We also have another post on context sensitivity if you're curious: https://www.surgehq.ai/blog/why-context-aware-datasets-are-crucial-for-data-centric-ai https://www.surgehq.ai/blog/why-context-aware-datasets-are-c...
numpad0 4y ago
“Quantitative degree context sensitivity matters” sounds like a notable phrase here, I’m guessing such indications do not exist yet. As an end user I face similar problems in UI translations: A lot of failed translations are made on context deprived text, on a false notion that additional contexts are only required in nuanced edge cases. In reality it is almost always necessary especially in UI texts where words are more loaded and supplemented by visual indications. “Context” as often said might need to be defined with more depth. As it stands it is used as almost a post-hoc explanation as to why a particular output of an arbitrary language-related tasks is considered incorrect and should be discarded.

rkagerer 4y ago

Context is key. My dog has a better understanding of context than any AI I've ever met. (I mean that sincerely - since becoming a pet owner it's something I've really marveled at).

kortex 4y ago
Seriously. Half the time, I feel like the command I'm giving is just being interpreted as "do the needful", and she just figures it out, based mostly on tone, context, and memory.

echen 4y ago

Great question! I'd love to measure that more rigorously too. Although from what we've seen, the amount context sensitivity matters really depends on the labeling task / application. For example, when you're trying to label a tweet that's a reply, context matters even more than when you're labeling a parent tweet: it's often hard to understand what the reply tweet is talking about when you can't see the full thread, it can be hard to tell whether something is a joke or an insult when you can't tell whether the replier and original tweeter follow each other or not, etc. This is important because sometimes our customers don't realize this, and will send us tweet text by itself instead of a full tweet link. It's also important because even if your models are using text alone (and not a richer set of context/features), there may be patterns in the text itself that an ML could pick up on that a human wouldn't without that extra context. We also have another post on context sensitivity if you're curious: https://www.surgehq.ai/blog/why-context-aware-datasets-are-crucial-for-data-centric-ai https://www.surgehq.ai/blog/why-context-aware-datasets-are-c...

numpad0 4y ago

“Quantitative degree context sensitivity matters” sounds like a notable phrase here, I’m guessing such indications do not exist yet. As an end user I face similar problems in UI translations: A lot of failed translations are made on context deprived text, on a false notion that additional contexts are only required in nuanced edge cases. In reality it is almost always necessary especially in UI texts where words are more loaded and supplemented by visual indications. “Context” as often said might need to be defined with more depth. As it stands it is used as almost a post-hoc explanation as to why a particular output of an arbitrary language-related tasks is considered incorrect and should be discarded.