3 ms·

Can someone explain this like I am 5. What are the use cases when it says works on images, text etc? Why is this a big deal? What's the human input here? And wh

Can someone explain this like I am 5. What are the use cases when it says works on images, text etc? Why is this a big deal? What's the human input here? And what output to expect? From what I understand, human validation (supervision) is not happening while algorithm is training on data. Is that right? Will this be open to the public via standard ML frameworks or proprietary?

hahanbyul 5y ago
It has already been public on their repository, fairseq.
- kordlessagain 5y ago
  Except not the vision part, which is what they were discussing. Also, I don't see any way to run the examples as the code is missing those files: https://github.com/pytorch/fairseq/tree/main/examples/data2vec https://github.com/pytorch/fairseq/tree/main/examples/data2v... The code is what we're interested in here, not "hold onto your papers" talk that tells us how to be excited about it.
iamstupidsimple 5y ago
I imagine a multimodal platform like Facebook (with people using photos and text together in some way) would find a single model valuable.
zcw100 5y ago
The benefits are similar to when you consolidate tasks with a lot of similarities. Advancements in one area can be immediately applied to another area. You can consolidate your effort into a single point. Training a network is increasingly cost prohibitive, into the millions of dollars, and this would allow you to consolidate these expenses. EDIT: Looks like they still need to train for each modality so no benefit from that here. I'd imagine on the flip side the potential problems would be when gains in one modality are offset by a decrease in performance in another or you are prevented from trying new things because of the choices made to support a unified architecture.
powerapple 5y ago
Validation is different from supervision. Supervision happens during training. Typically when we train speech model, we prepare a speech segment and its transcript, this transcript is generated manually by human. For language models such as BERT or GPT-3, you don't really need human generated labels, what it does is to mask out some text, and the masked out text is the label to be predicted (self-supervise), the label is already in the data. What this model does is to universally train on images, text and speech this way, and when you combine these data, they will be complementary to each other. Think what you can do with a language model like GPT-3 now, you would be able to ask the model to do something with text, speech and images. The only thing I would worry about this model is that I would imagine Facebook trained on lots of data, and the model may be used to extract sensitive data maybe. The multi-modal part is a big deal because we could have a model be able to understand all inputs human needs.
axg11 5y ago
I work on ML recommendation algos for a big Facebook-sized company, so perhaps I can give some insight on how this model could be used. This is an example and might not resemble anything they're doing internally. This is more ELI16: Facebook marketplace: - Seller posts a listing with image + description - FB uses data2vec to transform the image + description into one single vector (e.g. you could average the two vectors) - Buyer searches for a product using text search - the text is also encoded into a vector with data2vec - To serve product search results, you find the closest match to the text query vector from your pool of product vectors This pattern described above is generally called vector search and is very common for recommendation algorithms and much more. There's a shift towards algorithms like data2vec that can combine different types of data into one vector. The aim is that an image of a dog and the word "dog" would map to the same vector, meaning that the vector represents the concept of a dog, regardless of the input data type. The advantage of these "multi-modal" algorithms (i.e. they can take multiple data types as input) is that you can (in theory) use them across all of your ML algorithm needs. If you're Facebook, you have 100s of teams and services that have this need. A few examples: - Instagram ads prioritization - Instagram search - Harmful content moderation - Facebook content search - Facebook marketplace search Each of these is likely a separate team, very likely using a separate embedding algorithm. As approaches like data2vec improve, there will be some consolidation. N.B. - I've made a lot of assumptions based on what I've seen at my current employer. If anyone from Meta/Facebook is reading this, please chime in!
- motoxpro 5y ago
  Very cool!
  - temp12913231 5y ago
- hungryforcodes 5y ago
  That won't really help them-- their UI is so horrific for search, especially in the market place -- that it won't improve anyone's experience there.

hahanbyul 5y ago

It has already been public on their repository, fairseq.

kordlessagain 5y ago
Except not the vision part, which is what they were discussing. Also, I don't see any way to run the examples as the code is missing those files: https://github.com/pytorch/fairseq/tree/main/examples/data2vec https://github.com/pytorch/fairseq/tree/main/examples/data2v... The code is what we're interested in here, not "hold onto your papers" talk that tells us how to be excited about it.

iamstupidsimple 5y ago

I imagine a multimodal platform like Facebook (with people using photos and text together in some way) would find a single model valuable.

zcw100 5y ago

The benefits are similar to when you consolidate tasks with a lot of similarities. Advancements in one area can be immediately applied to another area. You can consolidate your effort into a single point. Training a network is increasingly cost prohibitive, into the millions of dollars, and this would allow you to consolidate these expenses. EDIT: Looks like they still need to train for each modality so no benefit from that here. I'd imagine on the flip side the potential problems would be when gains in one modality are offset by a decrease in performance in another or you are prevented from trying new things because of the choices made to support a unified architecture.

powerapple 5y ago

Validation is different from supervision. Supervision happens during training. Typically when we train speech model, we prepare a speech segment and its transcript, this transcript is generated manually by human. For language models such as BERT or GPT-3, you don't really need human generated labels, what it does is to mask out some text, and the masked out text is the label to be predicted (self-supervise), the label is already in the data. What this model does is to universally train on images, text and speech this way, and when you combine these data, they will be complementary to each other. Think what you can do with a language model like GPT-3 now, you would be able to ask the model to do something with text, speech and images. The only thing I would worry about this model is that I would imagine Facebook trained on lots of data, and the model may be used to extract sensitive data maybe. The multi-modal part is a big deal because we could have a model be able to understand all inputs human needs.

axg11 5y ago

I work on ML recommendation algos for a big Facebook-sized company, so perhaps I can give some insight on how this model could be used. This is an example and might not resemble anything they're doing internally. This is more ELI16: Facebook marketplace: - Seller posts a listing with image + description - FB uses data2vec to transform the image + description into one single vector (e.g. you could average the two vectors) - Buyer searches for a product using text search - the text is also encoded into a vector with data2vec - To serve product search results, you find the closest match to the text query vector from your pool of product vectors This pattern described above is generally called vector search and is very common for recommendation algorithms and much more. There's a shift towards algorithms like data2vec that can combine different types of data into one vector. The aim is that an image of a dog and the word "dog" would map to the same vector, meaning that the vector represents the concept of a dog, regardless of the input data type. The advantage of these "multi-modal" algorithms (i.e. they can take multiple data types as input) is that you can (in theory) use them across all of your ML algorithm needs. If you're Facebook, you have 100s of teams and services that have this need. A few examples: - Instagram ads prioritization - Instagram search - Harmful content moderation - Facebook content search - Facebook marketplace search Each of these is likely a separate team, very likely using a separate embedding algorithm. As approaches like data2vec improve, there will be some consolidation. N.B. - I've made a lot of assumptions based on what I've seen at my current employer. If anyone from Meta/Facebook is reading this, please chime in!

motoxpro 5y ago
Very cool!
- temp12913231 5y ago
hungryforcodes 5y ago
That won't really help them-- their UI is so horrific for search, especially in the market place -- that it won't improve anyone's experience there.