10 ms·

That was a good post. Vector Embeddings are in some sense a summary of a doc that's unique similar to a hashcode of a doc. It makes me think it would be cool if

That was a good post. Vector Embeddings are in some sense a summary of a doc that's unique similar to a hashcode of a doc. It makes me think it would be cool if there were some universal standard for generating embeddings, but I guess they'll be different for each AI model, so they can't have the same kind of "permanence" hash codes have. It definitely also seems like there should be lots of ways to utilize "Cosine Similarity" (or other closeness algos) in databases and other information processing apps that we haven't really exploited yet. For example you could almost build a new kind of Job Search Service that matches job descriptions to job candidates based on nothing but a vector similarity between resume and job description. That's probably so obvious it's being done, already.

kqr 2y ago
For one point of inspiration, see https://entropicthoughts.com/determining-tag-quality https://entropicthoughts.com/determining-tag-quality I really like the picture you are drawing with "semantic hashes"!
- quantadev 2y ago
  Yeah for "Semantic Hashes" (that's a good word for them!) we'd need some sort of "Canonical LLM" model that isn't necessarily used for inference, nor does it need to even be all that smart, but it just needs to be public for the world. It would need to be updated like every 2 to 5 years tho to account for new words or words changing meaning? ...but maybe could be updated in such a way as to not "invalidate" prior vectors, if that makes sense? For example "ride a bicycle" would still point in the same direction even after a refresh of the canonical model? It seems like feeding the same training set could replicate the same model values, but there are nonlinear instabilities which could make it disintegrate.
  - kqr 2y ago
    Maybe the embedding could be paired up with a set of words that embed to somewhere close to the original embedding? Then the embedding can be updated for new models by re-embedding those words. (And it would be more interpretible by a human.)
    - quantadev 2y ago
      I mean it was just a thought I had. May be a "solution in search of a problem". I generate those a lot! haha. But it seems to me like having some sort of canonical set of training data and a canonical LLM architecture, we'd end up able to generate consistent embeddings of course, but I'm just not sure what the use cases are.
helloplanets 2y ago
I guess it might be possible to retroactively create an embeddings model which could take several different models' embeddings, and translate them into the same format.
- genuinelydang 2y ago
  No. That’s like saying you can transplant a person’s neuronal action potentials into another person’s brain and have it make sense to them.
  - helloplanets 2y ago
    That metaphor is skipping the most important part in between! You wouldn't be transplanting anything directly, you'd have a separate step in between, which would attempt to translate these action potentials. The point of the translating model in between would be that it would re weight each and every one of the values of the embedding, after being trained on a massive dataset of original text -> vector embedding for model A + vector embedding for model B. If you have billions of parameters trained to do this translation between just two specific models to start with, wouldn't this be in the realm of possible?
    - quantadev 2y ago
      A translation between models doesn't seem possible because there are actually no "common dimensions" at all between models. That is, each dimension has a completely different semantic meaning, in different models, but also it's the combination of dimension values that begin to impart real "meaning". For example, the number of different unit vector combinations in a 1500 dimensional space is like the number of different ways of "ordering" the components, which is 5^4114 . EDIT: And the point of that factorial is that even if the dimensions were "identical" across two different LLMs but merely "scrambled" (in ordering) there would be that large number to contend with to "unscramble".
  - tempusalaria 2y ago
    This is very similar to how LLMs are taught to understand images in llava style models (the image embeddings are encoded into the existing language token stream)
genuinelydang 2y ago
”you could almost build a new kind of Job Search Service that matches job descriptions to job candidates” The key word being ”almost”. Yes, you can get similarity matches between job requirements and candidate resumes, but those matches are not useful for the task of finding an optimal candidate for a job. For example, say a job requires A and B. Candidate 1 is a junior who has done some work with A, B and C. Candidate 2 is a senior and knows A, B, C, D, E and F by heart. All are relevant to the job and would make 2 the optimal candidate, even though C–F are not explicitly stated in the job requirements. Candidate 1 would seem a much better candidate than 2, because 1’s embedding vector is closer to the job embedding vector.
- coldtea 2y ago
  Even that is just static information. We don't know if Candidate 2 really "knows A, B, C, D, E and F by heart", just that they claim to. They could be adding whatever to their skill list just, even though they hardly used it, just because it' a buzzword. So Candidate 1 could still blow them out of the water in performance, and even be able to trivially learn D, and E in a short while on the job if needed. The skill vector wont tell much by itself, and even prevent finding the better candidate if its used for screening.
  - quantadev 2y ago
    So your point is that LLMs can't tell when job candidates are lying on their resume? Well that's true, but neither can humans. lol.
  - inkyoto 2y ago
    > We don't know if Candidate 2 really "knows A, B, C, D, E and F by heart", just that they claim to. They could be adding whatever to their skill list just, even though they hardly used it, just because it' a buzzword. That is indeed a problem. I have been thinking about a possible solution to the very same problem for a while. The fact: people lie on their resumes, and they do it for different reasons. There are white lies (e.g. pumps something up because they aspire to something but were not presented with an opportunity to do it, yet they are eager to skill themselves up, learn and do it, if given an opportunity). Then there are other lies. Generally speaking, lies are never black or white, true or false; they are a shade of grey. So the best idea I have been able to come up with so far is a hybrid solution that entails the text embeddings (the skills similarity match and search) coupled with the sentiment analysis (to score the sincerity of the information stated on a resume) to gain an extra insight into the candidate's intentions. Granted, the sentiment analysis is an ethically murky area…
rasulkireev 2y ago
I tried doing something like that: https://gettjalerts.com/ https://gettjalerts.com/ I added semantic search, but I'm workin on adding resume upload/parsing to do automatic matching.
SCUSKU 2y ago
It does exist! I built this for the monthly Who's Hiring threads: https://hnresumetojobs.com/ https://hnresumetojobs.com/ It just does cosine similarity with OpenAI embeddings + pgVector. It's not perfect by any means, but it's useful. It could probably stand to be improved with a re-ranker, but I just never got around to it.
- quantadev 2y ago
  Very cool. I knew it was too obvious an idea to be missed! Did you read my comments below about how you can maybe "scale up" a vector based on number of years of experience. I think that will work. It makes somebody with 10 yrs Java Experience closer to the target than someone with only 5yrs, if the target is 10 years! -- but the problem is someone with 20yrs looks even worse when they should look better! My problem in my life. hahaha. Too much experience. I think the best "matching" factor is to minimize total distance where each distance is the time-multiplied vector for a specific skill.
connor11528 2y ago
> For example you could almost build a new kind of Job Search Service that matches job descriptions to job candidates based on nothing but a vector similarity between resume and job description. That's probably so obvious it's being done, already. Literally the next item on my roadmap for employbl dot com lol. we're calling it a "personalized job board" and using PGVector for storing the embeddings. I've also heard good things about Typesense though. One thing I've found to be important when creating the embeddings is to not do an embedding of the whole job description. Instead use an LLM to make a concise summary of the job listing (location, skills etc.) in a structured format. Then store that store as the embedding. It reduces noise and increases accuracy for vector search.

kqr 2y ago

For one point of inspiration, see https://entropicthoughts.com/determining-tag-quality https://entropicthoughts.com/determining-tag-quality I really like the picture you are drawing with "semantic hashes"!

quantadev 2y ago
Yeah for "Semantic Hashes" (that's a good word for them!) we'd need some sort of "Canonical LLM" model that isn't necessarily used for inference, nor does it need to even be all that smart, but it just needs to be public for the world. It would need to be updated like every 2 to 5 years tho to account for new words or words changing meaning? ...but maybe could be updated in such a way as to not "invalidate" prior vectors, if that makes sense? For example "ride a bicycle" would still point in the same direction even after a refresh of the canonical model? It seems like feeding the same training set could replicate the same model values, but there are nonlinear instabilities which could make it disintegrate.
- kqr 2y ago
  Maybe the embedding could be paired up with a set of words that embed to somewhere close to the original embedding? Then the embedding can be updated for new models by re-embedding those words. (And it would be more interpretible by a human.)
  - quantadev 2y ago
    I mean it was just a thought I had. May be a "solution in search of a problem". I generate those a lot! haha. But it seems to me like having some sort of canonical set of training data and a canonical LLM architecture, we'd end up able to generate consistent embeddings of course, but I'm just not sure what the use cases are.

helloplanets 2y ago

I guess it might be possible to retroactively create an embeddings model which could take several different models' embeddings, and translate them into the same format.

genuinelydang 2y ago
No. That’s like saying you can transplant a person’s neuronal action potentials into another person’s brain and have it make sense to them.
- helloplanets 2y ago
  That metaphor is skipping the most important part in between! You wouldn't be transplanting anything directly, you'd have a separate step in between, which would attempt to translate these action potentials. The point of the translating model in between would be that it would re weight each and every one of the values of the embedding, after being trained on a massive dataset of original text -> vector embedding for model A + vector embedding for model B. If you have billions of parameters trained to do this translation between just two specific models to start with, wouldn't this be in the realm of possible?
  - quantadev 2y ago
    A translation between models doesn't seem possible because there are actually no "common dimensions" at all between models. That is, each dimension has a completely different semantic meaning, in different models, but also it's the combination of dimension values that begin to impart real "meaning". For example, the number of different unit vector combinations in a 1500 dimensional space is like the number of different ways of "ordering" the components, which is 5^4114 . EDIT: And the point of that factorial is that even if the dimensions were "identical" across two different LLMs but merely "scrambled" (in ordering) there would be that large number to contend with to "unscramble".
- tempusalaria 2y ago
  This is very similar to how LLMs are taught to understand images in llava style models (the image embeddings are encoded into the existing language token stream)

genuinelydang 2y ago

”you could almost build a new kind of Job Search Service that matches job descriptions to job candidates” The key word being ”almost”. Yes, you can get similarity matches between job requirements and candidate resumes, but those matches are not useful for the task of finding an optimal candidate for a job. For example, say a job requires A and B. Candidate 1 is a junior who has done some work with A, B and C. Candidate 2 is a senior and knows A, B, C, D, E and F by heart. All are relevant to the job and would make 2 the optimal candidate, even though C–F are not explicitly stated in the job requirements. Candidate 1 would seem a much better candidate than 2, because 1’s embedding vector is closer to the job embedding vector.

coldtea 2y ago
Even that is just static information. We don't know if Candidate 2 really "knows A, B, C, D, E and F by heart", just that they claim to. They could be adding whatever to their skill list just, even though they hardly used it, just because it' a buzzword. So Candidate 1 could still blow them out of the water in performance, and even be able to trivially learn D, and E in a short while on the job if needed. The skill vector wont tell much by itself, and even prevent finding the better candidate if its used for screening.
- quantadev 2y ago
  So your point is that LLMs can't tell when job candidates are lying on their resume? Well that's true, but neither can humans. lol.
- inkyoto 2y ago
  > We don't know if Candidate 2 really "knows A, B, C, D, E and F by heart", just that they claim to. They could be adding whatever to their skill list just, even though they hardly used it, just because it' a buzzword. That is indeed a problem. I have been thinking about a possible solution to the very same problem for a while. The fact: people lie on their resumes, and they do it for different reasons. There are white lies (e.g. pumps something up because they aspire to something but were not presented with an opportunity to do it, yet they are eager to skill themselves up, learn and do it, if given an opportunity). Then there are other lies. Generally speaking, lies are never black or white, true or false; they are a shade of grey. So the best idea I have been able to come up with so far is a hybrid solution that entails the text embeddings (the skills similarity match and search) coupled with the sentiment analysis (to score the sincerity of the information stated on a resume) to gain an extra insight into the candidate's intentions. Granted, the sentiment analysis is an ethically murky area…

rasulkireev 2y ago

I tried doing something like that: https://gettjalerts.com/ https://gettjalerts.com/ I added semantic search, but I'm workin on adding resume upload/parsing to do automatic matching.

SCUSKU 2y ago

It does exist! I built this for the monthly Who's Hiring threads: https://hnresumetojobs.com/ https://hnresumetojobs.com/ It just does cosine similarity with OpenAI embeddings + pgVector. It's not perfect by any means, but it's useful. It could probably stand to be improved with a re-ranker, but I just never got around to it.

quantadev 2y ago
Very cool. I knew it was too obvious an idea to be missed! Did you read my comments below about how you can maybe "scale up" a vector based on number of years of experience. I think that will work. It makes somebody with 10 yrs Java Experience closer to the target than someone with only 5yrs, if the target is 10 years! -- but the problem is someone with 20yrs looks even worse when they should look better! My problem in my life. hahaha. Too much experience. I think the best "matching" factor is to minimize total distance where each distance is the time-multiplied vector for a specific skill.

connor11528 2y ago

> For example you could almost build a new kind of Job Search Service that matches job descriptions to job candidates based on nothing but a vector similarity between resume and job description. That's probably so obvious it's being done, already. Literally the next item on my roadmap for employbl dot com lol. we're calling it a "personalized job board" and using PGVector for storing the embeddings. I've also heard good things about Typesense though. One thing I've found to be important when creating the embeddings is to not do an embedding of the whole job description. Instead use an LLM to make a concise summary of the job listing (location, skills etc.) in a structured format. Then store that store as the embedding. It reduces noise and increases accuracy for vector search.