3 ms·

Congrats on the launch! Few questions/thoughts: - What kind of overheads do you have right now with calling this API? - What scales have you pressure-tested

Congrats on the launch! Few questions/thoughts: - What kind of overheads do you have right now with calling this API? - What scales have you pressure-tested this with? Demo seems to show few 100s of embeddings. Selfishly, I'd like to see a demo of handling 10M+ vectors to be reasonably certain that any company can truly build infrastructure in this context. I guess I'm more interested in the out-of-core applications where I can really shove all my data in here, and see if the system can handle it. - (dovetails with the previous one): What kind of access patterns are you seeing today, more indie developers pushing few 1000s of vectors into a DB or some heavy users pushing 100K-1M+ vectors. - Less of a question, but one thought would be to partner with labeling companies to automatically fine-tune embeddings as part of a single embeddings-management platform. - Would you eventually look to build your own vector DB + metadata / features stores as part of the long-term strategy or try to integrate with existing ones?

jxodwyer1 4y ago
Thank you so much for your questions! - As a managed service there are some overheads. We need to auth, validate and parse the inputs, fetch the index that is getting queried as we then need to use the index’s model to generate the embeddings. Then if the index is fine tuned/customized, we need to transform the embedding to the new vector space, to then call our vector index. We then fetch the metadata of the results from the db and parse the response to send it back. - We’ve only tested upwards of 1M vectors ~ 1500 dimensions. But, more formal testing is required here and we plan to do so. I’m particularly curious about pg_vector and how it stacks up with other players as keeping the data central is a significant upside. We started with these lower vector indices to get something out there and iterate as a startup. But, scalability is part of what we want long term. - We see both; we’ve had to turn down a very early lead with 100M+ vectors because it would derail other engineering efforts while we were starting. We’re now much better positioned to tackle that challenge as we have all the foundations. - We haven’t considered this, but it’s an excellent idea. We’re currently discussing this with the team. We would love to chat more; we appreciate your questions and feedback. Always happy to riff with someone who has seen issues around these use cases, like yourself. Feel free to reach us at founders@getmetal.io !
- fzysingularity 4y ago
  Thanks for your reply. - I have a good sense of the overheads - I was more curious about the latencies (ms) you are observing with the system today. - Out of curiosity, why did you pick Redis? Is it mostly due to familiarity and experience with it in the past? I'm curious if you foresee any challenges scaling to larger datasets due to the in-memory limitations. - I'm assuming you're going with a usage-based model for large volumes of data managed? Do you support spinning down the service (moving things to cold-storage), and auto-scaling things back up when users actually search for things. Wondering how you're thinking about this especially if customers don't use the APIs daily. - For the 100M+ vectors, what type of data were they dealing with, documents, images or something else? Thanks!

jxodwyer1 4y ago

Thank you so much for your questions! - As a managed service there are some overheads. We need to auth, validate and parse the inputs, fetch the index that is getting queried as we then need to use the index’s model to generate the embeddings. Then if the index is fine tuned/customized, we need to transform the embedding to the new vector space, to then call our vector index. We then fetch the metadata of the results from the db and parse the response to send it back. - We’ve only tested upwards of 1M vectors ~ 1500 dimensions. But, more formal testing is required here and we plan to do so. I’m particularly curious about pg_vector and how it stacks up with other players as keeping the data central is a significant upside. We started with these lower vector indices to get something out there and iterate as a startup. But, scalability is part of what we want long term. - We see both; we’ve had to turn down a very early lead with 100M+ vectors because it would derail other engineering efforts while we were starting. We’re now much better positioned to tackle that challenge as we have all the foundations. - We haven’t considered this, but it’s an excellent idea. We’re currently discussing this with the team. We would love to chat more; we appreciate your questions and feedback. Always happy to riff with someone who has seen issues around these use cases, like yourself. Feel free to reach us at founders@getmetal.io !

fzysingularity 4y ago
Thanks for your reply. - I have a good sense of the overheads - I was more curious about the latencies (ms) you are observing with the system today. - Out of curiosity, why did you pick Redis? Is it mostly due to familiarity and experience with it in the past? I'm curious if you foresee any challenges scaling to larger datasets due to the in-memory limitations. - I'm assuming you're going with a usage-based model for large volumes of data managed? Do you support spinning down the service (moving things to cold-storage), and auto-scaling things back up when users actually search for things. Wondering how you're thinking about this especially if customers don't use the APIs daily. - For the 100M+ vectors, what type of data were they dealing with, documents, images or something else? Thanks!