3 ms·

Nice, seems like you're entering into a pretty competitive space. I know multiple teams across different companies using Databricks and their datalake and ML fe

Nice, seems like you're entering into a pretty competitive space. I know multiple teams across different companies using Databricks and their datalake and ML feature stores for this purpose. I get that your tool is more specifically optimized for the task of large scale ML but what's your strategy for going up against the likes of Databricks especially when they can point to their solution and go, hey you can use our datalake both for normal business intelligence solutions and ML. Here it seems like your tool would be separate from the BI datalake and would likely be thought of and implemented a while after the BI datalake is implemented since ML maturity seems to come after BI maturity in companies I've worked for.

davidbuniat 4y ago
a very fair point, we do usually say that our main (possible) competition are the "traditional" data lakes! We've spent 4+ years designing this system to be specifically resolving the issue for unstructured AI data like videos, images, audio (and multimodal datasets, like the ones used to train models like Stable Diffusion, for instance). Our main competitive advantage against the players you've mentioned is just that - our bet is that Deep Learning will overtake traditional BI workflows (especially with >90% of data generated today being unstructured), and we've been preparing for it. Traditional "BI datalakes" are pretty inefficient when it comes to storing the data specifically for deep learning workflows. They currently also lack an entire suite of key features (visualization for those data types, query engine based on tensors, etc.) to be able to successfully convince the potential users. As a matter of fact, we're seeing not only adoption from AI-first companies/startups who are building their infrastructure from the ground up, but mature companies who are hitting the limits of the traditional setups. Keeping that in mind, we're working on making the onboarding for such companies much easier, so their cost of switching to a more efficient/performant setup is much lower. As for Databricks specifically, we see them more as a complement, rather than a competitor.
- streetcat1 4y ago
  So 90% of the data is unstructured, but 99% of the ML use cases are tabular (structured), where tree based approach can win against DL. Also, 90% of the structured data is unlabeled. Hence, your calculation should be for "labeled" unstructured data, is this 90%?. I would argue that outside big tech, this is 0.1%. You competition is not Databricks. Databricks main use case is tabular data (both in the delta lake and in ML). I.e. Databricks compete with snowflake. It tries to be a database. I.e. it tries to get out of the data lake. I think that your competition is with S3 and R2 from the storage side, and with transformer based models (Hugging face). Correct me if I am wrong, but the whole idea with transformers models is that the training was already done , and you can use small amount of domain specific data? I.e. you do not need a lot of storage?
  - davidbuniat 4y ago
    "...I would argue that outside big tech, this is 0.1%." Fair point regarding the unlabeled/unstructured data. One could also argue that labeled data isn't going to be a prerequisite forever (see https://ai.facebook.com/blog/the-first-high-performance-self-supervised-algorithm-that-works-for-speech-vision-and-text/ https://ai.facebook.com/blog/the-first-high-performance-self...). We see a very sharp rise in unstructured data use for ML (especially a large spike caused by large language models like Dall-E 2 and Stable Diffusion). In my opinion, the majority of the novel use cases are outside of big tech, and we also see a trend in "legacy" companies like media, manufacturing, etc. start building dedicated ML teams. The industry is still nascent, but it is growing fast. Frankly, we see the pain points we're solving resonate with so many more companies than just a year ago. Agree re Snowflake/Databricks, they are partners rather than competitors. We sit on top of S3/GCS or other blob storages and currently are competing with various in-house solutions that ML scientists built themselves. I do see your point regarding large foundational models that would be only fine-tuned on the tail end for various use cases. I believe there still would be still companies building foundational models from scratch (currently at 5 billion images) so they can serve more application-specific products and unstructured data generators that partner with those companies creating a good enough market for the tool.
- clusterhacks 4y ago
  "... our bet is that Deep Learning will overtake traditional BI workflows ..." This is an interesting perspective. I have spent years in the traditional BI space and my gut feeling there is that analytics there are very much not fancy. Simple stuff seems to be where the real ROI is at. Are you saying that data storage, data model, etc that Activeloop puts in place to better support deep learning workflows will replace the data storage, data model, etc as the store of information but visualization and querying will still be like BI work? Or alternatively, are you saying that deep learning is on a roaring path to replace traditional BI analytics?
  - davidbuniat 4y ago
    Thanks - it's very insightful to also hear your perspective as someone coming in from the BI space (if you have any more insights, please post them here, too). We have this internal joke where when one says their analytics is based on regressions, it's really an excel sheet, and if they say it's AI, it's a simple Ordinary Least Squares regression, and only a handful do actual AI/ML. From what we are seeing in the market, both domains grow, but with an overlap, and it's expanding, too. I think while BI/Analytics would still be a major space, we would see more DL-based novel applications generating increasingly more business value (i.e. self-driving cars, robotics, agritech). After all, even in VERY traditional workflows/companies like economic growth estimation, we're seeing DL being applied (e.g. they look at nightlight satellite imagery to estimate economic growth/urbanization). So to answer your question, for some parts, I think it would be the former (complement), and other applications it would call for replacement (particularly in the cases where companies use multi-modal data).
dimatura 4y ago
I work in a startup that uses ML for computer vision and I get the impression that there's significantly less commercial tooling to handle computer vision (or more generally, computer perception)-type data than tabular or text data. I get tons of salespeople contacting me about all sorts of solutions that I have no use for because they're not designed for image data. Don't get me wrong, there's tons of startups with tools for image/audio/video/point cloud/etc data as well, but generally they seem less mature/useful to me at this point in time. Activeloop is definitely interesting to me - I've tried it already, in fact, but it was a while ago so it was still a little bit half-baked I think.
- davidbuniat 4y ago
  As a matter of fact, I do agree - the CV space is less crowded (until now). That's hopefully where we come in! thanks a lot for the input, and thanks for trying us out. You know, it's always a work-in-progress, but we've actually done a major overhaul - this is our biggest release yet and I'd love it if you gave it a try, especially the querying feature we're super-proud of. :) https://docs.activeloop.ai/tutorials/querying-datasets https://docs.activeloop.ai/tutorials/querying-datasets here's a couple of playbooks on how the new features (visualization + querying + version control) play together to solve complex workflows. - https://docs.activeloop.ai/playbooks/training-with-lineage https://docs.activeloop.ai/playbooks/training-with-lineage - https://docs.activeloop.ai/playbooks/evaluating-model-performance https://docs.activeloop.ai/playbooks/evaluating-model-perfor... - https://docs.activeloop.ai/playbooks/training-reproducibility-with-wandb https://docs.activeloop.ai/playbooks/training-reproducibilit...

davidbuniat 4y ago

a very fair point, we do usually say that our main (possible) competition are the "traditional" data lakes! We've spent 4+ years designing this system to be specifically resolving the issue for unstructured AI data like videos, images, audio (and multimodal datasets, like the ones used to train models like Stable Diffusion, for instance). Our main competitive advantage against the players you've mentioned is just that - our bet is that Deep Learning will overtake traditional BI workflows (especially with >90% of data generated today being unstructured), and we've been preparing for it. Traditional "BI datalakes" are pretty inefficient when it comes to storing the data specifically for deep learning workflows. They currently also lack an entire suite of key features (visualization for those data types, query engine based on tensors, etc.) to be able to successfully convince the potential users. As a matter of fact, we're seeing not only adoption from AI-first companies/startups who are building their infrastructure from the ground up, but mature companies who are hitting the limits of the traditional setups. Keeping that in mind, we're working on making the onboarding for such companies much easier, so their cost of switching to a more efficient/performant setup is much lower. As for Databricks specifically, we see them more as a complement, rather than a competitor.

streetcat1 4y ago
So 90% of the data is unstructured, but 99% of the ML use cases are tabular (structured), where tree based approach can win against DL. Also, 90% of the structured data is unlabeled. Hence, your calculation should be for "labeled" unstructured data, is this 90%?. I would argue that outside big tech, this is 0.1%. You competition is not Databricks. Databricks main use case is tabular data (both in the delta lake and in ML). I.e. Databricks compete with snowflake. It tries to be a database. I.e. it tries to get out of the data lake. I think that your competition is with S3 and R2 from the storage side, and with transformer based models (Hugging face). Correct me if I am wrong, but the whole idea with transformers models is that the training was already done , and you can use small amount of domain specific data? I.e. you do not need a lot of storage?
- davidbuniat 4y ago
  "...I would argue that outside big tech, this is 0.1%." Fair point regarding the unlabeled/unstructured data. One could also argue that labeled data isn't going to be a prerequisite forever (see https://ai.facebook.com/blog/the-first-high-performance-self-supervised-algorithm-that-works-for-speech-vision-and-text/ https://ai.facebook.com/blog/the-first-high-performance-self...). We see a very sharp rise in unstructured data use for ML (especially a large spike caused by large language models like Dall-E 2 and Stable Diffusion). In my opinion, the majority of the novel use cases are outside of big tech, and we also see a trend in "legacy" companies like media, manufacturing, etc. start building dedicated ML teams. The industry is still nascent, but it is growing fast. Frankly, we see the pain points we're solving resonate with so many more companies than just a year ago. Agree re Snowflake/Databricks, they are partners rather than competitors. We sit on top of S3/GCS or other blob storages and currently are competing with various in-house solutions that ML scientists built themselves. I do see your point regarding large foundational models that would be only fine-tuned on the tail end for various use cases. I believe there still would be still companies building foundational models from scratch (currently at 5 billion images) so they can serve more application-specific products and unstructured data generators that partner with those companies creating a good enough market for the tool.
clusterhacks 4y ago
"... our bet is that Deep Learning will overtake traditional BI workflows ..." This is an interesting perspective. I have spent years in the traditional BI space and my gut feeling there is that analytics there are very much not fancy. Simple stuff seems to be where the real ROI is at. Are you saying that data storage, data model, etc that Activeloop puts in place to better support deep learning workflows will replace the data storage, data model, etc as the store of information but visualization and querying will still be like BI work? Or alternatively, are you saying that deep learning is on a roaring path to replace traditional BI analytics?
- davidbuniat 4y ago
  Thanks - it's very insightful to also hear your perspective as someone coming in from the BI space (if you have any more insights, please post them here, too). We have this internal joke where when one says their analytics is based on regressions, it's really an excel sheet, and if they say it's AI, it's a simple Ordinary Least Squares regression, and only a handful do actual AI/ML. From what we are seeing in the market, both domains grow, but with an overlap, and it's expanding, too. I think while BI/Analytics would still be a major space, we would see more DL-based novel applications generating increasingly more business value (i.e. self-driving cars, robotics, agritech). After all, even in VERY traditional workflows/companies like economic growth estimation, we're seeing DL being applied (e.g. they look at nightlight satellite imagery to estimate economic growth/urbanization). So to answer your question, for some parts, I think it would be the former (complement), and other applications it would call for replacement (particularly in the cases where companies use multi-modal data).

dimatura 4y ago

I work in a startup that uses ML for computer vision and I get the impression that there's significantly less commercial tooling to handle computer vision (or more generally, computer perception)-type data than tabular or text data. I get tons of salespeople contacting me about all sorts of solutions that I have no use for because they're not designed for image data. Don't get me wrong, there's tons of startups with tools for image/audio/video/point cloud/etc data as well, but generally they seem less mature/useful to me at this point in time. Activeloop is definitely interesting to me - I've tried it already, in fact, but it was a while ago so it was still a little bit half-baked I think.

davidbuniat 4y ago
As a matter of fact, I do agree - the CV space is less crowded (until now). That's hopefully where we come in! thanks a lot for the input, and thanks for trying us out. You know, it's always a work-in-progress, but we've actually done a major overhaul - this is our biggest release yet and I'd love it if you gave it a try, especially the querying feature we're super-proud of. :) https://docs.activeloop.ai/tutorials/querying-datasets https://docs.activeloop.ai/tutorials/querying-datasets here's a couple of playbooks on how the new features (visualization + querying + version control) play together to solve complex workflows. - https://docs.activeloop.ai/playbooks/training-with-lineage https://docs.activeloop.ai/playbooks/training-with-lineage - https://docs.activeloop.ai/playbooks/evaluating-model-performance https://docs.activeloop.ai/playbooks/evaluating-model-perfor... - https://docs.activeloop.ai/playbooks/training-reproducibility-with-wandb https://docs.activeloop.ai/playbooks/training-reproducibilit...