3 ms·
Some slightly different views based on our experiences with startups/techco's/enterprises/govs. A lot of what there is true, but at the same time, we've been se
by lmeyerov 5y ago
Some slightly different views based on our experiences with startups/techco's/enterprises/govs. A lot of what there is true, but at the same time, we've been seeing major appetite in 2 areas that lead to quite different directions from the article.
To be clear, I do think the CACM authors are right about a lot of what is commercially + academically popular in big graph land today. Ex: Powering better knowledge graphs, especially for NLP neural network empowered use cases (BERT/GPT). Ex: Enriching inputs into traditional neural network inference calls, like a user 360 before making a recommendation. I expect graph DBs to keep improving to help with theses.
At the same time, working with teams from first-principles to help skate to where the puck is going, we've come to 2 overall different priorities where those ideas aren't the primary technology enablers for big graph use cases in terms of engineering + big $$$$ enablers. Broadly, it's useful to think though what Data Analysts are doing and what Data Scientists are doing:
1. Graph BI: Similar to Tableau, Looker, and more autoML-y successors, there are a LOT of analysts who are trying to better understand all that data in their data warehouses. Operational data in Splunk/ELK, transactions & business data in SalesForce CRM / RedShift transactions / etc. Graph questions are powerful here. Graphs are all about relationships like user/cohort behavior, so opens up a lot of questions/answers for analysts, especially if made accessible. However, from a data engineering perspective, copying that data into graph DBs as a second system of record is historically a "now you have 2 problems" thing, and part of why most data lake projects fail. So we've been focusing on building graph query pushdown + virtual fabric style projects that expose graph thinking over your existing data infra. The discussion in the CACM article on dynamic graphs touches on this, but that's only one piece of that problem. Getting analyst graph insights over warehouses as ~easy as it is today for basic time questions on events requires a lot more. Surprisingly, we suspect so much so that it may commoditize the particular choice of graph compute engine (vs say connectors/virtualization/UI/automation/AI.)
2. Graph AI: Traditional neural networks have been increasingly dominating due to strong ability to combine categorical & continuous data around entities of interest (think computer vision realizing some pixels are a cat)... but struggled to leverage relationship structure: imagine a communication or transaction between people/devices/companies with rich histories & communities. Being able to fuse those matters a LOT on commercial contexts, so we'd been watching graph neural networks for awhile. Over the last ~2 years, with the mounting focus by the AI community, key problems around performance, heteorgeneous data, etc., have been getting solved, so GNNs are starting to show up in all sorts of places now. Interestingly, most companies using them don't plug graph databases or knowledge graphs or even traditional graph compute engines into them. If you look at how pytorch, dgl, etc. are being built, they largely ignore what traditional graph infra vendors focus on as 1-3 magnitudes too slow, and overall, wrong tool for the job: different problem space.
In both cases, teams building/using graph DB or graph compute engines, like the CACM authors, get enlisted to help as natural stakeholders for pieces of the problem. Likewise, they naturally want to help + grow their scope. Unfortunately, from a perspective of "you're lucky to do even 1-3 things well", they've been struggling to provide the needed end-to-end solutions. There are plenty of critical & fascinating scale, design, + use case challenges to tackle far beyond the core db/compute engine perspective here.
Anyways, this is an awesome time for this stuff so I encourage getting involved! We're actively hiring for help creating these new categories (all levels: fullstack, UI, BI, data, AI) + launching projects with lighthouse design partners, so if either is relevant, please reach out! Likewise, traditional graph DB/compute is important too (see: CACM article), and been a no-brainer for attracting most of the VC attention for the last few years due to being a traditional DB play, and we've enjoyed partnering with friends at them, so I'd recommend looking at those companies as well.