7 ms·
Here are a few, and I frequently have these conversations with VCs, albeit biased toward areas I work in: - Spatiotemporal analytics usually in the context of
by jandrewrogers 10y ago
Here are a few, and I frequently have these conversations with VCs, albeit biased toward areas I work in:
- Spatiotemporal analytics usually in the context of IoT. Most people currently repurpose cartographic tools for this purpose but the impedance match is poor and the tools are seriously lacking elementary functionality. There is no magic technology here, just exceptional UX/UI and an understanding of the problem domain and tooling requirements.
- IoT database platforms, no one offers a credible solution for this currently. Everyone defines this in terms of what they can do, not in terms of what is required in practice. There are many VCs currently hunting for this product but the problem is one of fundamental tech; you can't solve it using open source backends.
- Also for IoT, ad hoc clusters of compute at the edge being able to cooperate for analytical applications. The future of large-scale data analytics is planetary scale federation for many applications. Significant tech gaps here.
- Remote sensing analytics. Drones and satellites are generating spectacular volumes of this data and no one can usefully analyze data of this type at scale. Today, companies wait weeks for a single analytic output on less than a terabyte of data.
- Population-scale behavioral analytics. Many startups claim to do this but none of them can actually work with relevant data at a scale that would deliver on it despite increasing availability of the necessary data.
- AI based on algorithmic induction tech i.e. not the usual DNN and ML tech everyone calls AI. This is way more interesting if you have a novel approach.
- yeukhon 10y agoA lot of that is really really specific for what you do. I think people are generally trying to solve ONE problem rather than creating a solution for the general problem, because usually, a general solution breaks down very very quickly and thus low retention rate.
- shortoncash 10y agoCould you be more specific about the IoT DB issue? What would a credible solution do? Thanks in advance.
- jandrewrogers 10y agoAn IoT database requires the intersection of three software capabilities in a single platform, which have varying degrees of accessibility (the technology exists in principle) and none are available in open source currently. - An exabyte-scale storage engine. Nothing too exotic here technically and a few companies have built them, but the design needs to address continuous data corruption, continuous hardware failure, geo-federation, etc. - A real-time database kernel that supports very high throughput for mixed workloads. A production kernel of this type doesn't currently exist though several people working in closed source databases understand the necessary computer science in principle; the academic literature is far behind the state-of-the-art. The ability to gracefully shift load and transients between servers under full load with many millions of writes per second is not trivial. - Native discrete topology operators. Necessary for geospatial analytics, sensor coverages, etc. If you can do it natively in the database kernel, it makes the second requirement easier to achieve since you don't need secondary indexing generally. Any solution even halfway toward the general solution would be viable. The value possible if you have such a system is hard to overestimate. Companies have paid half a million dollars for the output of a single analytic query on tens of trillions of IoT records; the differentiator was that it was possible to execute such a query at all. It is extremely high-end and polymathic computer science, but serious valuable if you can make a credible dent in it. And unlike some advanced topics in computer science, there are no epic unsolved theoretical problems you have to solve, though some relevant computer science may be unpublished.
- bostik 10y agoA friend is working on an IoT problem with two of your three capabilities: storage and analytics. He has voiced similar concerns. (To be fair, he gripes constantly that nothing exists which would work for him, within the budgetary and timing constraints.) His company provides sensor systems for civil engineering projects. A single large bridge can have sensor packs every 50m or so - per beam. The amount of sensor data coming in for a single municipality or region is already staggering. Vibration and stress analytics are required on a daily basis. The final requirement, one which you didn't mention, is that this setup should be fairly low maintenance. If you need a team of rocket scientists to operate and just keep it from falling over, the cost structure will be unsustainable. A service that provided all this in a platform with sane APIs and good BI integration should be making tons of money.
- seahckr 10y agoVery insightful. In your expert opinion, what are some of the core features / functionality that an IoT database platform should have (that existing solutions or combinations of them on AWS / Google Cloud / Azure don't provide)?
- huevosabio 10y agoAs someone that has "repurposed cartographic tools" to analyze spatiotemporal datasets, what specifically do you feel is missing from current tools?
- jandrewrogers 10y agoI spent many years repurposing cartographic tools myself. :-) Three big missing features off the top of my head: - Insufficiently correct and high precision computational geometry, which compounds with the iterative/recursive nature of many complex sensor analytics. Many people don't notice unless they ground truth their analytic process; I learned this the hard way. For many industries, 1% cumulative computational error is a catastrophic bug for analytics and the reality can be much worse in many common systems. - Lack of first class tessellation types and operators. Once your data scientists have them, they'll wonder how they lived without them. Such things are completely useless for cartography and therefore don't exist in those platforms. - High-performance computational geometry. This is particularly noticeable if you work with sensor coverages (like drone data). Your typical cartographic system has serious difficulty joining a few terabytes of complex polygons, but these are tiny data sets for many remote sensing sources. It literally takes weeks or months to run these types of queries. You can optimize this to be much faster but there was no pressing market need in cartography and cartographic systems aren't designed for scale-out generally.
- huevosabio 10y agoInteresting! I'll ponder over these; thanks for the insight!
- marmaduke 10y agoAren't these addressed by CGAL? Or you mean, you'd want database primitives using CGAL?
- jandrewrogers 10y agoUnfortunately CGAL is not suitable, and yes you do need to support it in the underlying representation and database. The litmus test for a suitable computational geometry engine is the ability to do arbitrary ellipsoid computational geometry at something like one part per trillion error, and the ability to compute intersections of very complex polygons on that surface quickly. On that last point, you can execute an effective denial of service attack on most GIS databases with a well-crafted set of polygons and polygon intersection queries. It doesn't even require malicious intent. (Yet another thing I learned the hard way.)
- ColanR 10y ago> AI based on algorithmic induction tech i.e. not the usual DNN and ML tech everyone calls AI. This is way more interesting if you have a novel approach. Any chance you can elaborate on what you're talking about here?
- dil8 10y agoI think the OP is referring to algorithmic probability. See [1] R. J. Solomonoff. A formal theory of inductive inference: Parts 1 and 2. Information and Control, 7:1--22 and 224--254, 1964. [2] R. J. Solomonoff. Complexity-based induction systems: Comparisons and convergence theorems. IEEE Transactions on Information Theory, IT-24:422-432, 1978. [3] http://www.scholarpedia.org/article/Algorithmic_probability http://www.scholarpedia.org/article/Algorithmic_probability
- Jzast 10y agoI believe he is referring to AI that can generate new ideas, similar to 'thinking' versus pattern recognition, which is 'I've seen something like that before (during training!).
- derrickdirge 10y agoSo AI that can develop algorithms on its own? Basically AI that can write code?
- mikelevins 10y agoPerhaps he means old-fashioned AI techniques like production-rule systems, knowledge representation, and truth maintenance. If so, and if he's right, that would interest me, since I know how to do those things.
- paulsutter 10y agoCould you describe the applications for these tools?
- jonobelotti 10y agoCan you expand on the last point?
- kozikow 10y ago> Remote sensing analytics. Drones and satellites are generating spectacular volumes of this data and no one can usefully analyze data of this type at scale. Today, companies wait weeks for a single analytic output on less than a terabyte of data. I am co-founder of tensorflight.com. We do computer vision analytics of drone imagery. Interesting that you mention it, as I thought it's a somewhat obscure market. When we talk to investors in the valley 75%+ have to be educated about why what we do is a viable business. Please get in touch at kozikow@tensorflight.com if you have any ideas!
- deleted 10y ago[deleted]
- deepnotderp 10y ago> - AI based on algorithmic induction tech i.e. not the usual DNN and ML tech everyone calls AI. This is way more interesting if you have a novel approach. I know it's fashionable to hate on deep learning, but algorithm induction is literally what deep learning does.
- derrickdirge 10y agoI would love to read an elaboration on this. There's so much I don't understand about AI.
- mywittyname 10y agoI believe he's talking about https://en.wikipedia.org/wiki/Solomonoff's_theory_of_inductive_inference https://en.wikipedia.org/wiki/Solomonoff's_theory_of_inducti...
- rz2k 10y agoI understood the question as, what makes you think: the current system could be streamlined so easily that I can hardly believe someone hasn't already disrupted the complacent firms extracting rent. Such a question usually gets answers mostly comprised of work that is extremely boring at first glance. For example, create a system than halves the work to complete documentation for some sort of compliance with regulations. Every one of these ideas just sounds incredibly interesting to work on. The type that gets a lot of really bright people together on a team (that I'd really want to be on too), yet they might not deliver a product, and possibly even less likely figure out how to monetize it. Anyway, I'm not developing a friction-reducing product either, but worry about choosing something because it sounds like a rewarding, intellectual challenge.
- AznHisoka 10y agoCan you clarify what you mean by behavior analytics and why companies eould find that useful?
- Crespyl 10y agoThink Hari Seldon/psychohistory. Predicting mid to long-term trends and demographic shifts far enough ahead of time to make investments.
- deleted 10y ago[deleted]
- armenarmen 10y ago> Population-scale behavioral analytics. Many startups claim to do this but none of them can actually work with relevant data at a scale that would deliver on it despite increasing availability of the necessary data. Making Asimov's Psychohistory from Foundation a reality!
- deleted 10y ago[deleted]
- Chris2048 10y ago> no one offers a credible solution > There are many VCs currently hunting for this product Implies a difficult problem. > algorithmic induction tech Do you mean inference?