5 ms·

Hi, I’m one of the (prospective) co-founders of TiloDB, a serverless “entity-resolution” technology. We built TiloDB as the tech team at a European consumer cr

Hi, I’m one of the (prospective) co-founders of TiloDB, a serverless “entity-resolution” technology. We built TiloDB as the tech team at a European consumer credit bureau when we were faced with the technical challenge of how to assemble hundreds of millions of data sets about tens of millions of people in a way that is scalable and allows fast searching, without breaking the bank. We tried various technologies, such as graph databases, but none of them could give us satisfactory performance. So we turned to the opportunities of serverless technology (AWS specifically) to build a new type of entity resolution technology. In this article we write about the technology breakthroughs that led to TiloDB, and there is also an interactive demo where you can submit data, see it linked, and see other people submitting data in real-time. We want to spin the tech out into a new company, release it as OSS, and so are keen to hear about potential use cases you might have.

willvarfar 5y ago
Very cool project. Setting up a business based on a new DB tech that has one user, though, is tricky. Playing devil's advocate, how do you plan to make money? Who are the users, why do they turn to TiloDB, how do they learn about it, how do they adopt it, how do they be convinced to pay you something for it? Etc
- Major_Grooves 5y ago
  Thanks for your question. You are right - it is not an easy business to start. Investors are more used to open source projects that are already released and have community adoption that they can measure. We are kinda the opposite - enterprise ready software that wants to go open source. So we want to make the software open source, but restrict a few modules that would be necessary for enterprise customers, such as security and auditing features. We have quite a few companies lined up that want to do proof of concept trials with us. So far they are mostly big fintech companies that use it for anti-fraud, also AML/KYC companies that need to match and search lots of data from different sources in real time. Also very large companies that need to solve their "data silo" problem. Adoption - hopefully they start with the OSS version, play with then want to upgrade to the enterprise version. One area we have less experience is with which type of OSS licence to use.
  - ALLTaken 5y ago
    Also highly interested! It would be awesome if the database could be accessible via C/C++ or Rust library in order integrate into existing applications, if that makes sense.
    - Major_Grooves 5y ago
      You would be able to access everything via the GraphQL API to integrate into existing applications.
  - kall 5y ago
    Just a quick suggestion: if you restrict the security module, do so in a way that someone can still run the OSS version in a basic secure way. If there are lots of insecure instances of your db out there, or someone else steps in and provides a solution, that doesn‘t reflect well on the project. This wasn‘t great about elasticsearch and they changed it later.
    - skafoi 5y ago
      The idea for that is, that typical enterprise features like authorization for certain records or even attributes are not publicly available. Also e.g. encryption of the data in S3 and other parts may be an enterprise only feature. Other things, like API authorization, preventing public access to S3 and therelike must be included in the OSS version for the same reasons you mentioned.
  - boulos 5y ago
    Disclosure: I used to work on Google Cloud. I see a lot of similarities between Kafka and Confluent. You're looking to spin out a tool that worked well for you, and offer it commercially. I'd suggest planning more around operating TiloDB as a managed service. You happen to just need lambda, s3, and dynamo today, but the "capturable" value for many customers will be if you also manage it all for them (especially upgrades). You can still offer the open core and let folks run their own, but it sounds like a lot of the goodness comes from the way you run it. Having said that, licensing is currently fraught in this space. Each major "database" vendor (Elastic, Redis Labs, Confluent) is basically trying to find a way to figure out how to avoid AWS (and other clouds) from just taking their code and operating it as a service. People have very strong opinions on this topic, ranging from "open-source isn't a business plan" to "AWS is violating the spirit of the OSS community" and many more. My personal advice would be to assess more clearly why you want to be open source (you mentioned community and applications you couldn't imagine) and whether you think open source better achieves those goals than say a free tier or distributing a core binary / container image for free. What, more specifically, do you want to get out of being open source? Contributions to the core? Contributions to the operational part? More users and feedback?

willvarfar 5y ago

Very cool project. Setting up a business based on a new DB tech that has one user, though, is tricky. Playing devil's advocate, how do you plan to make money? Who are the users, why do they turn to TiloDB, how do they learn about it, how do they adopt it, how do they be convinced to pay you something for it? Etc

Major_Grooves 5y ago
Thanks for your question. You are right - it is not an easy business to start. Investors are more used to open source projects that are already released and have community adoption that they can measure. We are kinda the opposite - enterprise ready software that wants to go open source. So we want to make the software open source, but restrict a few modules that would be necessary for enterprise customers, such as security and auditing features. We have quite a few companies lined up that want to do proof of concept trials with us. So far they are mostly big fintech companies that use it for anti-fraud, also AML/KYC companies that need to match and search lots of data from different sources in real time. Also very large companies that need to solve their "data silo" problem. Adoption - hopefully they start with the OSS version, play with then want to upgrade to the enterprise version. One area we have less experience is with which type of OSS licence to use.
- ALLTaken 5y ago
  Also highly interested! It would be awesome if the database could be accessible via C/C++ or Rust library in order integrate into existing applications, if that makes sense.
  - Major_Grooves 5y ago
    You would be able to access everything via the GraphQL API to integrate into existing applications.
- kall 5y ago
  Just a quick suggestion: if you restrict the security module, do so in a way that someone can still run the OSS version in a basic secure way. If there are lots of insecure instances of your db out there, or someone else steps in and provides a solution, that doesn‘t reflect well on the project. This wasn‘t great about elasticsearch and they changed it later.
  - skafoi 5y ago
    The idea for that is, that typical enterprise features like authorization for certain records or even attributes are not publicly available. Also e.g. encryption of the data in S3 and other parts may be an enterprise only feature. Other things, like API authorization, preventing public access to S3 and therelike must be included in the OSS version for the same reasons you mentioned.
- boulos 5y ago
  Disclosure: I used to work on Google Cloud. I see a lot of similarities between Kafka and Confluent. You're looking to spin out a tool that worked well for you, and offer it commercially. I'd suggest planning more around operating TiloDB as a managed service. You happen to just need lambda, s3, and dynamo today, but the "capturable" value for many customers will be if you also manage it all for them (especially upgrades). You can still offer the open core and let folks run their own, but it sounds like a lot of the goodness comes from the way you run it. Having said that, licensing is currently fraught in this space. Each major "database" vendor (Elastic, Redis Labs, Confluent) is basically trying to find a way to figure out how to avoid AWS (and other clouds) from just taking their code and operating it as a service. People have very strong opinions on this topic, ranging from "open-source isn't a business plan" to "AWS is violating the spirit of the OSS community" and many more. My personal advice would be to assess more clearly why you want to be open source (you mentioned community and applications you couldn't imagine) and whether you think open source better achieves those goals than say a free tier or distributing a core binary / container image for free. What, more specifically, do you want to get out of being open source? Contributions to the core? Contributions to the operational part? More users and feedback?