4 ms·

Despite being the developer, I understand where you're coming from. It's always hard to choose an open-source project and decide that you're going to base an i

by zombodb 11y ago

Despite being the developer, I understand where you're coming from. It's always hard to choose an open-source project and decide that you're going to base an important part of your "app" on it. ZomboDB hit github on Friday and I tweeted about it on Sunday. There's no doubt it's new. There's no doubt it's different. There's no doubt it ain't for everybody. As the developer, this thing is my full time job and livelihood now, so I've got a lot of motivation to continue development and definitely a lot of work to do to turn it into something lucrative. If you evaluate it on its own merits, you might find it's worth at least spending a few minutes playing with. It really is nice to do something like... SELECT * FROM table WHERE zdb(table) ==> '(beer w/3 wine) wo/10 cheese and title:(food, dinner, restaurant) and operating_hours:"11:00am"'; ... against 100M rows of mixed structured and full-text data. Plus, and here's the kicker: it's transaction safe. If an INSERT/UPDATE/DELETE statement aborts, the ES index isn't DOA. VACUUM will come along later and cleanup the dead row and in the meantime PG will maintain MVCC correctness for all active/future sessions. The pain of trying to keep an external index synchronized literally goes away. On the other hand, maybe you don't care about transactions, have a read-only dataset, and don't have a "source of truth" system for your data. If any of that is true, then you don't need ZomboDB.

nstart 11y ago
As you've seen it being used in production environments, I'm curious. What kind of applications have you seen being built with this?
- zombodb 11y ago
  It was built as _the_ search platform for a web-based document search and review system, providing real-time search, analytics, and high-volume updates.
  - zombodb 11y ago
    Let me try to be a little less vague here (I guess to much time has passed to edit?). In general, we developed this for searching structured (but not necessarily relational) full-text content with an enormous amount of associated metadata. ZomboDB has come out of the legal e-Discovery world and is the backing search platform for one of the major e-Disco service providers in the industry. It's hard to describe a typical dataset, but anywhere from 600k rows to 100M rows. Some datasets are just a few gig (on disk in Postgres) and others approach 1TB. A typical usage pattern for a document review system is that humans (and sometimes automated computer processes) literally read each document, make judgement calls about each one, apply metadata, and move to the next document. Rinse-wash-repeat. On a large-scale litigation review, it's not uncommon to have hundreds of users doing this simultaneously. As such, over time, every document gets UPDATEd at least once (typically 4-5 times due to administrative tasks). You might can imagine that 100M documents with maybe 400 reviewers is a bit of an organizational problem in terms of teams of reviewers, what they should be reviewing, etc, so it's important that the data never lie. If the system says there's 1,863,462 "Potentially Privileged" documents, then that better the actual answer. Because a system like this has to provide real-time progress reporting, analytics on judgement calls, along with the ability to generally search and organize a lot of data, we needed something that first and foremost provided transaction isolation. Enter Postgres. We also needed sophisticated full-text searching. Enter Elasticsearch. From there it was trying to answer the question "how do we make full-text searching honor our transaction isolation rules?" That's how ZomboDB came to be. I would think any kind of data management system that wants a canonical "source of truth" system (ie, Postgres) easily text-searchable, ZomboDB would be useful. Document Review, Medical Records Management, Inventory/Catalog Systems, etc. As I said in another post, the fact that ES is abstracted away and that need to asynchronously synchronize data into it goes away, it's fairly compelling. My hope is that some of its bigger caveats (such as crash recovery) get solved sooner rather than later.

by zombodb 11y ago

nstart 11y ago

As you've seen it being used in production environments, I'm curious. What kind of applications have you seen being built with this?

zombodb 11y ago
It was built as _the_ search platform for a web-based document search and review system, providing real-time search, analytics, and high-volume updates.
- zombodb 11y ago
  Let me try to be a little less vague here (I guess to much time has passed to edit?). In general, we developed this for searching structured (but not necessarily relational) full-text content with an enormous amount of associated metadata. ZomboDB has come out of the legal e-Discovery world and is the backing search platform for one of the major e-Disco service providers in the industry. It's hard to describe a typical dataset, but anywhere from 600k rows to 100M rows. Some datasets are just a few gig (on disk in Postgres) and others approach 1TB. A typical usage pattern for a document review system is that humans (and sometimes automated computer processes) literally read each document, make judgement calls about each one, apply metadata, and move to the next document. Rinse-wash-repeat. On a large-scale litigation review, it's not uncommon to have hundreds of users doing this simultaneously. As such, over time, every document gets UPDATEd at least once (typically 4-5 times due to administrative tasks). You might can imagine that 100M documents with maybe 400 reviewers is a bit of an organizational problem in terms of teams of reviewers, what they should be reviewing, etc, so it's important that the data never lie. If the system says there's 1,863,462 "Potentially Privileged" documents, then that better the actual answer. Because a system like this has to provide real-time progress reporting, analytics on judgement calls, along with the ability to generally search and organize a lot of data, we needed something that first and foremost provided transaction isolation. Enter Postgres. We also needed sophisticated full-text searching. Enter Elasticsearch. From there it was trying to answer the question "how do we make full-text searching honor our transaction isolation rules?" That's how ZomboDB came to be. I would think any kind of data management system that wants a canonical "source of truth" system (ie, Postgres) easily text-searchable, ZomboDB would be useful. Document Review, Medical Records Management, Inventory/Catalog Systems, etc. As I said in another post, the fact that ES is abstracted away and that need to asynchronously synchronize data into it goes away, it's fairly compelling. My hope is that some of its bigger caveats (such as crash recovery) get solved sooner rather than later.