6 ms·

SenseiDB: Open-source, distributed, realtime, semi-structured database

163 points by thomas11 15y ago

ruslansv 15y ago
A truly amazing peace of technology
untog 15y ago
I can't speak for SenseiDB itself, but I think that LinkedIn deserves a lot of credit for the technology they have been open sourcing in the last few months. Fantastic to see.
joshhart 15y ago
Looks like someone non-LinkedIn posted this so the rest of the team is unlikely to be around for a few more hours. Let me know if you have any questions! I worked mostly on the distribution & rpc piece.
- threepointone 15y ago
  Congratulations on the project, and compliments on the breadth of documentation. Nicely done. (Nitpick - none of the urls change. Some js error where you're not doing a pushState? You should report this.) edit - this works -http://senseidb.github.com/sensei/index.html http://senseidb.github.com/sensei/index.html
  - joshhart 15y ago
    Thanks! I'll make sure that gets corrected.
- lemming 15y ago
  How does the consistency model work? I saw the page describing the eventual consistency referred to a routing parameter (which I assumes routes to the same node to enforce read-your-writes), but I assume you can have conflicts between writes from separate sessions? How is the conflict resolution handled?
  - joshhart 15y ago
    This is exactly how it works. Queries based on some kind of id will always hit the same server, except when the server goes down. Sensei expects a semi-durable "data stream" for writes, which is very different from how most other databases work. Imagine a near-real time processing system though where updates go into a queue to be processes (financial transactions work like this). SenseiDB works great for indexing these types of events. Text search is a good example here - after you update your profile, it's put into a queue to be indexed. We don't want to wait for your profile to be indexed before we'll let you move onto another page.
hello_moto 15y ago
How many home-grown databases have linkedin developed in the past few years?
- javasoze 15y ago
  A few... :) I think each has its own strengths and weaknesses and we use them in different projects with different requirements.
- strlen 15y ago
  How many programming languages do you use? :-) Just like we use Python, Java, Scala, etc... where each is appropriate, we also use different databases where appropriate. Each system serves a need. We also use off the shelf databases, where appropriate: it should be noted Sensei uses Lucene (a well known search library), Voldemort has a pluggable storage engine (where we mostly use BerkeleyDB and a custom read-only storage engine), and Espresso uses MySQL. We don't build databases for reasons of NIH: we focus on building features (faceting, real-time indexing, partitioning, fail over, etc...) that enable us to build fast, usable, feature rich, scalable, and reliable applications. We readily use open source components in many places within both our infrastructure systems (search, various databases, Kafka) and in the applications, and contribute to existing open source projects.
zrail 15y ago
From a quick glance at the docs there's no database-side aggregations. Is that correct?
- joshhart 15y ago
  The facets let you support counting and group by - is there something else you had in mind? Edit: You can get a pretty quick idea of what's supported the query language at http://senseidb.github.com/sensei/bql.html http://senseidb.github.com/sensei/bql.html. Since Sensei was designed with document and text uses cases in mind, we have plans to support advanced queries where a relevance model is part of the query, allowing you to perform a custom sort on the server.
  - zrail 15y ago
    sum() and distinct() come to mind for sure, but I don't know how easy they'd be to implement. The particular use case I have in mind is where we have events streaming in and we usually don't care about the actual events, just various sums, counts and distinct counts over them grouped by various dimensions.
- javasoze 15y ago
  Database-side aggregation is something in the works. We are incorporating a patch as we speak. We find more and more in house projects using Sensei as a BI tool, we see this functionality more and more important, so stay tuned :)
  - DEinspanjer 15y ago
    Would you guys be interested in collaborating on that? At Mozilla, we've been using ElasticSearch in a few places as a NoSQL style BI tool. I read through the SenseiDB docs and can see many places it could be almost as good of a fit as ES, and with a little work, I bet it could be just the thing.
    - javasoze 15y ago
opendomain 15y ago
3 separate NoSQL stores from ONE company? And their primary business is not Databases? There are currently more than 100 different NoSQL solutions - I believe we will see a consolidation of NoSQL in 2012
- efnx 15y ago
  On the flip side, we might see a lot of "build your own NoSQL store" tutorials in 2012.
dougk 15y ago
Can Sensei be used as a replacement to MySQL + lucene?
- LukeHoersten 15y ago
  Yeah, it'd be nice to see a side-by-side against other common DBs like MySQL and Oracle.
  - deleted 15y ago
    [deleted]
  - Ecio78 15y ago
    there's a comparison vs Mysql here: http://senseidb.github.com/sensei/performance.html http://senseidb.github.com/sensei/performance.html for insert and query performance
jamesu 15y ago
Nice looking wheel. I have a feeling it could be rounder though.
- deleted 15y ago
  [deleted]
  - deleted 15y ago
    [deleted]
cultureulterior 15y ago
Is it single-point-of-failure free?
- joshhart 15y ago
  Yes. It's a partitioned system - you just need to have enough replicas per partition. Clients track cluster state by registering listeners to zookeeper.
andres 15y ago
SenseiDB looks really impressive. We were actually thinking about the same problem when we built ThriftDB (http://www.thriftdb.com http://www.thriftdb.com) for Octopart so it's especially interesting to see LinkedIn's implementation. Looking through the documentation a couple of things come to mind: * Does SenseiDB support nested data structures? * What happens when you modify the schema? Can you add/delete/modify attributes?
- javasoze 15y ago
  re: nested data structure is in the roadmap, but no target date yet. There are 2 ways of implementing this: 1) pre-pend field name reflecting the nesting, much like elasticsearch 2) encoding payloads in the term list indicating nested nature. 1) is straightforward, but can be restrictive if you want to in-nest faceting. We are still debating on the route to take. As for schema changes, we should document that better: certain schema changes simply requires you bouncing the node, but if there are data integrity changes, you would need to reindex. We will work on more documentation on the specifics.
  - ericmoritz 15y ago
    If I change the faceting does bouncing the nodes cause a re-index automatically or is it more involved?
    - javasoze 15y ago
      bouncing node will start reindexing from the last committed version on the disk. If the index is up to date, no re-indexing will be triggered.
franklovecchio 15y ago
Looking at its config, I'm reminded of Cassandra 0.6, but then I see the ZooKeeper functionality. Interesting. Love the open-sourcing! I showed an example of how to distribute inter-mingled DBs using MQtt a bit ago, but never posted here: http://franklovecchio.tumblr.com/post/13051814890/mqdb-distributing-databases-with-mqtt http://franklovecchio.tumblr.com/post/13051814890/mqdb-distr.... One thing is for sure -- NoSQL isn't so mysterious anymore :)
agentultra 15y ago
Looks like the marketing copy is off from the actual implementation, AFAICT. Sensei (先生) means teacher or professor in Japanese(http://en.wikipedia.org/wiki/Sensei). It shares the same pronunciation and writing with the Chinese word that has the same meaning. This name indicates that the system can be used in place of Oracle database in many applications. Okay, so on the name alone I can replace my Oracle database! Great! Seriously though it goes on... They claim the database is ACID: http://javasoze.github.com/sensei/data-guarantee.html http://javasoze.github.com/sensei/data-guarantee.html But they built the entire thing around "eventual consistency." And statements like: "Sensei provides a high-level of durability by maintaining N replicas of each shard to guarantee a level of availability and fault-tolerance" Don't seem to make sense when talking about ACID given that a write operation will happen at some point. Looks like the data event producers will shard the data across N replicas without quorum... so there's no guarantee that there will be N replicas available... is that right (and that the transaction won't be lost mid-stream either)? Skimming through the source it doesn't seem to be doing anything terribly revolutionary... and I can see the usefulness of the trade-offs they made in this database for certain scenarios. However I don't think the claims of ACID guarantees and "real time" are particularly representative of what this DB will actually do. They just don't seem to jive with "eventual consistency" models. I'm not a hardcore database guru though so maybe I'm missing something?
- joshhart 15y ago
  Where does that page claim perfect ACID semantics? It's meant to describe what Sensei gives you for each aspect of ACID. Sensei needs an event stream to process. We've open-sourced and apache-fied Kafka which is a great candidate for an event stream. For Atomicity and Isolation, the event stream must provide these guarantees. Consistency is handled with a routing parameter. Requests partitioned around an id will always go to the same searcher, so they won't go backwards in the stream except in failure scenarios. This is eventually consistent, but tries to keep things sane. Durability: The event stream helps with this. We don't immediately flush while indexing in Lucene, so if there's a crash we can replay the persistent event stream. Does this make sense? SenseiDB is not intended for purely transactional processing. For some applications, sensei would make a good candidate for replacing your DB. For others, not so much.
javajosh 15y ago
Options are great (and thanks for contributing to the OS community), but the biggest barrier to adoption will be confusion about all the NoSQL solutions floating around right now (including two others from LinkedIn). Therefore, I recommend that you spend more time on the homepage comparing Sensei with other NoSQL stores. Perhaps take a two-tier approach for those coming from Oracle or MySQL: why NoSQL, and if NoSQL, why Sensei NoSQL?
ericmoritz 15y ago
So to get data into this database, I have to have a Gateway implementation in Java? Would I have to publish messages to Kafka in my language of choice to avoid writing any Java?
- javasoze 15y ago
  There is a Kafka gateway packaged with Sensei, see sensei-gateways module. We love Kafka!
foobarbazetc 15y ago
This is awesome. Thanks for sharing this with the world. :)

163 points by thomas11 15y ago

A truly amazing peace of technology

untog 15y ago

I can't speak for SenseiDB itself, but I think that LinkedIn deserves a lot of credit for the technology they have been open sourcing in the last few months. Fantastic to see.

joshhart 15y ago

Looks like someone non-LinkedIn posted this so the rest of the team is unlikely to be around for a few more hours. Let me know if you have any questions! I worked mostly on the distribution & rpc piece.

threepointone 15y ago
Congratulations on the project, and compliments on the breadth of documentation. Nicely done. (Nitpick - none of the urls change. Some js error where you're not doing a pushState? You should report this.) edit - this works -http://senseidb.github.com/sensei/index.html http://senseidb.github.com/sensei/index.html
- joshhart 15y ago
  Thanks! I'll make sure that gets corrected.
lemming 15y ago
How does the consistency model work? I saw the page describing the eventual consistency referred to a routing parameter (which I assumes routes to the same node to enforce read-your-writes), but I assume you can have conflicts between writes from separate sessions? How is the conflict resolution handled?
- joshhart 15y ago
  This is exactly how it works. Queries based on some kind of id will always hit the same server, except when the server goes down. Sensei expects a semi-durable "data stream" for writes, which is very different from how most other databases work. Imagine a near-real time processing system though where updates go into a queue to be processes (financial transactions work like this). SenseiDB works great for indexing these types of events. Text search is a good example here - after you update your profile, it's put into a queue to be indexed. We don't want to wait for your profile to be indexed before we'll let you move onto another page.

hello_moto 15y ago

How many home-grown databases have linkedin developed in the past few years?

javasoze 15y ago
A few... :) I think each has its own strengths and weaknesses and we use them in different projects with different requirements.
strlen 15y ago
How many programming languages do you use? :-) Just like we use Python, Java, Scala, etc... where each is appropriate, we also use different databases where appropriate. Each system serves a need. We also use off the shelf databases, where appropriate: it should be noted Sensei uses Lucene (a well known search library), Voldemort has a pluggable storage engine (where we mostly use BerkeleyDB and a custom read-only storage engine), and Espresso uses MySQL. We don't build databases for reasons of NIH: we focus on building features (faceting, real-time indexing, partitioning, fail over, etc...) that enable us to build fast, usable, feature rich, scalable, and reliable applications. We readily use open source components in many places within both our infrastructure systems (search, various databases, Kafka) and in the applications, and contribute to existing open source projects.

zrail 15y ago

From a quick glance at the docs there's no database-side aggregations. Is that correct?

joshhart 15y ago
The facets let you support counting and group by - is there something else you had in mind? Edit: You can get a pretty quick idea of what's supported the query language at http://senseidb.github.com/sensei/bql.html http://senseidb.github.com/sensei/bql.html. Since Sensei was designed with document and text uses cases in mind, we have plans to support advanced queries where a relevance model is part of the query, allowing you to perform a custom sort on the server.
- zrail 15y ago
  sum() and distinct() come to mind for sure, but I don't know how easy they'd be to implement. The particular use case I have in mind is where we have events streaming in and we usually don't care about the actual events, just various sums, counts and distinct counts over them grouped by various dimensions.
javasoze 15y ago
Database-side aggregation is something in the works. We are incorporating a patch as we speak. We find more and more in house projects using Sensei as a BI tool, we see this functionality more and more important, so stay tuned :)
- DEinspanjer 15y ago
  Would you guys be interested in collaborating on that? At Mozilla, we've been using ElasticSearch in a few places as a NoSQL style BI tool. I read through the SenseiDB docs and can see many places it could be almost as good of a fit as ES, and with a little work, I bet it could be just the thing.
  - javasoze 15y ago

opendomain 15y ago

3 separate NoSQL stores from ONE company? And their primary business is not Databases? There are currently more than 100 different NoSQL solutions - I believe we will see a consolidation of NoSQL in 2012

efnx 15y ago
On the flip side, we might see a lot of "build your own NoSQL store" tutorials in 2012.

dougk 15y ago

Can Sensei be used as a replacement to MySQL + lucene?

LukeHoersten 15y ago
Yeah, it'd be nice to see a side-by-side against other common DBs like MySQL and Oracle.
- deleted 15y ago
  [deleted]
- Ecio78 15y ago
  there's a comparison vs Mysql here: http://senseidb.github.com/sensei/performance.html http://senseidb.github.com/sensei/performance.html for insert and query performance

jamesu 15y ago

Nice looking wheel. I have a feeling it could be rounder though.

deleted 15y ago
[deleted]
- deleted 15y ago
  [deleted]

cultureulterior 15y ago

Is it single-point-of-failure free?

joshhart 15y ago
Yes. It's a partitioned system - you just need to have enough replicas per partition. Clients track cluster state by registering listeners to zookeeper.

andres 15y ago

SenseiDB looks really impressive. We were actually thinking about the same problem when we built ThriftDB (http://www.thriftdb.com http://www.thriftdb.com) for Octopart so it's especially interesting to see LinkedIn's implementation. Looking through the documentation a couple of things come to mind: * Does SenseiDB support nested data structures? * What happens when you modify the schema? Can you add/delete/modify attributes?

javasoze 15y ago
re: nested data structure is in the roadmap, but no target date yet. There are 2 ways of implementing this: 1) pre-pend field name reflecting the nesting, much like elasticsearch 2) encoding payloads in the term list indicating nested nature. 1) is straightforward, but can be restrictive if you want to in-nest faceting. We are still debating on the route to take. As for schema changes, we should document that better: certain schema changes simply requires you bouncing the node, but if there are data integrity changes, you would need to reindex. We will work on more documentation on the specifics.
- ericmoritz 15y ago
  If I change the faceting does bouncing the nodes cause a re-index automatically or is it more involved?
  - javasoze 15y ago
    bouncing node will start reindexing from the last committed version on the disk. If the index is up to date, no re-indexing will be triggered.

franklovecchio 15y ago

Looking at its config, I'm reminded of Cassandra 0.6, but then I see the ZooKeeper functionality. Interesting. Love the open-sourcing! I showed an example of how to distribute inter-mingled DBs using MQtt a bit ago, but never posted here: http://franklovecchio.tumblr.com/post/13051814890/mqdb-distributing-databases-with-mqtt http://franklovecchio.tumblr.com/post/13051814890/mqdb-distr.... One thing is for sure -- NoSQL isn't so mysterious anymore :)

agentultra 15y ago

Looks like the marketing copy is off from the actual implementation, AFAICT. Sensei (先生) means teacher or professor in Japanese(http://en.wikipedia.org/wiki/Sensei). It shares the same pronunciation and writing with the Chinese word that has the same meaning. This name indicates that the system can be used in place of Oracle database in many applications. Okay, so on the name alone I can replace my Oracle database! Great! Seriously though it goes on... They claim the database is ACID: http://javasoze.github.com/sensei/data-guarantee.html http://javasoze.github.com/sensei/data-guarantee.html But they built the entire thing around "eventual consistency." And statements like: "Sensei provides a high-level of durability by maintaining N replicas of each shard to guarantee a level of availability and fault-tolerance" Don't seem to make sense when talking about ACID given that a write operation will happen at some point. Looks like the data event producers will shard the data across N replicas without quorum... so there's no guarantee that there will be N replicas available... is that right (and that the transaction won't be lost mid-stream either)? Skimming through the source it doesn't seem to be doing anything terribly revolutionary... and I can see the usefulness of the trade-offs they made in this database for certain scenarios. However I don't think the claims of ACID guarantees and "real time" are particularly representative of what this DB will actually do. They just don't seem to jive with "eventual consistency" models. I'm not a hardcore database guru though so maybe I'm missing something?

joshhart 15y ago
Where does that page claim perfect ACID semantics? It's meant to describe what Sensei gives you for each aspect of ACID. Sensei needs an event stream to process. We've open-sourced and apache-fied Kafka which is a great candidate for an event stream. For Atomicity and Isolation, the event stream must provide these guarantees. Consistency is handled with a routing parameter. Requests partitioned around an id will always go to the same searcher, so they won't go backwards in the stream except in failure scenarios. This is eventually consistent, but tries to keep things sane. Durability: The event stream helps with this. We don't immediately flush while indexing in Lucene, so if there's a crash we can replay the persistent event stream. Does this make sense? SenseiDB is not intended for purely transactional processing. For some applications, sensei would make a good candidate for replacing your DB. For others, not so much.

javajosh 15y ago

Options are great (and thanks for contributing to the OS community), but the biggest barrier to adoption will be confusion about all the NoSQL solutions floating around right now (including two others from LinkedIn). Therefore, I recommend that you spend more time on the homepage comparing Sensei with other NoSQL stores. Perhaps take a two-tier approach for those coming from Oracle or MySQL: why NoSQL, and if NoSQL, why Sensei NoSQL?

ericmoritz 15y ago

So to get data into this database, I have to have a Gateway implementation in Java? Would I have to publish messages to Kafka in my language of choice to avoid writing any Java?

javasoze 15y ago
There is a Kafka gateway packaged with Sensei, see sensei-gateways module. We love Kafka!

foobarbazetc 15y ago

This is awesome. Thanks for sharing this with the world. :)