7 ms·

CrimsonDB: A Self-Designing Key-Value Store

84 points by ngaut 8y ago

pacuna 8y ago
Sounds kind of similar to CMU's Peloton.
graycat 8y ago
I keep reading about key-value stores with this and that property. But for the Web site for my startup, early on it was clear that I needed a good key-value store as a Web user session state store (server). So, I designed one, coded it up, and have been using it. It seems to work fine. But on key-value stores, I don't get it because what I did was easy, too easy, really, to say much about it except how easy it was to do. So, a Web server has a user and their session state. For the user, I also have a GUID (globally unique identifier) that is the key for the user and their session. The session state is in an instance of a class I've defined. So, the Web server serializes the instance of the class to a byte array and passes that and the GUID key to a subroutine that uses TCP/IP sockets to start a TCP/IP session with my session state store. To Windows, my session state store is just a simple, console application. The session state store is listening for a TCP/IP connection request, accepts the request, and gets the GUID key and serialized object instance with the session state. The session state store has two instances of a standard .NET collection class, say, A and B. Instance A uses the GUID as the key and the serialized session state as the value. Class B uses the then current system time-date (GMT) as the key and the GUID as the value. So, whenever the session state store runs, before it returns, it looks at the earliest time-date stamp in instance B and deletes from A and B all the key-value pairs that are too old, that is, timed out, that is, have encountered session time out. The code of the session state store is simple -- single threaded. Then the code needs a first-in, first-out (FIFO) queue, and for that I am using just what TCP/IP offers with a maximum queue depth of 100 which I hope is enough -- will be able to tell by looking at the site log file. That's about it. It's simple. It's darned fast and nicely scalable via virtual memory and, then, sharding, etc. I designed the whole thing in a few minutes and implemented it in about a day, but I'm no expert in session state stores, key-value stores, Redis, etc. So, what is wrong with what I designed? Thanks.
- StreamBright 8y ago
  >>> But for the Web site for my startup, early on it was clear that I needed a good key-value store as a Web user session state store (server). So, I designed one, coded it up, and have been using it. It seems to work fine. Well depending on what error scenarios you might run into your code will work until it runs into those scenarios. Few ideas what could go wrong you can get from this video: https://www.youtube.com/watch?v=4fFDFbi3toc https://www.youtube.com/watch?v=4fFDFbi3toc
  - graycat 8y ago
    Thanks for the video. Yup. So far the server side of my Web site has servers of 6 types -- Web server, session state store (server), log server, SQL Server, and two core servers for the real work, doing some applied math to get the users the good stuff that will make them happy. Soon there will also be an ad server for a total of 7. As in the sense of the video, that sounds like a non-deterministic hell to debug. Ah, in this case, I don't think the debugging is so difficult. Mostly the reason is that so far all the interactions are conceptually and as in the architecture quite simple. E.g., the code seems to run fine, and I never encountered any significant problems getting the code, first version and first production version, running fine. But I do intend to add to the code lots more writes to the log server and be able to turn on/off those writes while the system is live. Then I could write some code to look at the log file and for each user session check all the traffic and work. Yes, things could fail. I do have a lot of reasonable value checks in the code and, thus, could get some diagnostics then. Eventually I'll want to implement some fault tolerance: E.g., if a log server becomes unresponsive, then switch to another one -- there maybe several anyway due to load and desire for redundancy. Similarly for the session state store -- a user might lose their session, but that would get logged, but the Web server would then start to use a different session state server. So far my site is simple enough in purpose, work, architecture, etc. that some reasonably careful thinking, documentation, coding, and testing should continue to be okay. But, sure, if the site gets to 1000 users a second, I'll have to rethink the architecture in big ways. Then I'll try to keep using the "keep it simple, stupid" KISS principle and be willing to waste a factor of 2 to 10 or so in checking and being sure and being able to do the right things for some failure scenarios. Uh, my Web site is not for banking, exchanging crypto tokens, etc.!!! Thanks, especially for the video. Right, for non-deterministic systems, can be tough to duplicate a problem in order to analyze it! So, tough, in practice, try never to have to confront directly the non-determinism -- i.e., have enough in tracing in logs that have in effect converted the work to something deterministic, the randomness removed. So, first cut lessons, try to avoid non-deterministic designs; if have one, then try to keep it dirt simple and likely relatively isolated from everything else; and if can't do all that, then during development and testing, at various levels, have lots of scaffolding and for production put in lots of checks, redundancy, thought, system management, monitoring, reporting, and analysis infrastructure, etc. E.g., I did and published some high end work in anomaly detection for real time monitoring to detect problems never seen before, and I intend to use that work when appropriate. The intuitive idea is that if the system is not humming like it usually does and should, then I'll discover that right away and, then, be able to zoom down a tree of partitions of the systems to localize the problem -- hopefully in a usefully large fraction of such cases. My work is good because it is multi-dimensional, distribution-free, has adjustable, known in advance, exact control over false alarm rate, and in a useful sense is the most powerful such monitoring possible from any means of processing the data. No way should ever have to see that the system occasionally gets sick and just look at the system, in racks in 100,000 square feet of floor space, and start to guess. Instead should have a LOT of darned good additional data from the various over-designed internal checks, logs, redundancies, etc. Really, the system should come close to just making it obvious about where the problem is. No joke: In 100,000 square feet of racks of humming servers, want some darned good health and wellness monitoring that, at least in the design, architecture, development, testing, and first months of production operation, appears to be able to localize problems. Suggesting that such work is important; not suggesting that it's easy or will get it good enough right away. Am suggesting that should take the problem seriously and try. For some important systems, may spend several times as much computing checking and monitoring as doing the actual production work. But no way do I ever want to have to know there is something wrong and have to be forced just to look at 100,000 square feet of servers and start with Ethernet diagnostic hardware, reading files, etc. I hope never have to consider a main memory dump. My first development computer got flaky -- I became able just to smell flaky. It really stinks!
jensenbox 8y ago
Can we see some source code for this project?

84 points by ngaut 8y ago

Sounds kind of similar to CMU's Peloton.

I keep reading about key-value stores with this and that property. But for the Web site for my startup, early on it was clear that I needed a good key-value store as a Web user session state store (server). So, I designed one, coded it up, and have been using it. It seems to work fine. But on key-value stores, I don't get it because what I did was easy, too easy, really, to say much about it except how easy it was to do. So, a Web server has a user and their session state. For the user, I also have a GUID (globally unique identifier) that is the key for the user and their session. The session state is in an instance of a class I've defined. So, the Web server serializes the instance of the class to a byte array and passes that and the GUID key to a subroutine that uses TCP/IP sockets to start a TCP/IP session with my session state store. To Windows, my session state store is just a simple, console application. The session state store is listening for a TCP/IP connection request, accepts the request, and gets the GUID key and serialized object instance with the session state. The session state store has two instances of a standard .NET collection class, say, A and B. Instance A uses the GUID as the key and the serialized session state as the value. Class B uses the then current system time-date (GMT) as the key and the GUID as the value. So, whenever the session state store runs, before it returns, it looks at the earliest time-date stamp in instance B and deletes from A and B all the key-value pairs that are too old, that is, timed out, that is, have encountered session time out. The code of the session state store is simple -- single threaded. Then the code needs a first-in, first-out (FIFO) queue, and for that I am using just what TCP/IP offers with a maximum queue depth of 100 which I hope is enough -- will be able to tell by looking at the site log file. That's about it. It's simple. It's darned fast and nicely scalable via virtual memory and, then, sharding, etc. I designed the whole thing in a few minutes and implemented it in about a day, but I'm no expert in session state stores, key-value stores, Redis, etc. So, what is wrong with what I designed? Thanks.

StreamBright 8y ago
>>> But for the Web site for my startup, early on it was clear that I needed a good key-value store as a Web user session state store (server). So, I designed one, coded it up, and have been using it. It seems to work fine. Well depending on what error scenarios you might run into your code will work until it runs into those scenarios. Few ideas what could go wrong you can get from this video: https://www.youtube.com/watch?v=4fFDFbi3toc https://www.youtube.com/watch?v=4fFDFbi3toc
- graycat 8y ago
  Thanks for the video. Yup. So far the server side of my Web site has servers of 6 types -- Web server, session state store (server), log server, SQL Server, and two core servers for the real work, doing some applied math to get the users the good stuff that will make them happy. Soon there will also be an ad server for a total of 7. As in the sense of the video, that sounds like a non-deterministic hell to debug. Ah, in this case, I don't think the debugging is so difficult. Mostly the reason is that so far all the interactions are conceptually and as in the architecture quite simple. E.g., the code seems to run fine, and I never encountered any significant problems getting the code, first version and first production version, running fine. But I do intend to add to the code lots more writes to the log server and be able to turn on/off those writes while the system is live. Then I could write some code to look at the log file and for each user session check all the traffic and work. Yes, things could fail. I do have a lot of reasonable value checks in the code and, thus, could get some diagnostics then. Eventually I'll want to implement some fault tolerance: E.g., if a log server becomes unresponsive, then switch to another one -- there maybe several anyway due to load and desire for redundancy. Similarly for the session state store -- a user might lose their session, but that would get logged, but the Web server would then start to use a different session state server. So far my site is simple enough in purpose, work, architecture, etc. that some reasonably careful thinking, documentation, coding, and testing should continue to be okay. But, sure, if the site gets to 1000 users a second, I'll have to rethink the architecture in big ways. Then I'll try to keep using the "keep it simple, stupid" KISS principle and be willing to waste a factor of 2 to 10 or so in checking and being sure and being able to do the right things for some failure scenarios. Uh, my Web site is not for banking, exchanging crypto tokens, etc.!!! Thanks, especially for the video. Right, for non-deterministic systems, can be tough to duplicate a problem in order to analyze it! So, tough, in practice, try never to have to confront directly the non-determinism -- i.e., have enough in tracing in logs that have in effect converted the work to something deterministic, the randomness removed. So, first cut lessons, try to avoid non-deterministic designs; if have one, then try to keep it dirt simple and likely relatively isolated from everything else; and if can't do all that, then during development and testing, at various levels, have lots of scaffolding and for production put in lots of checks, redundancy, thought, system management, monitoring, reporting, and analysis infrastructure, etc. E.g., I did and published some high end work in anomaly detection for real time monitoring to detect problems never seen before, and I intend to use that work when appropriate. The intuitive idea is that if the system is not humming like it usually does and should, then I'll discover that right away and, then, be able to zoom down a tree of partitions of the systems to localize the problem -- hopefully in a usefully large fraction of such cases. My work is good because it is multi-dimensional, distribution-free, has adjustable, known in advance, exact control over false alarm rate, and in a useful sense is the most powerful such monitoring possible from any means of processing the data. No way should ever have to see that the system occasionally gets sick and just look at the system, in racks in 100,000 square feet of floor space, and start to guess. Instead should have a LOT of darned good additional data from the various over-designed internal checks, logs, redundancies, etc. Really, the system should come close to just making it obvious about where the problem is. No joke: In 100,000 square feet of racks of humming servers, want some darned good health and wellness monitoring that, at least in the design, architecture, development, testing, and first months of production operation, appears to be able to localize problems. Suggesting that such work is important; not suggesting that it's easy or will get it good enough right away. Am suggesting that should take the problem seriously and try. For some important systems, may spend several times as much computing checking and monitoring as doing the actual production work. But no way do I ever want to have to know there is something wrong and have to be forced just to look at 100,000 square feet of servers and start with Ethernet diagnostic hardware, reading files, etc. I hope never have to consider a main memory dump. My first development computer got flaky -- I became able just to smell flaky. It really stinks!

jensenbox 8y ago

Can we see some source code for this project?