8 ms·
Redis re-implemented with SQLite
- sudhirj 3y agoDid this for DynamoDB over the pandemic. Helped me learn both the Redis API and DynamoDB. https://github.com/dbProjectRED/redimo.go https://github.com/dbProjectRED/redimo.go Want to do the same thing with Postgres as well. The Redis API and data structures are really nice and have been tuned to be useful in a wide variety of situations. Implementing these APIs is different concurrent, compute and storage models is a nice way to learn them and get options open.
- PeterZaitsev 3y agoI think what would be interesting is integrating with dsqlite or other replicated solution. Performance is no the only requirement
- r14c 3y agoI'll be keeping a close eye on this project. the more sqlite in my life the better.
- akie 3y agoI would love to have a Redis alternative where I don't have to think about whether or not the dataset fits into memory.
- deleted 3y ago[deleted]
- jetbalsa 3y agoI've used SSDB[0] in the past for some really stupid large datasets (20TB)_and it worked really well in production [0] https://github.com/ideawu/ssdb https://github.com/ideawu/ssdb
- PlutoIsAPlanet 3y agoIts also worth checking out kvrocks, which is a redis interface on top of rockdb that's part of the Apache project, and very well maintained.
- mathfailure 3y agoAnd which is not in-memory at all.
- PlutoIsAPlanet 3y agoIt can cache in-memory using RocksDBs caching mechanisms.
- welder 3y agoI switched from SSDB to Kvrocks recently, because SSDB is abandoned and the author missing for 3 years now. I used to recommend SSDB, but now there's better alternatives available: https://github.com/apache/kvrocks https://github.com/apache/kvrocks https://github.com/sabledb-io/sabledb https://github.com/sabledb-io/sabledb
- akie 3y agoThese are great recommendations, thanks!
- yuppiepuppie 3y agoCurious, What’s the use case?
- qwertox 3y agoMy same thought, because some don't require much memory.
- akie 3y agoThe use case is caching 20 million API responses that almost never change, each about 20kb of JSON, for a high traffic site. Yes, I can pay for a 400Gb RAM instance of Redis, but it's expensive. I can also cache it on disk, but then I need to think about cache expiration myself. Or I can use something appropriate like a document database, but then I need additional code & additional configuration because we otherwise don't need that piece of infrastructure in our stack. It would be a lot easier if I could just store it in Redis with the other (more reasonably sized) things that I need to cache.
- emmanueloga_ 3y agoCheck memcached "extstore" [1]. -- 1: https://github.com/memcached/memcached/wiki/Extstore https://github.com/memcached/memcached/wiki/Extstore
- Nican 3y agoThis looks like a good use case for ScyllaDB with Compression and TTL. It is pretty simple to setup a single-node instance. If you rather have something in-process and writes to disk, to avoid extra infrastructure, I would also recommend RocksDB with Compression and TTL.
- 0cf8612b2e1e 3y agoWould DiskCache work for you? Runs via SQLite memory or persisted file database. Thread safe, has various expiration controls, etc. https://grantjenks.com/docs/diskcache/tutorial.html https://grantjenks.com/docs/diskcache/tutorial.html
- GraemeMeyer 3y agoI think Garnet might be what you're looking for: https://www.microsoft.com/en-us/research/project/garnet/ https://www.microsoft.com/en-us/research/project/garnet/
- Nican 3y agoAs usual, there is a spectrum of data safety vs. performance. Redis is at the "very fast, but unsafe" side of the scale. ScyllaDB for me is in the middle of being high performance key-value store, but not really supporting transactions. FoundationDB is another one that I would consider.
- tyre 3y agoDepends on the kind of safety you’re looking for. Redis is entirely safe from concurrency issues because it’s single-threaded. It supports an append-only file for persistence to disk.
- alerighi 3y agoAt that point why using Redis entirely? You can use any DBMS you want, either relational or NoSQL. The advantage of Redis is that it is a memory cache, if you take out the memory from it, just use Postgres or whatever DBMS you are using (I say Postgres because it has all the features of Redis).
- dalyons 3y agoPostgres has nowhere near all the features of redis. Go and have a look at the redis command’s documentation. They’re not even really similar at all, once you get past basic GET/SET stuff.
- from-nibly 3y agoCan you name an explicit thing that postgres does not do that redis does?
- jitl 3y agoThis is silly, Postgres doesn’t speak the Redis wire protocol. You will need a large army of connection proxies to get a Postgres database to handle the number of connections a single Redis shrugs off with no sweat. Maybe you like this answer more: At the end of the day you can embed a bunch of Turing-complete programming languages in Postgres, and Postgres can store binary blobs, so Postgres can do literally anything. Can it do it performantly, and for low cost? Probably not. But if you put in enough time and money I’m sure you can re-implement Redis on Postgres using BLOB column alone. Here’s a simpler answer: cuckoo filter is available out of the box in Redis, 2 seconds of Googling I didn’t find one for Postgres: https://redis.io/docs/latest/develop/data-types/probabilistic/cuckoo-filter/ https://redis.io/docs/latest/develop/data-types/probabilisti...
- codetrotter 3y agoNot sure if this one could be used in order to do what you want but maybe? https://www.postgresql.org/docs/current/bloom.html https://www.postgresql.org/docs/current/bloom.html Have a look
- jhatemyjob 3y agoThis is a great idea and I am glad it is BSD licensed. Unfortunately the execution is somewhat lacking. SQLite is best suited for embedded / clientside applications with minimal dependencies. The author of this project decided to use Go and make it a service.
- nalgeon 3y agoDid I? > Both in-process (Go API) and standalone (RESP) servers. In-process means that the database is "embedded / clientside" in your terms.
- jhatemyjob 3y agoIt's a server.
- orthecreedence 3y agoDo you usually embed redis?
- jhatemyjob 3y agoI would like to.
- orthecreedence 3y agoYou want to use an in-mem database like SQLite or DuckDB or even replicate the data structures in-code that redis uses. Redis doesn't really add a whole lot to the table over a general programming language with a good standard library other than being shared over a network.
- jhatemyjob 3y agoExactly, the standard library is what I'm after. I am currently using SQLite but I prefer the redis interface
- nalgeon 3y agoI'm a big fan of both Redis and SQLite, so I decided to combine the two. SQLite is specifically designed for many small queries[1], and it's probably as close as relational engines can get to Redis, so I think it might be a good fit. [1]: https://sqlite.org/np1queryprob.html https://sqlite.org/np1queryprob.html
- sesm 3y agoWhat are the project goals? I assume it's a drop-in replacement for Redis that is supposed to be better in certain cases? If yes, then what cases do you have in mind?
- nalgeon 3y agoThe goal is to have a convenient API to work with common data structures, with an SQL backend and all the benefits it provides. Such as: — Small memory footprint even for large datasets. — ACID transactions. — SQL interface for introspection and reporting.
- sesm 3y agoSo the goal is to have a Redis-like API but not actually be an in-memory data store and reduce memory consumption this way? For example, for a project/service that started with Redis, but then priorities shifted and small memory footprint became more important than performance? Did I get it right?
- grobbyy 3y agoI can tell you my use case. I have a lot of systems which I'd like to use on a single computer AND at scale. That's sometimes even just for development work. A lot of these use a common API to a more complex distributed store, as well as to something simple like files on disk, in memory, or SQLite. I'm most cases, it's one user at a time, so performance doesn't matter, but simplicity does. It can also be for the project which has a 1 percent chance of going viral. Etc. But I find relatively few cases between truly small scale and large scale.
- tehbeard 3y ago> reimplement the good parts of Redis Seems to be missing streams, hyperloglog and pubsub though, so mostly just the kv part of the side protocol with a different backend?
- nalgeon 3y agoCan't fit everything in 1.0, has to start with something. If the community is interested in the project, there will be more.
- poidos 3y agoWonderful idea and execution!
- kiitos 3y agoThe entire value proposition of Redis is that it operates out of memory, and therefore has memory-like performance. (edit: And doesn't provide the benefit of, and therefore pay the costs related to, ACID-like consistency guarantees.) If you move it to disk (edit: Or try to assert ACID-like consistency or transactional guarantees) there's little reason to use Redis any more.
- qbane 3y agoSQLite does allow one to keep the entire database in memory: https://www.sqlite.org/inmemorydb.html https://www.sqlite.org/inmemorydb.html
- j-pb 3y agoBut is still orders of magnitude slower than a hash-map.
- SJC_Hacker 3y agoIf you want ordering, then a hash-map doesn't help you.
- j-pb 3y agoStill orders of magnitude slower than a B-Tree Map.
- egeozcan 3y agoYou can also create an in-memory sqlite database though.
- PhilipRoman 3y ago>The entire value proposition of Redis is that it operates out of memory Not familiar with Redis specifically, but I doubt this idea. You can run anything on top of a ramdisk (granted, you can save a few pointer additions and get rid of some safety checks if you know you're working with memory)
- larodi 3y agoPotentially many things like session mgmt, queues, document graphs, etc, can be done right with simple facilities like tables. Tables represent sets, and set algebra seems very common in data representations. Thing is how the sets are combined, i.e. related. This' essentially API-to-SQL-in-Redis-clothes. Kudos to the author.
- nalgeon 3y agoThank you! I also think that the relational model can get you pretty far if you don't need to squeeze every last bit of performance out of the program. And the added benefit of using a battle-tested SQL engine is far fewer storage-related bugs.
- nasretdinov 3y agoBy the way, I noticed you're using SetMaxConnections(1), however in WAL mode (which you're using) SQLite does support writes that don't block reads, so you might benefit from allowing read concurrency (in theory).
- nalgeon 3y agoYeah, it's explained in the code[1] SQLite only allows one writer at a time, so concurrent writes will fail with a "database is locked" (SQLITE_BUSY) error. There are two ways to enforce the single writer rule: 1. Use a mutex for write operations. 2. Set the maximum number of DB connections to 1. Intuitively, the mutex approach seems better, because it does not limit the number of concurrent read operations. The benchmarks show the following results: - GET: 2% better rps and 25% better p50 response time with mutex - SET: 2% better rps and 60% worse p50 response time with mutex Due to the significant p50 response time mutex penalty for SET, I've decided to use the max connections approach for now. [1]: https://github.com/nalgeon/redka/blob/main/internal/sqlx/db.go#L78 https://github.com/nalgeon/redka/blob/main/internal/sqlx/db....
- nasretdinov 3y agoHow about having two pools, one for writes only, and the other one for reads? SQLite allows you to open the DB more than in one thread per application, so you can have a read pool and a write pool with SetMaxConnections(1) for better performance. This of course also means that reads should be handled separately from writes in the API layer too.
- nalgeon 3y agoThought about it, decided to start with simpler and good enough option. The goal here is not to beat Redis anyway.
- acrispino 3y agofor what it's worth, the two pool approach is suggested here by a collaborator to github.com/mattn/go-sqlite3: https://github.com/mattn/go-sqlite3/issues/1179#issuecomment-1638083995 https://github.com/mattn/go-sqlite3/issues/1179#issuecomment...
- TheChaplain 3y agoHmm, am I the only one who is not worried? Although I don't really see anything in the license change that would prevent me from using it at both home and business, Redis seem "complete" functionality wise so using a pre-license-change version can't hurt even long-term I think.
- jitl 3y agoI’m not sure to what degree you want to follow the Redis no concurrency “everything serialized on one thread” model. You can get substantially better performance out of sqlite by using the lower level https://github.com/crawshaw/sqlite https://github.com/crawshaw/sqlite, turning on WAL etc, using a connection per goroutine for reads, and sending batches of writes over a buffered channel / queue to a dedicated writer thread. That way you can turn off SQLite’s built in per-connection mutex but still be thread safe since each connection is only used on a single thread at a time. For this use-case you will also probably save a lot of time if you use some large arena-style buffers (probably N per conn?) and copy incoming parameter bytes from the network request/socket to the buffer, or copy straight from sqlite out to the socket, instead of allocating and passing around a bunch of individual strings. Boxing those strings in interface{} (as done by the high level sql stdlib) slows things down even more. None of this is necessary to get usable perf, even decently good perf, just sharing some tips from my experience trying to get absolutely maximum write throughput from SQLite in Golang.
- nalgeon 3y agoGreat tips, thank you! The thing is, getting maximum throughput is not the goal of the project (at least not at this stage). I'm using reasonable SQLite defaults (including WAL), but that's it for now.
- SJC_Hacker 3y ago> You can get substantially better performance out of sqlite by using the lower level https://github.com/crawshaw/sqlite https://github.com/crawshaw/sqlite, turning on WAL etc, using a connection per goroutine for reads, and sending batches of writes over a buffered channel / queue to a dedicated writer thread. That way you can turn off SQLite’s built in per-connection mutex but still be thread safe since each connection is only used on a single thread at a time. Would this protect against a row update in the middle of a read? e.g. would a row at least be internally consistent at the time it was read?
- lanstin 3y agoYes you would still get SQLite file locking; the thing you aren't using is SQLite thread locking around its own structures (according to GP, I don't know myself). But be sure not to use NFS mount for the files, the locking there is not that great.
- xrd 3y agoGo plus SQLite is producing some terrific projects. I love Pocketbase and this looks great as well.
- sitkack 3y agohttps://github.com/pocketbase/pocketbase https://github.com/pocketbase/pocketbase
- surfingdino 3y agoBack when Foursquare made MongoDB famous someone posted a PoC of a NoSQL DB implemented in MySQL. It did not seem to have caught on, but it did make me think of how much performance is traded for helping us not to reinvent SQL every time we need a DB. I like experiments like this one, they sometimes lead to new projects.
- coredog64 3y agoI believe DynamoDB uses MySQL under the hood as it’s storage engine. Or, if it doesn’t currently that it did for several years.
- efilife 3y agoIts* https://youryoure.com/?its https://youryoure.com/?its
- ukuina 3y agoPlease do not post replies that are trivial spelling or grammatical corrections; they add little to the discourse. To parent: I think all datastores can boil down to a SQL interface eventually. Someday a SQL-interface may dynamically choose a bespoke backend for your schema (or even sub-schema!)
- redskyluan 3y agoI would personally not recommend implementing a Redis protocol on top of SQLite, as I've seen too many failed cases like this. Users may perceive your product as a drop-in Redis replacement, and it may work fine in a PoC, but once it hits production, SQLite's performance and scalability will be severely challenged. It's much better to use RocksDB as the underlying storage engine for such a solution. RocksDB can provide the performance and scalability required.If you need a distributed solution, I would suggest looking at TiKV or FoundationDB. These are both excellent distributed storage systems that can handle production workloads much better than a SQLite-based approach.
- marcrosoft 3y agoThis “SQLite isn’t good for production” mentality really needs to die. It’s a myth.
- mrinterweb 3y agoA lot redis's fantastic performance evaporates when network IO is added to the equation. Of course, you can host your own redis, but as soon as people start using a SaaS hosted redis service, performance usually takes a huge hit. If this makes running your own redis compatible key/val store easier for people in your own cluster, that's a win.
- lukevp 3y agoDoes this or Garnet have more coverage of Redis commands? I'm looking to embed a Redis-compatible subset of functionality in a program for local debugging purposes, and there will be an API in between that can protect against Redis commands that lack coverage.
- whoitwas 3y agoI use redis as a cache layer in front of the DB. I don't understand the concept.
- SJC_Hacker 3y agoRedis is a KV store. With this you can run a full SQL query. I did something similiar for my small baseball statistics website switchfielder.com/baseball. On server startup, I copy the relevant tables into memory.
- whoitwas 3y agoI thought the point of Redis was to avoid hitting DB. Why not just use SQL?
- SJC_Hacker 3y agoBecause you are guaranteeing everything is in memory, so its very fast and you aren't relying on the DB memory caching. Also it is in-process, which (supposedly) has some benefits as well. Theoretically DBs such as Postgres can be set up similarly, although they will be out of process, and in the case of Postgres, behind at least a UNIX domain socket. This may be a difference of maybe ~ms, I"d love to see some benchmarks ... maybe I'll try that when I have time. Could make a decent Medium article ... For an OLAP (i.e., no writes), small databases with complicated queries(joins, grouping, partitioning, ordering, etc.) this works very well. Yes this is somewhat of an edge case, but I'd bet small shops could fit their entire DB into one of the bigger (several hundred GB) instances.
- c4pt0r 3y agoNot really wanting to comment on the use of this program (it's useful), but just from the quality of the code, it's very good!
- TimTheTinker 3y agoI'm waiting for someone to implement the Redis API by swapping out the state machine in TigerBeetle (which was built modularly such that the state machine can be swapped out). https://tigerbeetle.com/ https://tigerbeetle.com/
- himujjal 3y agoThis is state of the art. It will take time before Tigerbeetle themselves release their state machine as a library. They have done some incredible job. Implementing things I didnt know existed about software/hardware.
- tjtang2019 3y ago[dead]
- andrewfromx 3y agonice! Now just get cluster support and aws to offer elasticredka?
- moomoo11 3y agoHow often do you use any advanced redis features?