8 ms·
Show HN: CursusDB – A new scalable distributed document oriented database
- alexpadula 3y agoI'm free to answer any questions any of you may have! Also, https://www.youtube.com/watch?v=UHZ42NkN1OQ https://www.youtube.com/watch?v=UHZ42NkN1OQ <-- Introduction to CursusDB https://www.youtube.com/watch?v=n1t5Iexx4UY https://www.youtube.com/watch?v=n1t5Iexx4UY <-- Automatic Recovery, Backups, Replica Sync, Lost Reconnect Demonstration I've put lot's of effort into the documentation and continuously try to perfect it! https://cursusdb.com/documentation https://cursusdb.com/documentation
- esafak 3y agoFor new products I always recommend the same thing: name your competitors and use cases. Compare them in a table; don't make the user think. Who should and should not use your product? Being in-memory and distributed, I imagine your competitors include Ignite and Hazelcast? I read the title as CursorDB, which would have made a nice DB name.
- supportengineer 3y agoGood point, my first thought is, why wouldn't I just use Couchbase? It is proven and has a community around it.
- alexpadula 3y agoI don't like MongoDB or Couchbase. I dislike their query languages and designs in general. I built the DB for my own liking and use cases. If others like it that's great, if not it doesn't hurt me. I made it open source to follow the process and I'm not looking for any profit or gain for doing any of it. I'd like honest opinions obviously but that's all. Eventually if people really like the DB and it requires ALL my time then I'll try to expand on the cloud offering because I am mighty passionate about the project and want to see it being used as much as possible and easily as possible!
- alexpadula 3y agoHi esafak. I don't really care for competitors. In regards to use cases there are so many! Currently I am working on few tutorials on how to build different applications like a Hackernews clone with a real-time frontend. These tutorials are to bring light to the functionalities of the database and give users ideas. Yeah I don't know those companies, nor do I care to. I built and designed the database the way I wanted to use a database, simple as that. I have my own use cases for a transactional email service relaying what comes through an Observer to client webhooks for example.
- esafak 3y agoYou have competitors whether you like it or not -- unless you don't care for users, either. A sibling commenter asked "Why not just use Couchbase?" If your answer is "figure it out yourself; I already wrote documentation" you are shooting yourself in the foot. Documentation is not marketing. Marketing means helping potential users understand why they should use your product rather than something similar.
- alexpadula 3y agoI agree. I very much care for the users of the database because their experience if everyone elses in a way. My reply was pretty simple lol. You build something YOU like and have a need for there's a chance others will as well. If you just build something that you don't like and have no need for, others probably wont as well.
- alexpadula 3y agoCursus means course in Latin. The database is distributed by nature. Cluster routes data etc. You can piece it together.
- codetrotter 3y agoCouchDB is distributed by nature as well. So why use this over CouchDB.
- alexpadula 3y agoCouchDB's design to me personally is not that great. I don't like the replicated design or the query language. To boot I don't like complexities. I try to avoid them when designing something complex. CursusDB core code is under 6k lines because of that reason. I also wanted to create something I want to use. Simple. Setting up a secure, reliable and powerful database shouldn't be complex and I made certain of that when designing CursusDB. CursusDB's query language as well is very simplified yet powerful because of my complexities ideology.
- alexpadula 3y agoHazelcast seems like Redis. CursusDB is more like MongoDB I'd say in comparison.
- alexpadula 3y agoI do like your idea though esafak I may put something like that on the website.
- nusl 3y agoI get that you need to differentiate your project from others if you want to find adoption, and one way of doing that is to compare it to who you feel are competitors. I do feel that the word “competitor” should rather be “alternative” since I don’t think this or many other projects are trying to compete, but instead just trying to exist in their own way and do their own thing. If someone likes it enough but finds a pain point they might contribute that change, too. Also, part of the beauty of open source is not trying to conform or constantly compare your project to others. It’s awesome that such diversity exists in project communities and the cross-pollination of ideas resulting from it.
- cyanydeez 3y agopeople are simple LLMs. triggering comparisons does a lot of magic that paragraphs of flavor text doesn't do, especially if you're trying to get momentum. it doesn't really matter what you label those comparisons. think of it like LLM optimization.
- nusl 3y agoSure. I don’t feel that every larger project released publicly should seek momentum or recognition to be considered a contribution to software, or that this has to be a goal of any software aiming to grow. Often, folks that are familiar enough with specific domains are able to draw their own comparisons from reading basic docs. If a project was aiming to get wider adoption and traction/momentum then yes, I agree. It doesn’t seem that this is an immediate goal of this project though.
- alexpadula 3y agoNo it's not an immediate goal. It's just a good feeling building something useful to you and it be useful to others that's why I strive to perfect it in every way whether it be documentation, code, design.. doesn't matter! I'll also obviously try to share it with the world in every way I can! :)
- saghm 3y ago> Compare them in a table; don't make the user think Document-oriented databases might be reluctant to accept the premise that tables are better for users
- alexpadula 3y ago^^ Thats funny stuff right there lol!
- zubairq 3y agoI agree, always name your competitors as people will look them up anyway
- corobo 3y agoAlso write a blog post titled "YourProduct vs alt1 vs alt2 vs alt3 etc" 'cause "alt1 vs alt2 vs alt3" is what I'm searching
- alexpadula 3y agoHey corobo, thank you for that! I will surely work on this and post either on the Github Repo Wiki or on the website, or both! Really it's an alternative to a combination of MongoDB, MySQL and Redis.
- jwr 3y agoSo, is it Jepsen-tested?
- alexpadula 3y agoNo not using Jepsen. I've written custom tests in GO running thousands of concurrent client connections on local and remote clusters with multiple nodes and node replicas I test consistency, crisis, recovery, etc. The database hasn't shown me personally any flaws whatsoever. Anyone can go ahead and write their own Jepsen tests if they please.
- alexpadula 3y agoMaybe down the line if people REALLY want it I'll do it but I think I'll just post my tests instead. The thing is I'd rather make a video as you can't test power outages and so forth with just a script. I've done this using hardware locally and I can't find flaws. I'm not trying to make it seem any which way. I'd rather have more people write their own tests as well, why not!
- jwr 3y agoWell, Jepsen has a reputation for finding problems in pretty much every database they test and has become a sort of a benchmark by now. Many people (including myself) won't consider a distributed database if it hasn't been put through Jepsen. The only exception I know of is FoundationDB, which was written pretty much test-first, and about which Kyle Kingsbury said that their tests are likely better :-)
- alexpadula 3y agoThank you for the information @jwr! I've looked into it. I'm not trying to put money down on anything currently. I've tested, and I mean tested hundreds of hours possibly even thousands from the start to the now, every feature, multiple ways. I wouldn't deploy releases and mention features as an actual feature if I didn't test every single thing beforehand. I am thorough, I do crazy amounts of sanity checks on everything, I don't just claim things. I myself cannot do everything in the span of not even 60 days except test everything as I went on and continue to. I post about it on the Github. CursusDB must be manually and driven tested. Very easy to automate but very easy to miss steps. Thank you for your response.
- nusl 3y agoI think this has a place. I’ve found myself wanting something similar in the past though found other solutions. It seems you have similar reservations toward Mongo and similar software so it’s unlikely it will follow their path. I remember so many folks jumping on Mongo and hailing it as the holy grail of storage, dumping everything into it without much thought. I also remember many of those same people paying dearly for doing that. The older Mongo-based projects I’ve worked on were unmaintainable swamps of hacks and pain. Some took the time to care and used Mongo in a way that was sustainable.
- alexpadula 3y agoI remember MongoDB back in < 2014. Lot's of hype but the database had lots of holes.
- cmgriffing 3y agoHey OP, not sure if you saw but I noticed a pretty annoying UI bug in the marketing site. Here is a PR to fix it. Just a simple parseInt. https://github.com/cursusdb/cursusdb-web/pull/1 https://github.com/cursusdb/cursusdb-web/pull/1
- alexpadula 3y agoYou're awesome. Waow. Thank you!
- nsagent 3y agoSeems like you could do the same with rqlite [1], since SQLite supports JSON. [1]: https://rqlite.io https://rqlite.io
- alexpadula 3y agoI've seen this project, it's very cool! I'm all for it if you want to use it. I just didn't want to go the route of SQLite + a layer. SQLite as well is not encrypted by default. There's a lot to the comparison but it's not the same.
- alexpadula 3y agoTheir github mentions encryption but it's only node to node using TLS not much else. https://rqlite.io/docs/guides/security/ https://rqlite.io/docs/guides/security/
- otoolep 3y agorqlite creator here. Wishing you success with your new project. Before we jump into what is encrypted, and what is not, what are the threat vectors we are actually concerned about? With rqlite all traffic between the nodes can be encrypted, as can be the HTTP API. Yes, the data at rest is not, but there are many ways to protect that[1]. One way would be to use an encrypted file system, perhaps. Encrypted data-at-rest is a valid request, but what use cases would it address? [1] https://rqlite.io/docs/guides/security/ https://rqlite.io/docs/guides/security/
- alexpadula 3y agoHello, an absolute pleasure sir! Great work on rqlite. Thank you very much. I never liked the idea of having any database storing my data in a way which can be seen in plain sight regardless of permissions and so forth. Servers get broken into very often and protecting the data within the files is important to me. I've created another tool called qenc that may help the at rest problem if someone has a problem with it. qenc.org With a CursusDB node on signal or automatic backup the system will encrypt and compress block by block. Using ChaCha and DEFLATE just to put that out there, it's pretty cool stuff!
- peter_d_sherman 3y agoI always love to see new Databases! Welcome CursusDB! (Random thought: Someone should create an online list of all known databases, all known versions of those databases, Lines-Of-Code (LOC) counts, the language that the database was written in, and the OS'es that it is available on. Also, same thing but for database connectors/drivers/APIs written to interface to different languages -- show me those languages for each connector... Or, maybe combine these these ideas into a website which allows people to search for databases by OS support, LOC, language written in, languages connector/APIs support, etc., etc. That is, search for a database based on any and all attributes relative to a database). Anyway, welcome CursusDB!
- alexpadula 3y agoThere is one it's made by the kind individual(s) at Carnegie Mellon University. It's called dbdb.io :D Thank you by the way! Great to see enthusiasm for new database systems!
- peter_d_sherman 3y agoHey, thank you very much! Also, excellent link!
- alexpadula 3y agoNo problem! It's a great collection of all databases really!
- alexpadula 3y agoI am working on a video tutorial on how to build a FULL production grade Hackernews clone with real-time front end updates, authentication, automatic backups, replicas, TLS, and yes we will be using CursusDB as the database system using an Observer and Websockets to relay database activity in realtime as shown here https://github.com/cursusdb/cursusdb-observer-node-ws-example https://github.com/cursusdb/cursusdb-observer-node-ws-exampl... In the video I will deploy the final application to Cloud Run and keep it online for you all to play with. I will respond to this with the video! Cheers
- alexpadula 3y agoJust want to add this bit so I've recently finished node's transaction persistence. Mind you I've actually tested power outages at my own house. Don't wanna get into too much details about that lol!! But what I've found is that the .qqueue works as expected. If I make thousands of update requests to one document and the power goes out the CursusDB node system persists all updates on startup. Pretty cool stuff.
- nbittich 3y ago"... Powered by AI", you'd have the most buzzwords title
- alexpadula 3y agoAh buzzwords. I saw a database the other day with hundreds of buzzwords on their website and posts all over and nothing clicked at all actually playing with it. False marketing. I'm not into that. Could lead to disaster.
- shoehorse 3y agoWow.
- alexpadula 3y agoI’m hoping in a good way!
- shoehorse 3y agoI've read through the documentation and have played with database and I really enjoyed the experience. Good job!
- shoehorse 3y agoThe query language is interesting. It has like grammar guide lines? No upper-case testing upper-case the cluster does not accept the queries. I like it though pretty strict.
- latchkey 3y ago"When it starts to fail, please don't curse us."
- alexpadula 3y agoI’d say it leaves you cursed with joy but I may be biased as I created it. I really like the name Cursus and I found it one day just translating English to Latin, just randomly and really liked the meaning of it and it’s super easy to remember so I kept it in my back pocket until I had something to use it for and when designing CursusDB it sprung up whilst drawing out the initial algorithms.
- latchkey 3y agoIt is a good name. I'm just working on taglines for you. ;-)
- alexpadula 3y agoOh haha! I didn’t even catch on! Thank you
- latchkey 3y agoInteresting product. I'm reading through the source and... Woah... it is all in one single file with zero unit tests. Of course there are many ways to skin a cat, this seems like a very odd choice for organizing a database codebase. Some of the methods like cluster.HandleClientConnection() are crazy long for loop + if/else + switch + goto! I know that you've said this is just kind of in your best interests codebase, but if I was planning on making any sort of dependencies on this project, I'd probably shy away from it because it seems like unmanageable spaghetti.
- throwup238 3y ago> unmanageable spaghetti. It's not even 3,000 lines. That's not a lot of spaghetti. cluster.HandleClientConnection looks like a standard network service event loop, except the code is inline instead of spread across cluster.HandleSyncData, .HandleDelete, etc. The lack of tests though...
- alexpadula 3y agoIf you guys really want me to post the tests I will. No problems!
- alexpadula 3y agoThere is a lot of them so I will create a new repository for them, possibly; Or commit them to the core repo. I’ll see. I did have tests within the main repo at one point but I took them out before v2.0.0.
- latchkey 3y agoTaking the tests out is like doing the homework and not turning it in for grading.
- alexpadula 3y agoI mean you feel pretty good about removing them after hundreds of automated tests runs and hundreds of hours of manual testing. I could be biased in that I will put back tests for the sanity of you all :). I do understand for contribution sake to have them there.
- busymom0 3y agoOP, would you mind sharing your background (sorry if I missed this somewhere?). Like what kind of experiencer you had prior to this which got you to write a whole database from scratch?