4 ms·

As a hacker/tinkerer type programmer I can understand why it might be fun to build something like this. All of the data structure fun with none of the distribu

As a hacker/tinkerer type programmer I can understand why it might be fun to build something like this. All of the data structure fun with none of the distributed difficulty! NoSQL! I can even build in that cool new language I've been designing in my head! But like everyone else I'm straining to see a use case. "Serverless" NoSQL? If you've got something small enough that you want an embedded datastore, why use a NoSQL variant? I understand that NoSQL literally means "no SQL," but typically it's a buzzword reserved for unconventional datastores used under heavy distribution. Maybe you'd want a local embedded "NoSQL" to act as a local datacache? But then for that use case why create an impedance mismatch with the central datastore by rolling your own? Jx9? So wait, I need another language to interact with the thing that's already embedded directly within my program? Local offline data analysis? Now that makes sense. Does this have intelligent/optimized paging mechanisms? I have a feeling not, since the author seems to be couching this as an alternative to sqlite for people who like NoSQL. I'd sincerely love to hear more about what's motivating this project. Was it just for yucks, or is there some problem that the author(s) needed to solve that wasn't well-solved by another tool?

arkitaip 13y ago
The problem isn't the author's lust for tinkering and improving his dev chops. The problem is with HN, how it's designed to drive fads and constantly disturb our focus, and immature developers who change tech stacks as if it was underwear.
- benjamincburns 13y ago
  The problem? There's no problem here. I find the OP very interesting. Whether or not it's useful to myself or this community, it's something which someone very intelligent spent a significant amount of time to build. I'd argue that its usefulness is unrelated to how interesting it is. It genuinely makes me want to know more about the author(s) and what their motivation was.
wmf 13y ago
There are two different camps within NoSQL; some people are webscale and some people want NoSchema. This looks like it's aimed at the latter category.
continuations 13y ago
> I'm straining to see a use case. "Serverless" NoSQL? If you've got something small enough that you want an embedded datastore, why use a NoSQL variant? Serverless NoSQL embedded datastore aren't really that unusual. BerkeleyDB, Tokyo Cabinet, LevelDB are all commonly used serverless NoSQL.
- benjamincburns 13y ago
  I think what's happening here is that NoSQL is just an anti-category. When I think of NoSQL, I think of something like MongoDB or Cassandra which are designed to handle tons of data under heavy load. When I think of BerkeleyDB and friends, I think local persistence. It's a true statement that these too are NoSQL stores, but, at least for me, they're not of the variety that first comes to mind.
tracker1 13y ago
I'd have to agree with you here... I mean, there are lots of interfaces for reading/writing json, and simply outputting a JSON as a UTF-8 .json.gz file seems more appropriate... especially if your data can fit into memory. I would be more inclined to simply have my object structure in memory, using a more convenient high-level language, and do a load/dump from .json.gz files as needed. It's fast enough, and worst case scenario, I can backup/extract and read/write in any number of programming languages. If you need multiple records, that may be easier, having an index, and using line delimited flat json structures can work as well... (breaking on \n) This really seems like an also ran, where you could have abstracted out an SQLite db with a single table of (key VARCHAR(100) PRIMARY KEY, value VARCHAR(4000)) or something similar.
- shuzchen 13y ago
  Following the thread of building upon sqlite, Yelp (along with the fabulous mrjob library) also open-sourced sqlite3dbm (http://pythonhosted.org/sqlite3dbm/ http://pythonhosted.org/sqlite3dbm/), which is a document store on top of sqlite. Python only (meant to give random-access data to your EMR jobs running with mrjob), but the same approach could be applied to any other language.
ricardobeat 13y ago
"serverless" databases are usually called storage engines: LevelDB, BerkeleyDB, BitCask, InnoDB. There is a (good) trend towards modular software, separating the storage engine from the server is a natural consequence. But yeah, requiring a new obscure language pretty much obliterates any reason you'd want to use this. Lua is fast, popular, made for this exact use case and already used by Redis.
- benjamincburns 13y ago
  I probably came off too harsh. I'm quite familiar with the idea of embedded datastores, SQL and otherwise, and I didn't mean to attack the concept. I was more trying to illustrate (albeit weakly) that the marketing seems a bit off. Like I said elsewhere, to me NoSQL tends to mean "big, distributed storage thing that isn't RDBMS, usually with an emphasis on horizontal scalability." Others have said correctly that some also take it first to mean schemaless and relax the "big" requirement. However I'm still confused on what value a general-purpose "schemaless" datastore provides in an embedded context. I write schemaless in quotes there because there's still an impedance mismatch between the format in which your application works with and the format in which the datastore works. All of these rely on some general-purpose abstraction. But your application, if designed properly, probably makes use of a number of varying data structures. What are you gaining by pushing all of these through some abstraction? Assuming that all you're after is simple persistence, I have to ask - is writing data to disk really that hard? That said, I've used local persistence libraries plenty, usually as a persistent local cache, and/or as a tool to enforce a schema which allows for data migration between constantly updating versions of software.
ctide 13y ago
When I was at Singly working on the locker project, this was actually the absolute perfect solution to how we wanted to store data (it just wasn't available then.) We wanted to store all the raw JSON that was coming down the pipe from various sources (Facebook, Twitter, etc.) and wanted it all to be queryable. Also, since each user's data was completely isolated, having a tiny embeddable solution made a lot more sense vs. firing up a MongoDB or something similar for every user.

arkitaip 13y ago

The problem isn't the author's lust for tinkering and improving his dev chops. The problem is with HN, how it's designed to drive fads and constantly disturb our focus, and immature developers who change tech stacks as if it was underwear.

benjamincburns 13y ago
The problem? There's no problem here. I find the OP very interesting. Whether or not it's useful to myself or this community, it's something which someone very intelligent spent a significant amount of time to build. I'd argue that its usefulness is unrelated to how interesting it is. It genuinely makes me want to know more about the author(s) and what their motivation was.

wmf 13y ago

There are two different camps within NoSQL; some people are webscale and some people want NoSchema. This looks like it's aimed at the latter category.

continuations 13y ago

> I'm straining to see a use case. "Serverless" NoSQL? If you've got something small enough that you want an embedded datastore, why use a NoSQL variant? Serverless NoSQL embedded datastore aren't really that unusual. BerkeleyDB, Tokyo Cabinet, LevelDB are all commonly used serverless NoSQL.

benjamincburns 13y ago
I think what's happening here is that NoSQL is just an anti-category. When I think of NoSQL, I think of something like MongoDB or Cassandra which are designed to handle tons of data under heavy load. When I think of BerkeleyDB and friends, I think local persistence. It's a true statement that these too are NoSQL stores, but, at least for me, they're not of the variety that first comes to mind.

tracker1 13y ago

I'd have to agree with you here... I mean, there are lots of interfaces for reading/writing json, and simply outputting a JSON as a UTF-8 .json.gz file seems more appropriate... especially if your data can fit into memory. I would be more inclined to simply have my object structure in memory, using a more convenient high-level language, and do a load/dump from .json.gz files as needed. It's fast enough, and worst case scenario, I can backup/extract and read/write in any number of programming languages. If you need multiple records, that may be easier, having an index, and using line delimited flat json structures can work as well... (breaking on \n) This really seems like an also ran, where you could have abstracted out an SQLite db with a single table of (key VARCHAR(100) PRIMARY KEY, value VARCHAR(4000)) or something similar.

shuzchen 13y ago
Following the thread of building upon sqlite, Yelp (along with the fabulous mrjob library) also open-sourced sqlite3dbm (http://pythonhosted.org/sqlite3dbm/ http://pythonhosted.org/sqlite3dbm/), which is a document store on top of sqlite. Python only (meant to give random-access data to your EMR jobs running with mrjob), but the same approach could be applied to any other language.

ricardobeat 13y ago

"serverless" databases are usually called storage engines: LevelDB, BerkeleyDB, BitCask, InnoDB. There is a (good) trend towards modular software, separating the storage engine from the server is a natural consequence. But yeah, requiring a new obscure language pretty much obliterates any reason you'd want to use this. Lua is fast, popular, made for this exact use case and already used by Redis.

benjamincburns 13y ago
I probably came off too harsh. I'm quite familiar with the idea of embedded datastores, SQL and otherwise, and I didn't mean to attack the concept. I was more trying to illustrate (albeit weakly) that the marketing seems a bit off. Like I said elsewhere, to me NoSQL tends to mean "big, distributed storage thing that isn't RDBMS, usually with an emphasis on horizontal scalability." Others have said correctly that some also take it first to mean schemaless and relax the "big" requirement. However I'm still confused on what value a general-purpose "schemaless" datastore provides in an embedded context. I write schemaless in quotes there because there's still an impedance mismatch between the format in which your application works with and the format in which the datastore works. All of these rely on some general-purpose abstraction. But your application, if designed properly, probably makes use of a number of varying data structures. What are you gaining by pushing all of these through some abstraction? Assuming that all you're after is simple persistence, I have to ask - is writing data to disk really that hard? That said, I've used local persistence libraries plenty, usually as a persistent local cache, and/or as a tool to enforce a schema which allows for data migration between constantly updating versions of software.

ctide 13y ago

When I was at Singly working on the locker project, this was actually the absolute perfect solution to how we wanted to store data (it just wasn't available then.) We wanted to store all the raw JSON that was coming down the pipe from various sources (Facebook, Twitter, etc.) and wanted it all to be queryable. Also, since each user's data was completely isolated, having a tiny embeddable solution made a lot more sense vs. firing up a MongoDB or something similar for every user.