8 ms·

CodernityDB — pure Python, NoSQL, fast database

183 points by amarsahinovic 14y ago

louischatriot 14y ago
Interesting, but I'd like to see how this compares to other NoSQL such as MongoDB.
deleted 14y ago
[deleted]
gldnspud 14y ago
I'd love to check this out this morning, but the page is unreadable using Safari on an iPhone. The table of contents stays floating above the body text, and takes up almost the whole screen.
dbecker 14y ago
It would be helpful to add "How is this different from other NoSQL databases" to the FAQ.
- danudey 14y ago
  It seems as though the key differentiator is 'pure python'. For python environments, this is a huge bonus, as it makes it easy to install and support with existing tools.
AaronBBrown 14y ago
"CodernityDB pure python, NoSQL, fast database" Is the fact that it is written in "pure Python" really the most important thing to reenforce after the name of the product itself? Why would I use this over established products like Riak, Redis, MongoDB, etc?
- zimbatm 14y ago
  If you look at the speed benchs they compare it to kyoto cabinet. I think that CodernityDB is meant to be embedded and is thus only interesting to python people. In that case the advantage would be to avoid the compilation step on intall that kyoto cabinet or sqlite would incur. It's a library database like sqlite and kyoto cabinet
  - apendleton 14y ago
    Looks like they also have an HTTP service and a Python client library, so you can use it other ways than embedded, and for the Python client, you can switch back and forth from the embedded one to the HTTP one with the same API. Seems pretty slick.
- zalew 14y ago
  > Is the fact that it is written in "pure Python" really the most important thing to reenforce after the name of the product itself? It's definitely a selling point for Python programmers, because it's so easy to use with your Python projects. You just set the package requirement and you're done, it will work in the same environment wherever your app works, upgrades are a piece of cake, no need to worry about platform support, permissions, etc. f.ex. Whoosh (full-text search) gained a lot of traction in the Python world not because it was the fastest at the time nor the most full-featured (compared to the more mature Java-based ones), but because of convenience. Such solutions, even when they're not the most advanced player in the league, are great for starting up fast and pushing features out the door. On a side note, it's a pleasure for me to see it's from Poland. Will test it out on a feature project in the next few days.
  - Ingaz 14y ago
    The thing I like most in python is not python itself but a "pythonic way". "Pythonic" > Python. In my opinion, it's not "pythonic" to use DBMS, just because it written in python. So the "selling point" is flawed. Disclosure: I'm using python since 2006. I've made a lot of evangelizing it. It is my weapon of choice for many tasks. And I will never use python for many other tasks. (Don't tell me about PyPy or Stackless)
    - timwiseman 14y ago
      You make an excellent point, but when working in Python there are advantages to tying in to tools that are also written in Python. They will tend to be easier to integrate into your project than non-Python code[1] and more than that will generally be easier for you to extend or tweak[2] if you run into a case where you have to "look under the hoos" for some reason.[3] [1] Python works well with other languages of course, and is often used as a "glue" between other components. [2] This assumes that you know Python better than whatever language it was made in, but for many cases that will be true. [3] I often like to look at libraries just to understand how they work, but that is different. There are often cases, especially where the tool uses some abstraction that can leak, where you need to look under the hood just to get things working.
- nubela 14y ago
  perhaps it could be attracting devs, since python is a really awesome language to work with. but really, let's quit the snarkyness.
- draegtun 14y ago
  Avoids dependency issues when you try to deploy your work on an unmodifiable target. I've been looking at DBM::Deep (one of perl's equivalent to CodernityDB) and App::FatPacker recently just for this case (install a script on some Macs with only Perl being needed requirement). ref: https://metacpan.org/module/DBM::Deep https://metacpan.org/module/DBM::Deep | https://metacpan.org/module/App::FatPacker https://metacpan.org/module/App::FatPacker
- pbreit 14y ago
  "Is the fact that it is written in "pure Python" really the most important thing" It is for me. It's the most unique attribute about the project. Fast? Yawn. NoSQL? Yawn.
- mistercow 14y ago
  Python is on almost every host out there now. Riak, Redis, and MongoDB are not, and on many hosts you won't be able to compile them from the source. So yes, that's relevant.
  - justincormack 14y ago
    If you are on a host without a compiler you don't need a NOSQL db.
    - mistercow 14y ago
      How do you figure?
      - danudey 14y ago
        I believe the insinuation is that if you're actually building a serious app that might actually benefit from NoSQL, then you should host it on a serious machine and not some shared hosting solution. Basically conflating the idea that NoSQL is for 'real projects' and that only 'amateur hour' hosts have no compiler.
        adrinavarro 14y ago
        NoSQL is targeted at extreme performance. So, yes, it sounds like a reasonable assumption.
        mistercow 14y ago
        There are other advantages to NoSQL, like flexible data structure, and not having to squish your data into two incompatible data models.
    - bthomas 14y ago
      Wouldn't you want to remove extraneous software eg. a compiler from a dedicated database host for security? (Yes, Python could be included in that as well)
- shuzchen 14y ago
  For me, the db being in pure-python means a db that can be easily extended or augmented. If you have some quirky requirement your options are 1) implement this in the client, or 2) modify your db. Option 1 is probably what most people opt for (how many people have the confidence to dig into postgres/mysql source, how many of these people want to?), and a pure-python db makes option 2 more feasible.
- DocSavage 14y ago
  Although pure python seems great for some apps, for a piece of software I'd like to be highly optimized, it seems to hit the wrong sweet spot -- it means a larger memory footprint and less robust multi-threading than other pure language implementations. When things need to work fast and tight in python, most implementations duck down to C, like numpy and scipy. Or maybe I've been out of touch with improvements to CPython?
  - encoderer 14y ago
    Threading in Python is totally capable here. A database is more likely I/O bound than CPU bound.
    - DocSavage 14y ago
      What about memory? There's a fairly large overhead for pure python data structures compared to C/C++.
knightni 14y ago
Since these questions will inevitably come up: - You fsync yourself to ensure durability. I can't see at a glance what fsync settings are used for the speed tests. - It's not transactional, although single operations are atomic. - Indexes operate on a single-writer, multiple reader basis. - No traditional joins, although you can of course write a procedural function that joins for you.
- 3amOpsGuy 14y ago
  These have come to be common features of NoSQL in general. I wouldn't be too surprised to see them here in a product marketed as NoSQL. The other attributes i'm not too fussed about personally, i figure i can always work around them or face the question "am i using the right tool for this job?" but the fsync one still gives me the heeby-jeebies (even though it's never bitten me despite many many TBs of data). Cassandra does periodic fsync's (although it can be configured). I understand HBase and Mongo have similar shennanigans.
francesca 14y ago
This seems like a speedy solution for inserts but I'm curious about reads "Indexes tries to reuse as much space as possible, because metadata size is fixed, during every write operation, if index finds metadata marked as removed or so, it reuses it - writes new data into that place." I'm curious how this is implemented.
mattbasta 14y ago
The only thing that turns me off to a product more than a poor website is poor English in the documentation. If English isn't your primary language, PLEASE get someone that speaks it fluently to either write your docs, translate your docs, or edit what you've written.
- nakovet 14y ago
  They are nice to write the documentation in poor English, they could launch the product only with Polish documentation and then you would never be able to use it before someones translates it.
  - pyre 14y ago
    While this is true, it's to the detriment of their project not to have at least some English docs. It grants them the largest possible audience[1] for their project. Generating interest in their tool is one of the things that they need to do if they want it to be a successful open source project. [1]: I guess it's possible that 'Chinese' might be close too, but I have no idea how the different dialects (Mandarin/Cantonese) would affect this.
    - saraid216 14y ago
      > [1]: I guess it's possible that 'Chinese' might be close too, but I have no idea how the different dialects (Mandarin/Cantonese) would affect this. Cantonese isn't written down, strictly speaking; when you write Chinese, there aren't dialectic differences.
      - francesca 14y ago
        nginx didn't have english docs until 2 years after the project launched. Full story from Igor Here: http://www.ruby-forum.com/topic/151853 http://www.ruby-forum.com/topic/151853 Also: Don't blame people for english lang proficiency.
  - cabalamat 14y ago
mmcnickle 14y ago
A cursory glance through the code revealed a bug: https://bitbucket.org/codernity/codernitydb/issue/1/_rev-not-guaranteed-to-be-unique https://bitbucket.org/codernity/codernitydb/issue/1/_rev-not...
- deleted 14y ago
  [deleted]
- ecaron 14y ago
  You can learn a lot about a community, a product, and a company based on how they respond to bug reports. Promptness, formality, details and politeness are all incredible indicators about what's going on under the hood.
  - d0ugal 14y ago
    And what did you make of their reply?
- StavrosK 14y ago
  By the way, the risk of a collision with 10 times more members is much more than 10 times as likely, due to the birthday paradox.
  - mmcnickle 14y ago
    Yeah, I realised I got the probability wrong but haven't written a follow up (typing on my phone is tough, writing code is impossible). The probability of being able to update an old version of the document, is simply 1/ 65536. That's a very high probability given thousands of documents or a modest number of competing updates. The biggest issue is that it's undetectable and non-deterministic: the update that should fail will succeed silently and it can happen at any time. I'll re-open the bug in the morning with a proof-of-concept.
hoodoof 14y ago
No Python 3 = no thanks
ucee054 14y ago
In case, like me, you like SQL: http://gadfly.sourceforge.net/ http://gadfly.sourceforge.net/
lehgo 14y ago
python ehh
shuzchen 14y ago
I've registered ##codernitydb in freenode and will idle in there, if anybody is interested in forming a community in irc
tintor 14y ago
What is the difference between DatabaseThreadSafe and DatabaseSuperThreadSafe? http://labs.codernity.com/codernitydb/design.html#how-it-s-build http://labs.codernity.com/codernitydb/design.html#how-it-s-b...

183 points by amarsahinovic 14y ago

louischatriot 14y ago

Interesting, but I'd like to see how this compares to other NoSQL such as MongoDB.

deleted 14y ago

[deleted]

gldnspud 14y ago

I'd love to check this out this morning, but the page is unreadable using Safari on an iPhone. The table of contents stays floating above the body text, and takes up almost the whole screen.

dbecker 14y ago

It would be helpful to add "How is this different from other NoSQL databases" to the FAQ.

danudey 14y ago
It seems as though the key differentiator is 'pure python'. For python environments, this is a huge bonus, as it makes it easy to install and support with existing tools.

AaronBBrown 14y ago

"CodernityDB pure python, NoSQL, fast database" Is the fact that it is written in "pure Python" really the most important thing to reenforce after the name of the product itself? Why would I use this over established products like Riak, Redis, MongoDB, etc?

zimbatm 14y ago
If you look at the speed benchs they compare it to kyoto cabinet. I think that CodernityDB is meant to be embedded and is thus only interesting to python people. In that case the advantage would be to avoid the compilation step on intall that kyoto cabinet or sqlite would incur. It's a library database like sqlite and kyoto cabinet
- apendleton 14y ago
  Looks like they also have an HTTP service and a Python client library, so you can use it other ways than embedded, and for the Python client, you can switch back and forth from the embedded one to the HTTP one with the same API. Seems pretty slick.
zalew 14y ago
> Is the fact that it is written in "pure Python" really the most important thing to reenforce after the name of the product itself? It's definitely a selling point for Python programmers, because it's so easy to use with your Python projects. You just set the package requirement and you're done, it will work in the same environment wherever your app works, upgrades are a piece of cake, no need to worry about platform support, permissions, etc. f.ex. Whoosh (full-text search) gained a lot of traction in the Python world not because it was the fastest at the time nor the most full-featured (compared to the more mature Java-based ones), but because of convenience. Such solutions, even when they're not the most advanced player in the league, are great for starting up fast and pushing features out the door. On a side note, it's a pleasure for me to see it's from Poland. Will test it out on a feature project in the next few days.
- Ingaz 14y ago
  The thing I like most in python is not python itself but a "pythonic way". "Pythonic" > Python. In my opinion, it's not "pythonic" to use DBMS, just because it written in python. So the "selling point" is flawed. Disclosure: I'm using python since 2006. I've made a lot of evangelizing it. It is my weapon of choice for many tasks. And I will never use python for many other tasks. (Don't tell me about PyPy or Stackless)
  - timwiseman 14y ago
    You make an excellent point, but when working in Python there are advantages to tying in to tools that are also written in Python. They will tend to be easier to integrate into your project than non-Python code[1] and more than that will generally be easier for you to extend or tweak[2] if you run into a case where you have to "look under the hoos" for some reason.[3] [1] Python works well with other languages of course, and is often used as a "glue" between other components. [2] This assumes that you know Python better than whatever language it was made in, but for many cases that will be true. [3] I often like to look at libraries just to understand how they work, but that is different. There are often cases, especially where the tool uses some abstraction that can leak, where you need to look under the hood just to get things working.
nubela 14y ago
perhaps it could be attracting devs, since python is a really awesome language to work with. but really, let's quit the snarkyness.
draegtun 14y ago
Avoids dependency issues when you try to deploy your work on an unmodifiable target. I've been looking at DBM::Deep (one of perl's equivalent to CodernityDB) and App::FatPacker recently just for this case (install a script on some Macs with only Perl being needed requirement). ref: https://metacpan.org/module/DBM::Deep https://metacpan.org/module/DBM::Deep | https://metacpan.org/module/App::FatPacker https://metacpan.org/module/App::FatPacker
pbreit 14y ago
"Is the fact that it is written in "pure Python" really the most important thing" It is for me. It's the most unique attribute about the project. Fast? Yawn. NoSQL? Yawn.
mistercow 14y ago
Python is on almost every host out there now. Riak, Redis, and MongoDB are not, and on many hosts you won't be able to compile them from the source. So yes, that's relevant.
- justincormack 14y ago
  If you are on a host without a compiler you don't need a NOSQL db.
  - mistercow 14y ago
    How do you figure?
    - danudey 14y ago
      I believe the insinuation is that if you're actually building a serious app that might actually benefit from NoSQL, then you should host it on a serious machine and not some shared hosting solution. Basically conflating the idea that NoSQL is for 'real projects' and that only 'amateur hour' hosts have no compiler.
      - adrinavarro 14y ago
        NoSQL is targeted at extreme performance. So, yes, it sounds like a reasonable assumption.
        mistercow 14y ago
        There are other advantages to NoSQL, like flexible data structure, and not having to squish your data into two incompatible data models.
  - bthomas 14y ago
    Wouldn't you want to remove extraneous software eg. a compiler from a dedicated database host for security? (Yes, Python could be included in that as well)
shuzchen 14y ago
For me, the db being in pure-python means a db that can be easily extended or augmented. If you have some quirky requirement your options are 1) implement this in the client, or 2) modify your db. Option 1 is probably what most people opt for (how many people have the confidence to dig into postgres/mysql source, how many of these people want to?), and a pure-python db makes option 2 more feasible.
DocSavage 14y ago
Although pure python seems great for some apps, for a piece of software I'd like to be highly optimized, it seems to hit the wrong sweet spot -- it means a larger memory footprint and less robust multi-threading than other pure language implementations. When things need to work fast and tight in python, most implementations duck down to C, like numpy and scipy. Or maybe I've been out of touch with improvements to CPython?
- encoderer 14y ago
  Threading in Python is totally capable here. A database is more likely I/O bound than CPU bound.
  - DocSavage 14y ago
    What about memory? There's a fairly large overhead for pure python data structures compared to C/C++.

knightni 14y ago

Since these questions will inevitably come up: - You fsync yourself to ensure durability. I can't see at a glance what fsync settings are used for the speed tests. - It's not transactional, although single operations are atomic. - Indexes operate on a single-writer, multiple reader basis. - No traditional joins, although you can of course write a procedural function that joins for you.

3amOpsGuy 14y ago
These have come to be common features of NoSQL in general. I wouldn't be too surprised to see them here in a product marketed as NoSQL. The other attributes i'm not too fussed about personally, i figure i can always work around them or face the question "am i using the right tool for this job?" but the fsync one still gives me the heeby-jeebies (even though it's never bitten me despite many many TBs of data). Cassandra does periodic fsync's (although it can be configured). I understand HBase and Mongo have similar shennanigans.

francesca 14y ago

This seems like a speedy solution for inserts but I'm curious about reads "Indexes tries to reuse as much space as possible, because metadata size is fixed, during every write operation, if index finds metadata marked as removed or so, it reuses it - writes new data into that place." I'm curious how this is implemented.

mattbasta 14y ago

The only thing that turns me off to a product more than a poor website is poor English in the documentation. If English isn't your primary language, PLEASE get someone that speaks it fluently to either write your docs, translate your docs, or edit what you've written.

nakovet 14y ago
They are nice to write the documentation in poor English, they could launch the product only with Polish documentation and then you would never be able to use it before someones translates it.
- pyre 14y ago
  While this is true, it's to the detriment of their project not to have at least some English docs. It grants them the largest possible audience[1] for their project. Generating interest in their tool is one of the things that they need to do if they want it to be a successful open source project. [1]: I guess it's possible that 'Chinese' might be close too, but I have no idea how the different dialects (Mandarin/Cantonese) would affect this.
  - saraid216 14y ago
    > [1]: I guess it's possible that 'Chinese' might be close too, but I have no idea how the different dialects (Mandarin/Cantonese) would affect this. Cantonese isn't written down, strictly speaking; when you write Chinese, there aren't dialectic differences.
    - francesca 14y ago
      nginx didn't have english docs until 2 years after the project launched. Full story from Igor Here: http://www.ruby-forum.com/topic/151853 http://www.ruby-forum.com/topic/151853 Also: Don't blame people for english lang proficiency.
- cabalamat 14y ago

mmcnickle 14y ago

A cursory glance through the code revealed a bug: https://bitbucket.org/codernity/codernitydb/issue/1/_rev-not-guaranteed-to-be-unique https://bitbucket.org/codernity/codernitydb/issue/1/_rev-not...

deleted 14y ago
[deleted]
ecaron 14y ago
You can learn a lot about a community, a product, and a company based on how they respond to bug reports. Promptness, formality, details and politeness are all incredible indicators about what's going on under the hood.
- d0ugal 14y ago
  And what did you make of their reply?
StavrosK 14y ago
By the way, the risk of a collision with 10 times more members is much more than 10 times as likely, due to the birthday paradox.
- mmcnickle 14y ago
  Yeah, I realised I got the probability wrong but haven't written a follow up (typing on my phone is tough, writing code is impossible). The probability of being able to update an old version of the document, is simply 1/ 65536. That's a very high probability given thousands of documents or a modest number of competing updates. The biggest issue is that it's undetectable and non-deterministic: the update that should fail will succeed silently and it can happen at any time. I'll re-open the bug in the morning with a proof-of-concept.

hoodoof 14y ago

No Python 3 = no thanks

ucee054 14y ago

In case, like me, you like SQL: http://gadfly.sourceforge.net/ http://gadfly.sourceforge.net/

lehgo 14y ago

python ehh

shuzchen 14y ago

I've registered ##codernitydb in freenode and will idle in there, if anybody is interested in forming a community in irc

tintor 14y ago

What is the difference between DatabaseThreadSafe and DatabaseSuperThreadSafe? http://labs.codernity.com/codernitydb/design.html#how-it-s-build http://labs.codernity.com/codernitydb/design.html#how-it-s-b...