4 ms·

I find JSON-oriented databases to be a huge limitation for writing applications managing any kind of financial data, due to the lack of a decimal number type an

I find JSON-oriented databases to be a huge limitation for writing applications managing any kind of financial data, due to the lack of a decimal number type and a timestamp/date type, both of which SQL provides (and are used A LOT). Sure, you can put that stuff in strings, but then you'll run into limitation with queries where you want to, e.g., aggregate a total, or do timestamp arithmetic. I could do everything with strings, custom map-reduce, etc., if you're inclined to suggest that as a workaround. Still doesn't mean JSON's a good idea.

cuu508 14y ago
Or you could use fixed point math: 600 in database means $6.00 Then aggregates and comparison operators would work, but you would have to decide upfront how much precision you might ever need
- erichocean 14y ago
  That sounds good, but with only 53 bits of integer precision in JSON (51 if you move the decimal point to account for cents), there's just not enough digits for finance these days. A similar problem exists if you use JSON numbers (aka doubles) for timestamps –- the numbers just aren't big enough to do it accurately.
  - ksec 14y ago
    Interesting, that is something i not think about much, any solution to that?
  - moe 14y ago
    Excuse me, what financial application rounds to 51 digits?
    - leif 14y ago
      And doesn't moving the decimal point chop off between 6 and 7 bits?
  - haberman 14y ago
    JSON is not by definition limited to 53 bits of precision -- the standard itself does not specify range or precision. In practice, most implementations represent a JSON number with a double though.
    - shrughes 14y ago
      Maybe one that wants to keep working through high inflation scenarios. (But what financial software is running on a NoSQL database?)
      - haberman 14y ago
        I think perhaps you replied to the wrong comment? I didn't pose the question "what financial app needs >53 bits of precision?"
        shrughes 14y ago
        Yes I did, thank you.
erichocean 14y ago
The other thing that bothers me about all these new JSON databases is they aren't really novel anymore. Clustered databases are essentially a solved problem, and have been for years. What's needed today are databases solving the problem that Google Spanner addresses – global consistency across distributed clusters in separate data centers. If you want a challenge in the DB world, that's where it is. But another clustered, schema-less JSON database? Might as well open up Intro to Algorithms and run through the exercises -- it's no longer a challenge, algorithmically or otherwise. Sorry to be a downer on this, and it does still take a strong coder to implement one, so well done on that front. :)
- jchrisa 14y ago
  Try keeping it running while growing to millions of users in weeks. The simple data model lets us focus on elasticity and performance. There's a lot more to production quality software than the algorithms, but there are only a few NoSQL databases that get the algorithms right.
  - erichocean 14y ago
    Sure, and I'm not trying to imply the RethinkDB guys are writing shoddy code or anything. For all I know the thing is bug-free with fantastic performance, perfect linear scaling with both number of cores and number of nodes in the cluster, and really does let you run your analytic workload on the same cluster you're taking transactions on (though I really doubt this last one – running analytics on your transactional database tends to slow transaction latency to a crawl). That said, with a name like RethinkDB, I guess I expect more than a feature list I could have reasonably put together three years ago and gone, yeah, that's straightforward to do. I've written my own database (and continue to improve it), so I'm pretty familiar with the issues involved. You're absolutely right that many of these JSON database have serious problems under load with their clustering abilities (and it's always under load, they tend to work fine on simple workloads). Perhaps RethinkDB can carve out a niche for reliability-under-load among the existing JSON DB field. That's got to be worth something.
    - 14y ago
oh_sigh 14y ago
Just store it in a string. Or a special map structure: {"date": "2012-11-01", "format":"YYYY-MM-DD" }
alexpopescu 14y ago
We thought of supporting data types that are not part of JSON (date/time/timestamp/deltas, etc.), but we wanted to take the time to do it right so these didn't make it to this version. alex @ rethinkdb

cuu508 14y ago

Or you could use fixed point math: 600 in database means $6.00 Then aggregates and comparison operators would work, but you would have to decide upfront how much precision you might ever need

erichocean 14y ago
That sounds good, but with only 53 bits of integer precision in JSON (51 if you move the decimal point to account for cents), there's just not enough digits for finance these days. A similar problem exists if you use JSON numbers (aka doubles) for timestamps –- the numbers just aren't big enough to do it accurately.
- ksec 14y ago
  Interesting, that is something i not think about much, any solution to that?
- moe 14y ago
  Excuse me, what financial application rounds to 51 digits?
  - leif 14y ago
    And doesn't moving the decimal point chop off between 6 and 7 bits?
- haberman 14y ago
  JSON is not by definition limited to 53 bits of precision -- the standard itself does not specify range or precision. In practice, most implementations represent a JSON number with a double though.
  - shrughes 14y ago
    Maybe one that wants to keep working through high inflation scenarios. (But what financial software is running on a NoSQL database?)
    - haberman 14y ago
      I think perhaps you replied to the wrong comment? I didn't pose the question "what financial app needs >53 bits of precision?"
      - shrughes 14y ago
        Yes I did, thank you.

erichocean 14y ago

The other thing that bothers me about all these new JSON databases is they aren't really novel anymore. Clustered databases are essentially a solved problem, and have been for years. What's needed today are databases solving the problem that Google Spanner addresses – global consistency across distributed clusters in separate data centers. If you want a challenge in the DB world, that's where it is. But another clustered, schema-less JSON database? Might as well open up Intro to Algorithms and run through the exercises -- it's no longer a challenge, algorithmically or otherwise. Sorry to be a downer on this, and it does still take a strong coder to implement one, so well done on that front. :)

jchrisa 14y ago
Try keeping it running while growing to millions of users in weeks. The simple data model lets us focus on elasticity and performance. There's a lot more to production quality software than the algorithms, but there are only a few NoSQL databases that get the algorithms right.
- erichocean 14y ago
  Sure, and I'm not trying to imply the RethinkDB guys are writing shoddy code or anything. For all I know the thing is bug-free with fantastic performance, perfect linear scaling with both number of cores and number of nodes in the cluster, and really does let you run your analytic workload on the same cluster you're taking transactions on (though I really doubt this last one – running analytics on your transactional database tends to slow transaction latency to a crawl). That said, with a name like RethinkDB, I guess I expect more than a feature list I could have reasonably put together three years ago and gone, yeah, that's straightforward to do. I've written my own database (and continue to improve it), so I'm pretty familiar with the issues involved. You're absolutely right that many of these JSON database have serious problems under load with their clustering abilities (and it's always under load, they tend to work fine on simple workloads). Perhaps RethinkDB can carve out a niche for reliability-under-load among the existing JSON DB field. That's got to be worth something.
  - 14y ago

oh_sigh 14y ago

Just store it in a string. Or a special map structure: {"date": "2012-11-01", "format":"YYYY-MM-DD" }

alexpopescu 14y ago

We thought of supporting data types that are not part of JSON (date/time/timestamp/deltas, etc.), but we wanted to take the time to do it right so these didn't make it to this version. alex @ rethinkdb