4 ms·

After spending a decade working with Lotus Domino, I think all of these people who are willingly going back to document style databases are insane. Being able

by NDizzle 12y ago

After spending a decade working with Lotus Domino, I think all of these people who are willingly going back to document style databases are insane. Being able to fiddle with your schema on the fly isn't necessarily a good thing. Brainstorm, plan, execute. Or just slap another field on the end of the document. Whichever sounds better for the long term health of your data!

jchrisa 12y ago
Good to see a complaint about NoSQL that is at least targeted in a reasonable direction. Personally I see a lot of success coming from flexible document schemas but obviously it's not perfect for every app. What did you think about Domino replication?
- NDizzle 12y ago
  I have used it extensively. It's brittle. It's brilliant. The way Domino works at a high level is that you have a "data" and "design" template applied to a database. A database would be a single file on disk, and it's something like a table (or a set of related tables) and various views attached to that data. We had an internal Domino server with a set of internally focused 'design' templates applied to them. We used the notes clients, and our analysts had their workflow applications there. The design of these databases was focused on serving content to the notes clients - not to a web browser. We replicated the data from those databases to our staging and development machines. Those staging and development machines had their own designs - which was the web-centric, customer focused design. There was a minimum amount of information available to you in the notes clients - data at this point was meant to be seen in a browser. Those staging machines pushed the data on to our final production cluster. Every 15 minutes something was pushed. We'd go from internal to dev/stage 15 after the hour. 30 minutes after the hour the data from stage would go to production. We used normal build/release tools control the flow of design template replicas making their way out of their playgrounds, but sometimes it happened. I say it's brittle because if you accidentally push design to the wrong place you could grenade a lot of stuff. More than once we accidentally flipped the switch to push design changes to development. Here's a link to a presentation of the last thing I was a part of in the Lotus community. I'm pretty proud of the site we put together. It had a pretty nice feature set for the time period. This was put together by a coworker. https://www.youtube.com/watch?list=PL6D93ED85F970BAE6&v=v9IUly7np0A https://www.youtube.com/watch?list=PL6D93ED85F970BAE6&v=v9IU...
  - jchrisa 12y ago
    Looks like a lot of requirements added over time.
boomzilla 12y ago
There is a place for document store, and in my opinion it's for prototyping in a domain one's not necessarily experienced with yet. In that situation, trying to come up with the perfect (or just reasonably good enough) data model is overwhelming. So it's easier and faster to just go ahead with a document store. I like Solr because I know it best and Apache licensed and I'll likely need a search engine later. Postgres JSON type, Elastic search, Mongo or others are perfectly valid choices too. However, once the app has shaped up, I make it a priority to harden up the data model and move the core data set to a relational database (Postgres is the likely choice now). One nice thing with this migration is I can do a lot of clean up and normalization with the originally unstructured data set.
- oacgnol 12y ago
  I think this is the most reasonable take I've read and encompasses the whole "right tool for the job" mindset that I like. I've found that when starting a new project I get paralyzed with coming up with the perfect data model that will also scale in the future, so I'd rather punt, take the tradeoffs that come with schemaless DBs, and keep a plan to migrate to something like Postgres in the backlog. Holding on to our tools too tightly is IMO how projects get crippled.
- StavrosK 12y ago
  Exactly that, but I would also add that migrating things that are documents off to a document database (or leaving them there after prototyping) is perfectly fine.
untog 12y ago
One of the reasons I like Postgres having JSON. In that experimental phase, make up a schema. Then solidify it.
wereHamster 12y ago
I use RethinkDB as the backing database for a API server written in Haskell. So I have pretty strong guarantees that the data which enters the database is sane, and when I read it out Haskell automatically validates it against a very strict schema. SQL schemas are not expressive enough to describe the data types which I want to store in the database. The argument SQL vs NoSQL becomes moot when any database is degraded to essentially pure blob storage. What matters to me is ease of use, administration/maintenance burden, and of course speed. How strictly the database verifies the schema isn't even on the picture.
- teej 12y ago
  > SQL schemas are not expressive enough to describe the data types which I want to store in the database For my own edification, can you expand on this?
  - stingraycharles 12y ago
    You will have to educate yourself on Haskell's type system and type category theory in general. The bottom line is that you can be very, very expressive using higher level types that simply is not possible in any database at all. Personally I like foreign keys, a lot, and couldnt live without them. I dont think Haskell's type system would be a replacement to that, and would rather use a hybrid of those two.
    - SamReidHughes 12y ago
      You don't need to learn anything about category theory to use Haskell well.
  - wereHamster 12y ago
    I suspect with (non-standard?) SQL extensions and pl/sql functions you could add enough validation that is otherwise inexpressible (in standard SQL). But then you have to replicate some of that logic in your application code and the database. Anyways, let's start with something simple: sumtypes. While simple sumtypes could be described using enums, what if the alternatives have more data attached to them? data User = Anonymous | Registered UserData data UserData = UserData { userName :: UserName , registredAt :: UTCTime } Foreign keys is something that can't be easily verified in Haskell code. So that's something where a proper database still is useful.
jwilliams 12y ago
Hmm. Interesting anecdote. I've spent a couple of decades in and around relational databases. They tend to be half-denormalized crufty beasts too. Data integrity is less of a technology problem than people make out.

by NDizzle 12y ago

jchrisa 12y ago

Good to see a complaint about NoSQL that is at least targeted in a reasonable direction. Personally I see a lot of success coming from flexible document schemas but obviously it's not perfect for every app. What did you think about Domino replication?

NDizzle 12y ago
I have used it extensively. It's brittle. It's brilliant. The way Domino works at a high level is that you have a "data" and "design" template applied to a database. A database would be a single file on disk, and it's something like a table (or a set of related tables) and various views attached to that data. We had an internal Domino server with a set of internally focused 'design' templates applied to them. We used the notes clients, and our analysts had their workflow applications there. The design of these databases was focused on serving content to the notes clients - not to a web browser. We replicated the data from those databases to our staging and development machines. Those staging and development machines had their own designs - which was the web-centric, customer focused design. There was a minimum amount of information available to you in the notes clients - data at this point was meant to be seen in a browser. Those staging machines pushed the data on to our final production cluster. Every 15 minutes something was pushed. We'd go from internal to dev/stage 15 after the hour. 30 minutes after the hour the data from stage would go to production. We used normal build/release tools control the flow of design template replicas making their way out of their playgrounds, but sometimes it happened. I say it's brittle because if you accidentally push design to the wrong place you could grenade a lot of stuff. More than once we accidentally flipped the switch to push design changes to development. Here's a link to a presentation of the last thing I was a part of in the Lotus community. I'm pretty proud of the site we put together. It had a pretty nice feature set for the time period. This was put together by a coworker. https://www.youtube.com/watch?list=PL6D93ED85F970BAE6&v=v9IUly7np0A https://www.youtube.com/watch?list=PL6D93ED85F970BAE6&v=v9IU...
- jchrisa 12y ago
  Looks like a lot of requirements added over time.

boomzilla 12y ago

There is a place for document store, and in my opinion it's for prototyping in a domain one's not necessarily experienced with yet. In that situation, trying to come up with the perfect (or just reasonably good enough) data model is overwhelming. So it's easier and faster to just go ahead with a document store. I like Solr because I know it best and Apache licensed and I'll likely need a search engine later. Postgres JSON type, Elastic search, Mongo or others are perfectly valid choices too. However, once the app has shaped up, I make it a priority to harden up the data model and move the core data set to a relational database (Postgres is the likely choice now). One nice thing with this migration is I can do a lot of clean up and normalization with the originally unstructured data set.

oacgnol 12y ago
I think this is the most reasonable take I've read and encompasses the whole "right tool for the job" mindset that I like. I've found that when starting a new project I get paralyzed with coming up with the perfect data model that will also scale in the future, so I'd rather punt, take the tradeoffs that come with schemaless DBs, and keep a plan to migrate to something like Postgres in the backlog. Holding on to our tools too tightly is IMO how projects get crippled.
StavrosK 12y ago
Exactly that, but I would also add that migrating things that are documents off to a document database (or leaving them there after prototyping) is perfectly fine.

untog 12y ago

One of the reasons I like Postgres having JSON. In that experimental phase, make up a schema. Then solidify it.

wereHamster 12y ago

I use RethinkDB as the backing database for a API server written in Haskell. So I have pretty strong guarantees that the data which enters the database is sane, and when I read it out Haskell automatically validates it against a very strict schema. SQL schemas are not expressive enough to describe the data types which I want to store in the database. The argument SQL vs NoSQL becomes moot when any database is degraded to essentially pure blob storage. What matters to me is ease of use, administration/maintenance burden, and of course speed. How strictly the database verifies the schema isn't even on the picture.

teej 12y ago
> SQL schemas are not expressive enough to describe the data types which I want to store in the database For my own edification, can you expand on this?
- stingraycharles 12y ago
  You will have to educate yourself on Haskell's type system and type category theory in general. The bottom line is that you can be very, very expressive using higher level types that simply is not possible in any database at all. Personally I like foreign keys, a lot, and couldnt live without them. I dont think Haskell's type system would be a replacement to that, and would rather use a hybrid of those two.
  - SamReidHughes 12y ago
    You don't need to learn anything about category theory to use Haskell well.
- wereHamster 12y ago
  I suspect with (non-standard?) SQL extensions and pl/sql functions you could add enough validation that is otherwise inexpressible (in standard SQL). But then you have to replicate some of that logic in your application code and the database. Anyways, let's start with something simple: sumtypes. While simple sumtypes could be described using enums, what if the alternatives have more data attached to them? data User = Anonymous | Registered UserData data UserData = UserData { userName :: UserName , registredAt :: UTCTime } Foreign keys is something that can't be easily verified in Haskell code. So that's something where a proper database still is useful.

jwilliams 12y ago

Hmm. Interesting anecdote. I've spent a couple of decades in and around relational databases. They tend to be half-denormalized crufty beasts too. Data integrity is less of a technology problem than people make out.