4 ms·

The temporal aspect seems to be somewhat ignored by mainstream db development - in some fields, e.g. BI you'll have slowly changing dimensions; and the event so

by mtrn 14y ago

The temporal aspect seems to be somewhat ignored by mainstream db development - in some fields, e.g. BI you'll have slowly changing dimensions; and the event sourcing pattern promises a time-machine view on data. Since I'm writing a history-aware application at the moment, I recently looked into different patterns for this and trying a mixed strategy at the moment (SQL DB used as document store and a single event log, that accumulates changes - a lean approach though, a few hundred lines python for the data access layer; what always gets ugly is the validation, which your application must take care of). I wish, there was more hands-on material on the subject (some resources dive depth into bi-temporal modeling, but I feel your schemas can get complex (= expensive) very fast).

dustingetz 14y ago
by my understanding, datomic's persistence[1] layer is just a big huge persistent[2] data structure. there is tons of literature, books, papers, etc about persistent data structures in the functional programming community. [1][2] 'persistent' means different things here.
- calebmpeterson 14y ago
  So, Datomic is a persistent data structure persisted to nonvolatile storage... I've wanted one of these for at least the last 2 years. If only there was an open source implementation of this concept that I could run on my own hardware. Does anyone know of such a beast? And, no, I don't want to roll my own versioned/timestamped row schema in an RDBMS - been there, suffered that.
  - RehnoLindeque 14y ago
    Acid-State for haskell (http://acid-state.seize.it/ http://acid-state.seize.it/) is based on a similar ideology and in the public domain. (Oh, but minus the time dimension I'm afraid.)
    - DanWaterworth 14y ago
      Acid-State is quite different, it persists representations of operations to disk (to make it durable) and keeps the data structure in memory. I'm working on a project to bring persistent data-structures to disk ( https://github.com/DanielWaterworth/Siege https://github.com/DanielWaterworth/Siege ). It's currently undergoing a rethink, hence the lack of recent activity, but it's still in development.
  - willvarfar 14y ago
    At small (one machine) scale you could store simply store entity$attribute$time as the key for the value in any sorted key-value store e.g. levelDB. (that is, concatenate the entity, attribute name and timestamp in a lexically sorting string) At larger scale, hyperdex http://hyperdex.org/ http://hyperdex.org/ might do the job but I don't know much about it
    - DanWaterworth 14y ago
      key-values stores like levelDB are persistent [1], but they are not persistent [2]. [1] http://en.wikipedia.org/wiki/Persistence_(computer_science) http://en.wikipedia.org/wiki/Persistence_(computer_science) [2] http://en.wikipedia.org/wiki/Persistent_data_structure http://en.wikipedia.org/wiki/Persistent_data_structure
mtrimpe 14y ago
Big Data by Nathan Marz is a Manning Early Acces book (i.e. not fully written yet) that explains how to hand-roll your own custom Datomic style DB in Hadoop/HBase. It might be of interest to you given your current project.
- mtrn 14y ago
  Thanks for the tip, will take a look at it. Probably after this project, the current app must be lean (no big dependencies, just install a database scheme, pip install and go).
beagle3 14y ago
kdb+ is head and shoulders about the rest when it comes to introducing time / order into relational data. It's lean, minimalistic and ridiculously fast. Unfortunately, it is also ridiculously expensive.
- sbmassey 14y ago
  It's also easy to bring to its knees with seemingly innocent queries, and has a pretty darn opaque programmable interface in Q. (OK, the vaguely SQL-like stuff is not too hard, once you understand the differences from normal SQL, but doing anything lower level than that is awesome/crazy)
EzGraphs 14y ago
For what it is worth: http://en.wikipedia.org/wiki/Temporal_database http://en.wikipedia.org/wiki/Temporal_database A more extensive description/case study of bi-temporal database design: http://www.cs.arizona.edu/people/rts/tdbbook.pdf http://www.cs.arizona.edu/people/rts/tdbbook.pdf This case study tried to be somewhat rigorous, but from what I recall the SQL did get complex and expensive pretty quickly. There are a few attempts to create products to mask the complexity as well: http://www.timeconsult.com/Software/Software.html http://www.timeconsult.com/Software/Software.html For living in time and dealing with it regularly in a common sense sort of way, it is really rather challenging to manage effectively as data.

by mtrn 14y ago

dustingetz 14y ago

by my understanding, datomic's persistence[1] layer is just a big huge persistent[2] data structure. there is tons of literature, books, papers, etc about persistent data structures in the functional programming community. [1][2] 'persistent' means different things here.

calebmpeterson 14y ago
So, Datomic is a persistent data structure persisted to nonvolatile storage... I've wanted one of these for at least the last 2 years. If only there was an open source implementation of this concept that I could run on my own hardware. Does anyone know of such a beast? And, no, I don't want to roll my own versioned/timestamped row schema in an RDBMS - been there, suffered that.
- RehnoLindeque 14y ago
  Acid-State for haskell (http://acid-state.seize.it/ http://acid-state.seize.it/) is based on a similar ideology and in the public domain. (Oh, but minus the time dimension I'm afraid.)
  - DanWaterworth 14y ago
    Acid-State is quite different, it persists representations of operations to disk (to make it durable) and keeps the data structure in memory. I'm working on a project to bring persistent data-structures to disk ( https://github.com/DanielWaterworth/Siege https://github.com/DanielWaterworth/Siege ). It's currently undergoing a rethink, hence the lack of recent activity, but it's still in development.
- willvarfar 14y ago
  At small (one machine) scale you could store simply store entity$attribute$time as the key for the value in any sorted key-value store e.g. levelDB. (that is, concatenate the entity, attribute name and timestamp in a lexically sorting string) At larger scale, hyperdex http://hyperdex.org/ http://hyperdex.org/ might do the job but I don't know much about it
  - DanWaterworth 14y ago
    key-values stores like levelDB are persistent [1], but they are not persistent [2]. [1] http://en.wikipedia.org/wiki/Persistence_(computer_science) http://en.wikipedia.org/wiki/Persistence_(computer_science) [2] http://en.wikipedia.org/wiki/Persistent_data_structure http://en.wikipedia.org/wiki/Persistent_data_structure

mtrimpe 14y ago

Big Data by Nathan Marz is a Manning Early Acces book (i.e. not fully written yet) that explains how to hand-roll your own custom Datomic style DB in Hadoop/HBase. It might be of interest to you given your current project.

mtrn 14y ago
Thanks for the tip, will take a look at it. Probably after this project, the current app must be lean (no big dependencies, just install a database scheme, pip install and go).

beagle3 14y ago

kdb+ is head and shoulders about the rest when it comes to introducing time / order into relational data. It's lean, minimalistic and ridiculously fast. Unfortunately, it is also ridiculously expensive.

sbmassey 14y ago
It's also easy to bring to its knees with seemingly innocent queries, and has a pretty darn opaque programmable interface in Q. (OK, the vaguely SQL-like stuff is not too hard, once you understand the differences from normal SQL, but doing anything lower level than that is awesome/crazy)

EzGraphs 14y ago

For what it is worth: http://en.wikipedia.org/wiki/Temporal_database http://en.wikipedia.org/wiki/Temporal_database A more extensive description/case study of bi-temporal database design: http://www.cs.arizona.edu/people/rts/tdbbook.pdf http://www.cs.arizona.edu/people/rts/tdbbook.pdf This case study tried to be somewhat rigorous, but from what I recall the SQL did get complex and expensive pretty quickly. There are a few attempts to create products to mask the complexity as well: http://www.timeconsult.com/Software/Software.html http://www.timeconsult.com/Software/Software.html For living in time and dealing with it regularly in a common sense sort of way, it is really rather challenging to manage effectively as data.