3 ms·
This may or may not apply to your case: About two years ago I implemented a database app (relational, Mysql) where tracking versions [of certain tables'] was a
by Maro 17y ago
This may or may not apply to your case:
About two years ago I implemented a database app (relational, Mysql) where tracking versions [of certain tables'] was a requirement. I solved it by adding a ValidFrom timestamp to the table, and now every row has an associated validity period, eg. the customer's address was X between T0 and T1, Y between T1 and T2. Unfortunately this is not some kind of convoluted backup solution, as the application logic sometimes needs to know these values at some prior time (it's a billing system).
Roughly, the basic problem is that simple queries or enforcing consistency requirements are now very complicated, since tables have to be JOINed and WHEREd on the validity periods. People (other than me) are unable to modify the data per command like SQL, imagine 3 normalized tables each with validity periods, and you're UPDATEing them but don't want to break time-consistency, it's frustrating. The client programmer basically failed at creating a usable GUI that was able to manage these versions, and I don't blame him (think of a case where you have a versioned table T1, a versioned table T2, and you're connecting them with a versioned has-a table T3). UI-wise it's kind of a nightmare.
Lesson learned: BAD IDEA. Don't do it. Fight with your manager to have the requirements changed. I've been thinking ever since what would have been a good way to do this, and I can't figure out a good way, hence I think this is a stupid requirement. Also, now I actually have a picture of the business process that drove this requirement, and although this isn't a trivial issue, I believe it could be solved by fixing certain business processes that require application logic to know about past data [in this manner].
- profquail 17y agoYou're right, it is a huge pain to do this manually, but depending on what languages and platforms you're developing for, there are libraries to handle this automatically. I've done something similar with NHibernate, where you only need to add a couple of event handlers (maybe 30-40 lines of code + a bit of configuration XML) to set this feature up. I wish I'd heard of it earlier though, since it's really quite useful (though as you said, horrendous to implement manually). There's a wiki on NHForge about it, though I don't know that it's really the best way to do it: http://nhforge.org/wikis/howtonh/creating-an-audit-log-using-nhibernate-events.aspx http://nhforge.org/wikis/howtonh/creating-an-audit-log-using...
- Maro 17y agoThanks. I've checked out your link, but I'm not sure this would solve my core problem(s). The problem is that even without versioning, a lot of the system is made up of long SQL commands containing subtables and JOINs and such. In several instances I had to optimize the queries. So I wouldn't want to use some ActiveRecord thing or generators. And this is now further complicated by versioning. Technical issues aside, the fact that everything is versioned makes it hard to reason about the data. Eg. the people in charge of "controling" are always asking me for help. Many times it turns out that there's no problem, the data is consistent, they just got confused by the versions.
- siculars 17y agofamiliarize yourself with time series data, as mentioned earlier in the thread, and read the amazon dynamo paper specifically the parts dealing with vector clocks.
- Maro 17y agoAs my startup is developing a distributed key-value store, I'm quite familiar with Amazon's Dynamo, and what I was talking about previously has nothing to do with vector clocks.
- yellowbkpk 17y agoI wanted to point out that I wasn't really trying to use this idea to version database tables -- I am trying to store datapoints that change over time (like pulse rate, temperature, pressures, etc.) and have a specific well-known key. Basically, a 3-dimensional Excel spreadsheet.