3 ms·

I can see why the normalized approach is difficult to make performant, but it seems much simpler.

by swlkr 10y ago

arjunnarayan 10y ago
You're correct on both counts. A normalized approach is simpler, and is more difficult to make performant. So you start there, but as you scale, it becomes more and more untenable for your users as your latency goes up the wazoo. Thus, eventually your app hits a scale where you have to rewrite your code to explicitly denormalize things and maintain the denormalized views yourself. This makes any feature addition/backend update horrendously complicated, and everyone's life difficult. You slowly start losing consistency which causes "glitches in the matrix" for your users. To stop that you start adding in transactional semantics yourself at the application level, checking and cross-checking everything, and doing abort/restarts, and then finally, in an analogue to Greenspun's tenth rule[1], your application starts to contain an ad hoc, informally-specified, bug-ridden, slow implementation of a distributed database. I think there's a CircleCI blog post parody hidden inside this comment, but I'm not clever enough to materialize it[2]. [1]: Any sufficiently complicated C or Fortran program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Common Lisp., https://en.wikipedia.org/wiki/Greenspun%27s_tenth_rule https://en.wikipedia.org/wiki/Greenspun%27s_tenth_rule [2]: A reference to https://circleci.com/blog/its-the-future/ https://circleci.com/blog/its-the-future/
jacques_chester 10y ago
Broadly: Normalised data is more performant on write, because for each piece of state, there will be one and only one place to store it. It's less performant on read because you are constructing logical deductions out of a lattice of tiny pebbles of truth. It takes time to find, filter, sort and project those. Denormalisation "works" by skipping to a read-optimised schema for those applications where reads greatly outnumber writes. Which is very many. As an aside, "denormalised" doesn't have to mean "unstructured chaos". Do As Thou Wilt isn't necessarily the law of denormalising. There are well-studied, widely-used denormalisation techniques under the heading of "dimensional modelling". It's just that most of these discussions are dominated by web folk, not data warehouse folk. The shoutout to Event Sourcing is promising, given the interaction with the CQRS pattern. In short: read schema and write schema have different demand and performance characteristics. So split them up.
- LgWoodenBadger 10y ago
  I don't know that you can always conclude that normalized data is more performant for writes. Take for example the case where your normalized write consists of 5 insert statements into 5 tables, instead of 1 insert into 1 table that you'd do with a denormalized write.
  - jacques_chester 10y ago
    As always, consult your doctor, accountant, lawyer and profiler.

I can see why the normalized approach is difficult to make performant, but it seems much simpler.

by swlkr 10y ago

I can see why the normalized approach is difficult to make performant, but it seems much simpler.

You're correct on both counts. A normalized approach is simpler, and is more difficult to make performant. So you start there, but as you scale, it becomes more and more untenable for your users as your latency goes up the wazoo. Thus, eventually your app hits a scale where you have to rewrite your code to explicitly denormalize things and maintain the denormalized views yourself. This makes any feature addition/backend update horrendously complicated, and everyone's life difficult. You slowly start losing consistency which causes "glitches in the matrix" for your users. To stop that you start adding in transactional semantics yourself at the application level, checking and cross-checking everything, and doing abort/restarts, and then finally, in an analogue to Greenspun's tenth rule[1], your application starts to contain an ad hoc, informally-specified, bug-ridden, slow implementation of a distributed database. I think there's a CircleCI blog post parody hidden inside this comment, but I'm not clever enough to materialize it[2]. [1]: Any sufficiently complicated C or Fortran program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Common Lisp., https://en.wikipedia.org/wiki/Greenspun%27s_tenth_rule https://en.wikipedia.org/wiki/Greenspun%27s_tenth_rule [2]: A reference to https://circleci.com/blog/its-the-future/ https://circleci.com/blog/its-the-future/

jacques_chester 10y ago

Broadly: Normalised data is more performant on write, because for each piece of state, there will be one and only one place to store it. It's less performant on read because you are constructing logical deductions out of a lattice of tiny pebbles of truth. It takes time to find, filter, sort and project those. Denormalisation "works" by skipping to a read-optimised schema for those applications where reads greatly outnumber writes. Which is very many. As an aside, "denormalised" doesn't have to mean "unstructured chaos". Do As Thou Wilt isn't necessarily the law of denormalising. There are well-studied, widely-used denormalisation techniques under the heading of "dimensional modelling". It's just that most of these discussions are dominated by web folk, not data warehouse folk. The shoutout to Event Sourcing is promising, given the interaction with the CQRS pattern. In short: read schema and write schema have different demand and performance characteristics. So split them up.

LgWoodenBadger 10y ago
I don't know that you can always conclude that normalized data is more performant for writes. Take for example the case where your normalized write consists of 5 insert statements into 5 tables, instead of 1 insert into 1 table that you'd do with a denormalized write.
- jacques_chester 10y ago
  As always, consult your doctor, accountant, lawyer and profiler.