10 ms·

Death by Database

discordianfish 8y ago
The real problem here seems to be that changing the schema seems prohibitive expensive.
- ben509 8y ago
  That's a huge problem with SQL in general. It ought to be trivial to maintain multiple views of the data to allow you to migrate old clients bit by bit, but in practice it's rarely done. Also, it's a business problem. They can always ask for more cruft and IT has a huge incentive to take the short-sighted route, get promoted and move on.
  - tabtab 8y ago
    Re: "[flexible schema change is a] huge problem with SQL in general." -- I've been a proponent of the idea of "dynamic relational". You can add any column on whim to a row just by including it in an INSERT or UPDATE command, unless an explicit constraint/rule forbid it for a given table or database. You could even make tables similarly create-on-write. No DBA's needed! (within reason) Comparisons have to be done with care, though, as with any dynamic/implied typed system. But create-on-write doesn't solve cross-table relationship flexibility problems: such as changes between 1-to-1 to 1-to-many and/or to many-to-many. If you already have data, that's usually a tricky domain problem regardless of what kind of database technology you use. It's about the semantics of your data, not machines.
    - Scarbutt 8y ago
      Re: "[flexible schema change is a] huge problem with SQL in general." Graph databases mostly solve these problem but they have not seen much uptake, don't know why.
      - tabtab 8y ago
        Using "Dynamic Relational" (DR) is much more similar to existing RDBMS. You don't have to relearn much. I'm not necessarily saying DR is objectively better or worse than graph DB's, only that DR is a smaller learning curve to those using the most common RDBMS out there.
kyberias 8y ago
The author advocates starting with database design before software. Some other people advocate the opposite: the database is just a detail. I think the point is to spend time in designing a good domain model regardless of how it's stored (database).
- zzzcpan 8y ago
  Or all of them are wrong. You can't really get to a good model before you have a working software in production and before you learn about many of your broken assumptions.
  - UlisesAC4 8y ago
    What about a top to bottom and then a bottom to top design ?
    - zzzcpan 8y ago
      Top-down or bottom-up, they still don't validate any assumptions. You can spend years on design stage unable to come up with anything that actually works well for the client.
      - Izkata 8y ago
        > > and then I believe Ulises is talking about two stages: Top-down a prototype to shake out bad assumptions, then bottom-up main development for a solid foundation.
- wvenable 8y ago
  I also always start with the "database" design or the storage model or whatever you want to call it. If you do database design with proper foreign key relationships, etc, the entire rest of the design including the UI and most of the business rules can all flow directly from that. You can also do that with a good domain model but the rigors of actually having to physically construct it in a database ensures less mistakes. Now, if you're using a loosy-goosy JSON blog storage solution than there is no advantage to starting at that level.
3pt14159 8y ago
Same. I spend about 30% of my time writing out the migration and making sure I haven't missed anything. It saves so many headaches later. It's also one of the reasons I stubbornly re-implemented Rail's whitespace formatting of schema.rb into a rake task. I look at that thing 50 times a day. It needs to be readable, I don't care if it clutters up a pull request review or git diff.
- thirdsun 8y ago
  I'm sure you already know about this gem, but just in case: https://github.com/voormedia/rails-erd https://github.com/voormedia/rails-erd
dyeje 8y ago
I am surprised that their suggested solution was for a many-to-many relationship. Seems like you would want to stay one-to-many but remove the unique validations. If you're sharing address records then it seems there could be many possible frustrating cases. For example: What if someone goes to create an address, typos it, matches another customer's address, and then fixes it updating the other customer's address in the process? The other customer might accidentally send their luxury item to a stranger if they aren't paying attention during checkout. The use case with married couples is compelling, but similar to the previous scenario, what if they got divorced? The one that moves updates their address, which updates their ex's address as well. Same as the other scenario, you might end up with a customer sending their item the wrong place. These might be edge cases but in general I think it's dangerous to change things like an address without an explicit action from the user.
- Risord 8y ago
  I used once unique many-to-many addresses by making them immutable. You don't update update them, just add new one to customer and detach old one.
  - pluma 8y ago
    Sure but now you need to account for orphaned addresses. So in the scenario described you'd either end up accidentally purging the marketing addresses by cleaning up the orphans or you'd send marketing material to old addresses. That said, the story doesn't make any sense to me to begin with (privacy and junk mail concerns notwithstanding): If an address is to be shared between two people it should probably not include the recipient. But then what mailbox do you address it to if there is more than one person at the address? How do you handle business addresses (where the full shipping label may include a person, an apartment and a company)? If you keep all that info out of the "address", what is even the point of having them as separate entities? If not, how do you avoid accidentally leaking personal information if the user filled out the form incorrectly and you re-match the address to another user? If your answer is "only match if it's an exact match", how do you handle synonymous addresses (e.g. "5th Street", "Fifth Street", "5th St.")? If you treat them as distinct, why bother matching at all? Or simply: what do you gain from having addresses as separate entities (regardless of whether it's a many-to-one or many-to-many relationship with users)? If you just want to be able to say "shipping address is identical to invoice address", why not just have one or two addresses per user and never deduplicate? Have the address include the full recipient and stop trying to be clever.
castlegloom 8y ago
I worked on a codebase with a similar address problem. Oh, and there were no primary keys. Instead, each row referred back to a separate table with four keyed initials. Good database design upfront, and vigilant upkeep will keep the application layer tidier.
protoster 8y ago
Addresses is one of those categories for which people should regularly read the "falsehoods programmers believe about X" list. Here's one: https://www.mjt.me.uk/posts/falsehoods-programmers-believe-about-addresses/ https://www.mjt.me.uk/posts/falsehoods-programmers-believe-a...
- petre 8y ago
  Hmm, we have name (enter as many as you like, Li Pong Tai if you wish), address, country (select, excludes Kosovo and Western Sahara and other territories pretending to be countries), phone and email fields. Email is verified with a code. Forcing a certain format very rigidly is bad practice. Like first name, last name or state, city. What if the customer is Korean or lives on a ranch in the countryside or is a ranger and uses their work address? Or worse, lives in the Medina in Morocco.
  - wstuartcl 8y ago
    In the US there is also a trend for certain companies to use an address "fixer" or "reformater" API to automatically wither check and or fix an entered address. So bad. When I moved into a newly built apartment building a few years ago there were more than a few sites that just would not accept that the actual address was real and valid. ick.
  - deleted 8y ago
    [deleted]
- talaketu 8y ago
  Addresses are names for locations.
manigandham 8y ago
Wrong solution and nothing to do with database design. Why do the addresses even need to be in the ecommerce system to do the mailing? Couldn't they be in another table? Or a simple CSV? No reason to create fake customer records to send some spam. Something's missing in this story. But if for some reason they are forced to use the same db, multiple customers sharing the same address doesn't mean they need to share the same row in the table. Keep the 1:many relation without the unique constraint. Customers can then have multiple addresses, even if they're shared with others. Addresses are complicated too, so you might as well just have a country field + the rest and then use an API to standardize, catching errors and making queries easy.
- tynorf 8y ago
  > Addresses are complicated too so you might as well just have a country field + the rest and then use an API to standardize, which also makes future queries easy. This, big time. I worked on an ecommerce site that had to ship to Nigeria. No zip codes, often no street name or number. "Two doors north of the post office on the east side of Lagos." The couriers could find it.
  - manigandham 8y ago
    There is a system called Plus Codes: https://plus.codes/ https://plus.codes/ Designed as a way to address any location segment with a unique code, based on lat/long, and it's a great way to provide addressing for many areas that don't have formal systems.
    - craftyguy 8y ago
      Those, in a way, seem even worse compared to a system based on reference points. Where is G2HR+GW2 in relation to G2HR+GWD? Couriers in locations that use this system would need access to decode that into something that they can use to find a location, and people at locations would need a way to encode their current location. The current zip code system in the US is similar, with the exception of having been in use for about 50 years, and the number being relatively easier to remember.
wonnage 8y ago
What they're really asking for is multi-user accounts, which is a whole can of worms... also the address uniqueness requirement seems like it'll leak your customers' information. Also don't understand why the system for spamming marketing emails needs to share the same address table (and associated headaches) as actual customers. Presumably you will want to spam people who aren't in the system yet...
- tener 8y ago
  Exactly. These should simply be different databases. You could then enrich the spamming one using the client one, but no need to do the reverse...
yoava 8y ago
Very shallow post. Come on, this is not about database design. This is about making the wrong technical choice, the definition of "I have a hammer, so everything looks like a nail". What about export of the addresses, and using a second service on the export?
tabtab 8y ago
I've often found the "proper" database design is too confusing to users and/or managers. Users will have to understand the complex relationships to enter, inspect, and trouble-shoot data properly. Often they don't. Simplicity often trumps logic and the D.R.Y. Principle even if the limits (simplicity) create extra work in some situations. Further, you don't know what marketers will dream up in the future and cannot realistically anticipate enough of their harebrained ideas. Try to keep marketing separate from production when possible, but sometimes marketers and/or the bosses want something technically goofy and you have to fudge stuff to get it. Warn them about possible long-term consequences, but if they insist on Frankenstein, you just have to do it. Get your warnings in writing so that you have a record about their decision when bleep hits the fan later. Further if you avoid complaining a lot in general, then important complaints carry more weight. Otherwise, they'll mistake your important warnings for mundane ones.
- jacob019 8y ago
  Agree, the simple solution is the best solution.
- Apaec 8y ago
  The db is not only about tables, there's a reason why views exist, the users/managers should only query views, these are easy to change whenever the underlying private schema changes.
  - tabtab 8y ago
    Views are an imperfect abstraction (as usually implemented). Plus, pre-many-to-many-ing all relationships will make for a notably slower system even if views hide many-to-many-ness. (As I mentioned elsewhere, one often can't know which relationships will grow to many-to-many over time.)
alphanumeric0 8y ago
That Linus Torvald quote comes to mind: “Bad programmers worry about the code. Good programmers worry about data structures and their relationships.”
- tabtab 8y ago
  This is why I believe the future of source code management will be via relational databases instead of in hierarchical file systems. Large projects are too involved for trees. The same with GUI specifications: They are too complex to manage with OOP/OOM. Plus, a GUI spec in a database is (mostly) language-neutral so that you are not tied to a GUI library written in a specific programming language. You learn one engine, and then use it for PHP, C#, Java, Python, etc.
  - zzzcpan 8y ago
    Data structures have nothing to do with SQL though. That quote is pretty much anti-relational databases.
    - tabtab 8y ago
      Please elaborate. Non-trivial data structured will either need a database, or end up resembling one anyhow when you add features they'll eventually need as they mature (concurrency, data integrity, querying, adding indexes without changing functionality/design, backups, etc.) Note that didn't imply that Linux should designed around a DB. However, with virtualization, the idea of an "OS" may change or vanish. The view is toward "applications", with the associations between applications, users, file systems, and machinery more flexible. In that approach, a DB-centric view may make more sense. A (traditional) "OS" is a machine-centric view.
      - zzzcpan 8y ago
        Data-structures are machine-centric, not abstract relational algebra. You can understand why it's important to think about data structures for the machine. Which is a completely opposite view from relational databases, where you are supposed to not think of how data relates to the machine at all.
claydavisss 8y ago
Projects and even companies I have witnessed dying are more often than not done in by underinvesting in databases. When you start a project with a React UI and just throw whatever you need into the schema in order to make things come up on the page...you're headed for disaster. Or on the flip side...you know how databases work but you don't know how to say NO to features that are expensive. This seems to be getting worse over time. Databases are becoming a lost art.
- thrower123 8y ago
  There is a lot of hard-won knowledge that is aging out of the profession increasingly lately, I have been observing. It's tempting to blame web development, and its proverbial low bar to entry, but I think the real problem is computer UX getting too good, such that you can accomplish almost anything one would desire with a computer, without having to delve into the nasty implementation details. I am just barely starting to get grey in my beard (largely from dealing with frighteningly incompetent consulting firms, rather than age...) but I can remember the eldritch incantations dealing with autoexec.bat and DOS memory modes, or the clusterfuck of trying to get printers or new bits of hardware to work, or the panic of trying to fix BSODs when I'd trashed the system installing something dodgy from LimeWire or the shovelware bin at WalMart. The next generation coming through has been shielded from these horrors, and mostly matured in an environment where computers work reliably; and when they do fail, it is usually opaque, inscrutable, and largely hidden from their eyes. Aside from a crash-course in the scientific method for diagnosing and debugging issues, the old dodgy software world exposed one rather harshly to many of the underlying realities of the system, and our current software environments are still mostly built on those foundations, with a few dozen layers of lipstick applied to the pig. It certainly doesn't help that most instruction in software engineering either hews to the abstract and theoretical or the novel, with passing consideration of the practical realities and the history of the art. Ultimately, we write code that runs on silicon transistors, not ideal Turing machines, and in a great many fields we are retreading extensively explored ground, a hamster wheel of innovation. Every generation seems to have to need to have a go at yet another build system or object database, or rediscover the model-view-controller pattern. We delight in making endless new and exciting and broken wheels, in shameful ignorance of the hard-won lessons of the past.
candidtim 8y ago
This is why I don't think of database as an interface. Instead, create functional interface with some business meaning (REST API, GraphQL, a Java interface - whatever suits a particular use case), which will be versioned and properly maintained with backward compatibility where possible (which is, I think, almost always possible). It doesn't cancel a proper DB design though, but helps a lot when you need to change it. Or move to another DB engine altogether. You know, every problem can be solved with another level of abstraction. No sarcasm, I think it actually holds here.
- tabtab 8y ago
  While a many-to-many DB relationship can in theory model a 1-to-many relationship via constraints and/or domain logic coding, it seems kind of anti-YAGNI to me. Being that ANY 1-to-many relationship can potentially change into being many-to-many, which ones do you "pre many" and which ones are left 1-to-many? My crystal ball is not that powerful. If it were, I'd be competing with Warren Buffett instead of puzzling over database design.
davnicwil 8y ago
This isn't about database design, per se, it's about data model design in general. Starting a software project by thinking hard about the data model is always good advice, but even knowing this it is almost always impossible to design things perfectly up front. You need to hit problems square on in practice, and reshape your model to deal with them. If you think otherwise, if you really think you can think through in advance all the possible unexpected permutations of how your model might not work in practice, like with this address thing, you are just kidding yourself. It never happens that way. So, maybe the best advice is not only to start by thinking about the data model but to have a mindset of continually thinking about and changing your data model, and to build your software using languages, tools and technologies that better accomodate this.
- sooheon 8y ago
  Yeah, I think you're correct to point out the tension between designing up front and adapting to reality you learn as you go. If you haven't been exposed already, the way Datomic approaches this problem has been eye-opening for me: https://docs.datomic.com/on-prem/best-practices.html#plan-for-accretion https://docs.datomic.com/on-prem/best-practices.html#plan-fo...
- thirdsun 8y ago
  I agree and I'm surprised potential data models aren't as often discussed and debated as the choice of technology and stacks. At least to me considering and planning a lasting data model, despite knowing that it'll change, is a far more significant focus than choosing a framework - a choice which is often dictated by your experience and familiarity with those options anyway. And yet we rarely see discussions about best practices when it comes to data models. Maybe the requirements and solutions are often too specific.
  - dvfjsdhgfv 8y ago
    > I agree and I'm surprised potential data models aren't as often discussed and debated as the choice of technology and stacks. This. People can discuss stacks and tools for hours, whereas the optimal data model seems to be a boring topic, even though very often it turns out far more important in the long run.

The real problem here seems to be that changing the schema seems prohibitive expensive.

ben509 8y ago
That's a huge problem with SQL in general. It ought to be trivial to maintain multiple views of the data to allow you to migrate old clients bit by bit, but in practice it's rarely done. Also, it's a business problem. They can always ask for more cruft and IT has a huge incentive to take the short-sighted route, get promoted and move on.
- tabtab 8y ago
  Re: "[flexible schema change is a] huge problem with SQL in general." -- I've been a proponent of the idea of "dynamic relational". You can add any column on whim to a row just by including it in an INSERT or UPDATE command, unless an explicit constraint/rule forbid it for a given table or database. You could even make tables similarly create-on-write. No DBA's needed! (within reason) Comparisons have to be done with care, though, as with any dynamic/implied typed system. But create-on-write doesn't solve cross-table relationship flexibility problems: such as changes between 1-to-1 to 1-to-many and/or to many-to-many. If you already have data, that's usually a tricky domain problem regardless of what kind of database technology you use. It's about the semantics of your data, not machines.
  - Scarbutt 8y ago
    Re: "[flexible schema change is a] huge problem with SQL in general." Graph databases mostly solve these problem but they have not seen much uptake, don't know why.
    - tabtab 8y ago
      Using "Dynamic Relational" (DR) is much more similar to existing RDBMS. You don't have to relearn much. I'm not necessarily saying DR is objectively better or worse than graph DB's, only that DR is a smaller learning curve to those using the most common RDBMS out there.

The author advocates starting with database design before software. Some other people advocate the opposite: the database is just a detail. I think the point is to spend time in designing a good domain model regardless of how it's stored (database).

zzzcpan 8y ago
Or all of them are wrong. You can't really get to a good model before you have a working software in production and before you learn about many of your broken assumptions.
- UlisesAC4 8y ago
  What about a top to bottom and then a bottom to top design ?
  - zzzcpan 8y ago
    Top-down or bottom-up, they still don't validate any assumptions. You can spend years on design stage unable to come up with anything that actually works well for the client.
    - Izkata 8y ago
      > > and then I believe Ulises is talking about two stages: Top-down a prototype to shake out bad assumptions, then bottom-up main development for a solid foundation.
wvenable 8y ago
I also always start with the "database" design or the storage model or whatever you want to call it. If you do database design with proper foreign key relationships, etc, the entire rest of the design including the UI and most of the business rules can all flow directly from that. You can also do that with a good domain model but the rigors of actually having to physically construct it in a database ensures less mistakes. Now, if you're using a loosy-goosy JSON blog storage solution than there is no advantage to starting at that level.

3pt14159 8y ago

Same. I spend about 30% of my time writing out the migration and making sure I haven't missed anything. It saves so many headaches later. It's also one of the reasons I stubbornly re-implemented Rail's whitespace formatting of schema.rb into a rake task. I look at that thing 50 times a day. It needs to be readable, I don't care if it clutters up a pull request review or git diff.

thirdsun 8y ago
I'm sure you already know about this gem, but just in case: https://github.com/voormedia/rails-erd https://github.com/voormedia/rails-erd

dyeje 8y ago

I am surprised that their suggested solution was for a many-to-many relationship. Seems like you would want to stay one-to-many but remove the unique validations. If you're sharing address records then it seems there could be many possible frustrating cases. For example: What if someone goes to create an address, typos it, matches another customer's address, and then fixes it updating the other customer's address in the process? The other customer might accidentally send their luxury item to a stranger if they aren't paying attention during checkout. The use case with married couples is compelling, but similar to the previous scenario, what if they got divorced? The one that moves updates their address, which updates their ex's address as well. Same as the other scenario, you might end up with a customer sending their item the wrong place. These might be edge cases but in general I think it's dangerous to change things like an address without an explicit action from the user.

Risord 8y ago
I used once unique many-to-many addresses by making them immutable. You don't update update them, just add new one to customer and detach old one.
- pluma 8y ago
  Sure but now you need to account for orphaned addresses. So in the scenario described you'd either end up accidentally purging the marketing addresses by cleaning up the orphans or you'd send marketing material to old addresses. That said, the story doesn't make any sense to me to begin with (privacy and junk mail concerns notwithstanding): If an address is to be shared between two people it should probably not include the recipient. But then what mailbox do you address it to if there is more than one person at the address? How do you handle business addresses (where the full shipping label may include a person, an apartment and a company)? If you keep all that info out of the "address", what is even the point of having them as separate entities? If not, how do you avoid accidentally leaking personal information if the user filled out the form incorrectly and you re-match the address to another user? If your answer is "only match if it's an exact match", how do you handle synonymous addresses (e.g. "5th Street", "Fifth Street", "5th St.")? If you treat them as distinct, why bother matching at all? Or simply: what do you gain from having addresses as separate entities (regardless of whether it's a many-to-one or many-to-many relationship with users)? If you just want to be able to say "shipping address is identical to invoice address", why not just have one or two addresses per user and never deduplicate? Have the address include the full recipient and stop trying to be clever.

castlegloom 8y ago

I worked on a codebase with a similar address problem. Oh, and there were no primary keys. Instead, each row referred back to a separate table with four keyed initials. Good database design upfront, and vigilant upkeep will keep the application layer tidier.

protoster 8y ago

Addresses is one of those categories for which people should regularly read the "falsehoods programmers believe about X" list. Here's one: https://www.mjt.me.uk/posts/falsehoods-programmers-believe-about-addresses/ https://www.mjt.me.uk/posts/falsehoods-programmers-believe-a...

petre 8y ago
Hmm, we have name (enter as many as you like, Li Pong Tai if you wish), address, country (select, excludes Kosovo and Western Sahara and other territories pretending to be countries), phone and email fields. Email is verified with a code. Forcing a certain format very rigidly is bad practice. Like first name, last name or state, city. What if the customer is Korean or lives on a ranch in the countryside or is a ranger and uses their work address? Or worse, lives in the Medina in Morocco.
- wstuartcl 8y ago
  In the US there is also a trend for certain companies to use an address "fixer" or "reformater" API to automatically wither check and or fix an entered address. So bad. When I moved into a newly built apartment building a few years ago there were more than a few sites that just would not accept that the actual address was real and valid. ick.
- deleted 8y ago
  [deleted]
talaketu 8y ago
Addresses are names for locations.

manigandham 8y ago

Wrong solution and nothing to do with database design. Why do the addresses even need to be in the ecommerce system to do the mailing? Couldn't they be in another table? Or a simple CSV? No reason to create fake customer records to send some spam. Something's missing in this story. But if for some reason they are forced to use the same db, multiple customers sharing the same address doesn't mean they need to share the same row in the table. Keep the 1:many relation without the unique constraint. Customers can then have multiple addresses, even if they're shared with others. Addresses are complicated too, so you might as well just have a country field + the rest and then use an API to standardize, catching errors and making queries easy.

tynorf 8y ago
> Addresses are complicated too so you might as well just have a country field + the rest and then use an API to standardize, which also makes future queries easy. This, big time. I worked on an ecommerce site that had to ship to Nigeria. No zip codes, often no street name or number. "Two doors north of the post office on the east side of Lagos." The couriers could find it.
- manigandham 8y ago
  There is a system called Plus Codes: https://plus.codes/ https://plus.codes/ Designed as a way to address any location segment with a unique code, based on lat/long, and it's a great way to provide addressing for many areas that don't have formal systems.
  - craftyguy 8y ago
    Those, in a way, seem even worse compared to a system based on reference points. Where is G2HR+GW2 in relation to G2HR+GWD? Couriers in locations that use this system would need access to decode that into something that they can use to find a location, and people at locations would need a way to encode their current location. The current zip code system in the US is similar, with the exception of having been in use for about 50 years, and the number being relatively easier to remember.

wonnage 8y ago

What they're really asking for is multi-user accounts, which is a whole can of worms... also the address uniqueness requirement seems like it'll leak your customers' information. Also don't understand why the system for spamming marketing emails needs to share the same address table (and associated headaches) as actual customers. Presumably you will want to spam people who aren't in the system yet...

tener 8y ago
Exactly. These should simply be different databases. You could then enrich the spamming one using the client one, but no need to do the reverse...

yoava 8y ago

Very shallow post. Come on, this is not about database design. This is about making the wrong technical choice, the definition of "I have a hammer, so everything looks like a nail". What about export of the addresses, and using a second service on the export?

tabtab 8y ago

I've often found the "proper" database design is too confusing to users and/or managers. Users will have to understand the complex relationships to enter, inspect, and trouble-shoot data properly. Often they don't. Simplicity often trumps logic and the D.R.Y. Principle even if the limits (simplicity) create extra work in some situations. Further, you don't know what marketers will dream up in the future and cannot realistically anticipate enough of their harebrained ideas. Try to keep marketing separate from production when possible, but sometimes marketers and/or the bosses want something technically goofy and you have to fudge stuff to get it. Warn them about possible long-term consequences, but if they insist on Frankenstein, you just have to do it. Get your warnings in writing so that you have a record about their decision when bleep hits the fan later. Further if you avoid complaining a lot in general, then important complaints carry more weight. Otherwise, they'll mistake your important warnings for mundane ones.

jacob019 8y ago
Agree, the simple solution is the best solution.
Apaec 8y ago
The db is not only about tables, there's a reason why views exist, the users/managers should only query views, these are easy to change whenever the underlying private schema changes.
- tabtab 8y ago
  Views are an imperfect abstraction (as usually implemented). Plus, pre-many-to-many-ing all relationships will make for a notably slower system even if views hide many-to-many-ness. (As I mentioned elsewhere, one often can't know which relationships will grow to many-to-many over time.)

alphanumeric0 8y ago

That Linus Torvald quote comes to mind: “Bad programmers worry about the code. Good programmers worry about data structures and their relationships.”

tabtab 8y ago
This is why I believe the future of source code management will be via relational databases instead of in hierarchical file systems. Large projects are too involved for trees. The same with GUI specifications: They are too complex to manage with OOP/OOM. Plus, a GUI spec in a database is (mostly) language-neutral so that you are not tied to a GUI library written in a specific programming language. You learn one engine, and then use it for PHP, C#, Java, Python, etc.
- zzzcpan 8y ago
  Data structures have nothing to do with SQL though. That quote is pretty much anti-relational databases.
  - tabtab 8y ago
    Please elaborate. Non-trivial data structured will either need a database, or end up resembling one anyhow when you add features they'll eventually need as they mature (concurrency, data integrity, querying, adding indexes without changing functionality/design, backups, etc.) Note that didn't imply that Linux should designed around a DB. However, with virtualization, the idea of an "OS" may change or vanish. The view is toward "applications", with the associations between applications, users, file systems, and machinery more flexible. In that approach, a DB-centric view may make more sense. A (traditional) "OS" is a machine-centric view.
    - zzzcpan 8y ago
      Data-structures are machine-centric, not abstract relational algebra. You can understand why it's important to think about data structures for the machine. Which is a completely opposite view from relational databases, where you are supposed to not think of how data relates to the machine at all.

claydavisss 8y ago

Projects and even companies I have witnessed dying are more often than not done in by underinvesting in databases. When you start a project with a React UI and just throw whatever you need into the schema in order to make things come up on the page...you're headed for disaster. Or on the flip side...you know how databases work but you don't know how to say NO to features that are expensive. This seems to be getting worse over time. Databases are becoming a lost art.

thrower123 8y ago
There is a lot of hard-won knowledge that is aging out of the profession increasingly lately, I have been observing. It's tempting to blame web development, and its proverbial low bar to entry, but I think the real problem is computer UX getting too good, such that you can accomplish almost anything one would desire with a computer, without having to delve into the nasty implementation details. I am just barely starting to get grey in my beard (largely from dealing with frighteningly incompetent consulting firms, rather than age...) but I can remember the eldritch incantations dealing with autoexec.bat and DOS memory modes, or the clusterfuck of trying to get printers or new bits of hardware to work, or the panic of trying to fix BSODs when I'd trashed the system installing something dodgy from LimeWire or the shovelware bin at WalMart. The next generation coming through has been shielded from these horrors, and mostly matured in an environment where computers work reliably; and when they do fail, it is usually opaque, inscrutable, and largely hidden from their eyes. Aside from a crash-course in the scientific method for diagnosing and debugging issues, the old dodgy software world exposed one rather harshly to many of the underlying realities of the system, and our current software environments are still mostly built on those foundations, with a few dozen layers of lipstick applied to the pig. It certainly doesn't help that most instruction in software engineering either hews to the abstract and theoretical or the novel, with passing consideration of the practical realities and the history of the art. Ultimately, we write code that runs on silicon transistors, not ideal Turing machines, and in a great many fields we are retreading extensively explored ground, a hamster wheel of innovation. Every generation seems to have to need to have a go at yet another build system or object database, or rediscover the model-view-controller pattern. We delight in making endless new and exciting and broken wheels, in shameful ignorance of the hard-won lessons of the past.

candidtim 8y ago

This is why I don't think of database as an interface. Instead, create functional interface with some business meaning (REST API, GraphQL, a Java interface - whatever suits a particular use case), which will be versioned and properly maintained with backward compatibility where possible (which is, I think, almost always possible). It doesn't cancel a proper DB design though, but helps a lot when you need to change it. Or move to another DB engine altogether. You know, every problem can be solved with another level of abstraction. No sarcasm, I think it actually holds here.

tabtab 8y ago
While a many-to-many DB relationship can in theory model a 1-to-many relationship via constraints and/or domain logic coding, it seems kind of anti-YAGNI to me. Being that ANY 1-to-many relationship can potentially change into being many-to-many, which ones do you "pre many" and which ones are left 1-to-many? My crystal ball is not that powerful. If it were, I'd be competing with Warren Buffett instead of puzzling over database design.

davnicwil 8y ago

This isn't about database design, per se, it's about data model design in general. Starting a software project by thinking hard about the data model is always good advice, but even knowing this it is almost always impossible to design things perfectly up front. You need to hit problems square on in practice, and reshape your model to deal with them. If you think otherwise, if you really think you can think through in advance all the possible unexpected permutations of how your model might not work in practice, like with this address thing, you are just kidding yourself. It never happens that way. So, maybe the best advice is not only to start by thinking about the data model but to have a mindset of continually thinking about and changing your data model, and to build your software using languages, tools and technologies that better accomodate this.

sooheon 8y ago
Yeah, I think you're correct to point out the tension between designing up front and adapting to reality you learn as you go. If you haven't been exposed already, the way Datomic approaches this problem has been eye-opening for me: https://docs.datomic.com/on-prem/best-practices.html#plan-for-accretion https://docs.datomic.com/on-prem/best-practices.html#plan-fo...
thirdsun 8y ago
I agree and I'm surprised potential data models aren't as often discussed and debated as the choice of technology and stacks. At least to me considering and planning a lasting data model, despite knowing that it'll change, is a far more significant focus than choosing a framework - a choice which is often dictated by your experience and familiarity with those options anyway. And yet we rarely see discussions about best practices when it comes to data models. Maybe the requirements and solutions are often too specific.
- dvfjsdhgfv 8y ago
  > I agree and I'm surprised potential data models aren't as often discussed and debated as the choice of technology and stacks. This. People can discuss stacks and tools for hours, whereas the optimal data model seems to be a boring topic, even though very often it turns out far more important in the long run.