4 ms·

It's incredible how little attention is paid to data modeling and querying in the education of people entering the software engineering workforce. Getting your

It's incredible how little attention is paid to data modeling and querying in the education of people entering the software engineering workforce. Getting your database model right, on the logical and physical level, will make developing and deploying any data-driven app simpler and easier. Getting it wrong? No modern programming language or architectural pattern will save you from the worst kinds of bugs, workarounds and bad performance. Do yourself a favor and read this or any other legendary DB book.

msluyter 5y ago
To wit: I made it through a master's in CS without a database class. This reminds me of the famous Rob Pike quote: "Data dominates. If you've chosen the right data structures and organized things well, the algorithms will almost always be self-evident. Data structures, not algorithms, are central to programming." I've often found that if I'm coding something and the code starts looking increasingly gnarly, that rethinking the data structures / data model will clean up the code.
- AlphaSite 5y ago
  Although it says data structures, isn’t it more important to talk about the structure of the data than the choice of data structure?
- bshipp 5y ago
  every one of my data scraping projects is littered with files titled "database.db.bak1", "database.db.bak2".... for this exact reason. "Oh I scraped 1000 pages and realized I missed an entire nested data structure....better blow away the dB and try again.
- gautamdivgi 5y ago
  We had two books when I was doing my bachelors in CS many moons ago (late 90's). One was by Silberschatz & Galvin and the other by Ullman. The first one was used for relational algebra, various normal forms and introducing table design, keys, constraints,etc. The second was used to teach the theory for internals on how a DB is implemented - B+ trees, deadlock handling, etc, etc. That was a hard course and it was mandatory. I can see not electing to have databases as a course for masters though. Masters is to allow you to specialize and be more choice driven.
- andrewl 5y ago
  On that topic, Fred Brooks, author of The Mythical Man Month, said "Show me your flowchart and conceal your tables, and I shall continue to be mystified. Show me your tables, and I won't usually need your flowchart; it'll be obvious."
- slver 5y ago
  "Data structures, not algorithms, are central to programming. Can we unify this to simply "data structures are essential to algorithms". It's really weird to put these in opposition, when algorithm with no data and data without algorithm make no sense. Data structures are encoded with use cases in mind, those use cases at least at the very low level are their algorithms.
  - Twisol 5y ago
    I think programming is more than just algorithms. System design and architecture have a real impact on the longevity and maintainability of your system, and I'd argue that data models (not structures per se, the distinction being logical vs. physical) are foundational to a good design. In contrast, algorithms play a much more focused role. Put differently: > Data structures are encoded with use cases in mind, those use cases at least at the very low level are their algorithms. As you've lampshaded, it's the use cases that are most fundamental. The data model should be designed to serve those use cases. The data structures and algorithms reflect the physical reality of the logical data model.
  - goto11 5y ago
    In the context of persistent databases, the schema is more important than the algorithms. You can change the algorithms on the fly, but the schema is fundamental. The strength of the relational model is it is separated from any specific algorithm.
    - slver 5y ago
      There are underlying algorithms and data structures in databases (b-tree, indexing, WAL etc.) and then there are application level data structures and algorithms, like the schema and how you use it in your app. So if we see data and algorithms disconnected, we're just thinking about different architectural layers.
- macando 5y ago
  > "Data dominates. If you've chosen the right data structures and organized things well, the algorithms will almost always be self-evident. Data structures, not algorithms, are central to programming." I'm saving this quote. > I've often found that if I'm coding something and the code starts looking increasingly gnarly that rethinking the data structures / data model will clean up the code. I've faced this over and over again. Writing algorithms is hard when the underlying data is not optimal. It simply invites writing complicated code and workarounds.
  - Akronymus 5y ago
    > I've faced this over and over again. Writing algorithms is hard when the underlying data is not optimal. It simply invites writing complicated code and workarounds. This is actually one of the reasons I am moving more and more to functional programming for myself. I find that modeling the data as just data makes it easier to find a good representation. And that it usually results in simpler algorithms.
achn 5y ago
This is very true, but also a hugely unfortunate reality. It amazes me that we have reached this far with “object” graph oriented data models backed with relational DBs. They are very seldom conducive to one another.
adamnemecek 5y ago
> this or any other legendary DB book. What are the other legendary DB books?
- macando 5y ago
  https://www.amazon.com/Modeling-Essentials-Third-Graeme-Simsion/dp/0126445516 https://www.amazon.com/Modeling-Essentials-Third-Graeme-Sims... My favorite. Not an easy read, it took me weeks to complete it. https://www.amazon.com/Date-Database-Writings-2000-2006-Christopher/dp/1430243082 https://www.amazon.com/Date-Database-Writings-2000-2006-Chri... Relatively unknown but great book if you want to read the thoughts of a DB heavyweight Christopher Date who collaborated with Edgar Codd. https://www.amazon.com/NoSQL-Distilled-Emerging-Polyglot-Persistence/dp/0321826620 https://www.amazon.com/NoSQL-Distilled-Emerging-Polyglot-Per... The best book to start exploring the land outside the relational world. An easy read, can be completed in two to three days.
vishnugupta 5y ago
Copy pasting my comment[1] from an earlier thread. A couple of years ago I spent quite some time trying to evaluate the tech stack (and general engineering culture) of merger/acquisition targets of my employer. It was quite a fun exercise, all said and done. I encountered all sorts; from a small team start up who had their tech sorted out more or less to a largish organisation who relied on IBM's ESB which exactly one person in their team knew how it worked!! I discovered this exact method during the third tech evaluation exercise. When the team began explaining various modules top-down and user-flows etc., I politely interrupted them and asked for DB schema. It was just on a whim because I was bored of typical one way session interrupted by me asking minor questions. Once I had a hang of their schema rest of the session was literally me telling them what their control and user flows were and them validating it. Since then it's become my magic wand to understand a new company or team. Just go directly to the schema and work backwards. Conversely, I've begun paying more attention to data modelling. Because once a data model is fixed it's very hard to change and once enough data accumulates the inertia just increases and instead if changing the data model (for the fear of data migration etc.,) the tendency is to beat the use cases to fit the data model. It's not your usual fail-fast-and-iterate thing. [1] https://news.ycombinator.com/item?id=24137997 https://news.ycombinator.com/item?id=24137997
- macando 5y ago
  > I politely interrupted them and asked for DB schema. In order of importance: 1. DB schema 2. List of dependancies 3. List of 3rd party integrations Depending on the domain, 2. and 3. can be switched.
- macintux 5y ago
  Many years ago, my employer was transitioning ERP systems from...something CA-owned whose name escapes me and I'd probably have pretty awful memories dredged up if I actually went looking for it. Anyway, we were lucky enough to have a very talented intern who was assigned the task of understanding the database schema behind the CA product in order to migrate our data to the new system. It was...horrific. Absolute nightmare of spaghetti. I'd like to say I've seen worse, but I've never seen anything even in the same ballpark. I think we finally gave up after weeks of digging into it and just started over with a mostly clean slate.
ggregoire 5y ago
Perhaps it’s a French thing but I had hours and hours of data modeling, SQL and database administration during my first 2 years of college, and a class about the foundations of databases during the 4th year, which ended with the project to build a basic relational database in Java.

msluyter 5y ago

To wit: I made it through a master's in CS without a database class. This reminds me of the famous Rob Pike quote: "Data dominates. If you've chosen the right data structures and organized things well, the algorithms will almost always be self-evident. Data structures, not algorithms, are central to programming." I've often found that if I'm coding something and the code starts looking increasingly gnarly, that rethinking the data structures / data model will clean up the code.

AlphaSite 5y ago
Although it says data structures, isn’t it more important to talk about the structure of the data than the choice of data structure?
bshipp 5y ago
every one of my data scraping projects is littered with files titled "database.db.bak1", "database.db.bak2".... for this exact reason. "Oh I scraped 1000 pages and realized I missed an entire nested data structure....better blow away the dB and try again.
gautamdivgi 5y ago
We had two books when I was doing my bachelors in CS many moons ago (late 90's). One was by Silberschatz & Galvin and the other by Ullman. The first one was used for relational algebra, various normal forms and introducing table design, keys, constraints,etc. The second was used to teach the theory for internals on how a DB is implemented - B+ trees, deadlock handling, etc, etc. That was a hard course and it was mandatory. I can see not electing to have databases as a course for masters though. Masters is to allow you to specialize and be more choice driven.
andrewl 5y ago
On that topic, Fred Brooks, author of The Mythical Man Month, said "Show me your flowchart and conceal your tables, and I shall continue to be mystified. Show me your tables, and I won't usually need your flowchart; it'll be obvious."
slver 5y ago
"Data structures, not algorithms, are central to programming. Can we unify this to simply "data structures are essential to algorithms". It's really weird to put these in opposition, when algorithm with no data and data without algorithm make no sense. Data structures are encoded with use cases in mind, those use cases at least at the very low level are their algorithms.
- Twisol 5y ago
  I think programming is more than just algorithms. System design and architecture have a real impact on the longevity and maintainability of your system, and I'd argue that data models (not structures per se, the distinction being logical vs. physical) are foundational to a good design. In contrast, algorithms play a much more focused role. Put differently: > Data structures are encoded with use cases in mind, those use cases at least at the very low level are their algorithms. As you've lampshaded, it's the use cases that are most fundamental. The data model should be designed to serve those use cases. The data structures and algorithms reflect the physical reality of the logical data model.
- goto11 5y ago
  In the context of persistent databases, the schema is more important than the algorithms. You can change the algorithms on the fly, but the schema is fundamental. The strength of the relational model is it is separated from any specific algorithm.
  - slver 5y ago
    There are underlying algorithms and data structures in databases (b-tree, indexing, WAL etc.) and then there are application level data structures and algorithms, like the schema and how you use it in your app. So if we see data and algorithms disconnected, we're just thinking about different architectural layers.
macando 5y ago
> "Data dominates. If you've chosen the right data structures and organized things well, the algorithms will almost always be self-evident. Data structures, not algorithms, are central to programming." I'm saving this quote. > I've often found that if I'm coding something and the code starts looking increasingly gnarly that rethinking the data structures / data model will clean up the code. I've faced this over and over again. Writing algorithms is hard when the underlying data is not optimal. It simply invites writing complicated code and workarounds.
- Akronymus 5y ago
  > I've faced this over and over again. Writing algorithms is hard when the underlying data is not optimal. It simply invites writing complicated code and workarounds. This is actually one of the reasons I am moving more and more to functional programming for myself. I find that modeling the data as just data makes it easier to find a good representation. And that it usually results in simpler algorithms.

achn 5y ago

This is very true, but also a hugely unfortunate reality. It amazes me that we have reached this far with “object” graph oriented data models backed with relational DBs. They are very seldom conducive to one another.

adamnemecek 5y ago

> this or any other legendary DB book. What are the other legendary DB books?

macando 5y ago
https://www.amazon.com/Modeling-Essentials-Third-Graeme-Simsion/dp/0126445516 https://www.amazon.com/Modeling-Essentials-Third-Graeme-Sims... My favorite. Not an easy read, it took me weeks to complete it. https://www.amazon.com/Date-Database-Writings-2000-2006-Christopher/dp/1430243082 https://www.amazon.com/Date-Database-Writings-2000-2006-Chri... Relatively unknown but great book if you want to read the thoughts of a DB heavyweight Christopher Date who collaborated with Edgar Codd. https://www.amazon.com/NoSQL-Distilled-Emerging-Polyglot-Persistence/dp/0321826620 https://www.amazon.com/NoSQL-Distilled-Emerging-Polyglot-Per... The best book to start exploring the land outside the relational world. An easy read, can be completed in two to three days.

vishnugupta 5y ago

Copy pasting my comment[1] from an earlier thread. A couple of years ago I spent quite some time trying to evaluate the tech stack (and general engineering culture) of merger/acquisition targets of my employer. It was quite a fun exercise, all said and done. I encountered all sorts; from a small team start up who had their tech sorted out more or less to a largish organisation who relied on IBM's ESB which exactly one person in their team knew how it worked!! I discovered this exact method during the third tech evaluation exercise. When the team began explaining various modules top-down and user-flows etc., I politely interrupted them and asked for DB schema. It was just on a whim because I was bored of typical one way session interrupted by me asking minor questions. Once I had a hang of their schema rest of the session was literally me telling them what their control and user flows were and them validating it. Since then it's become my magic wand to understand a new company or team. Just go directly to the schema and work backwards. Conversely, I've begun paying more attention to data modelling. Because once a data model is fixed it's very hard to change and once enough data accumulates the inertia just increases and instead if changing the data model (for the fear of data migration etc.,) the tendency is to beat the use cases to fit the data model. It's not your usual fail-fast-and-iterate thing. [1] https://news.ycombinator.com/item?id=24137997 https://news.ycombinator.com/item?id=24137997

macando 5y ago
> I politely interrupted them and asked for DB schema. In order of importance: 1. DB schema 2. List of dependancies 3. List of 3rd party integrations Depending on the domain, 2. and 3. can be switched.
macintux 5y ago
Many years ago, my employer was transitioning ERP systems from...something CA-owned whose name escapes me and I'd probably have pretty awful memories dredged up if I actually went looking for it. Anyway, we were lucky enough to have a very talented intern who was assigned the task of understanding the database schema behind the CA product in order to migrate our data to the new system. It was...horrific. Absolute nightmare of spaghetti. I'd like to say I've seen worse, but I've never seen anything even in the same ballpark. I think we finally gave up after weeks of digging into it and just started over with a mostly clean slate.

ggregoire 5y ago

Perhaps it’s a French thing but I had hours and hours of data modeling, SQL and database administration during my first 2 years of college, and a class about the foundations of databases during the 4th year, which ended with the project to build a basic relational database in Java.