3 ms·

Modeling a system is an under-appreciated skill. In a new domain, however, this should really start with an analysis of the entire problem so as to capture both

Modeling a system is an under-appreciated skill. In a new domain, however, this should really start with an analysis of the entire problem so as to capture both the static structure of the system (i.e. class model) and its dynamic behavior (i.e. use cases). Jumping directly into a static database model tends to leave out the dynamic behavior. That might be OK in a simple CRUD app like this one, but could be a big mistake in more complex systems.

apwheele 2y ago
Data scientist, and I have had a few examples of seemingly simple "how would you build a schema" job interview questions that I had a difficult time with on the spot. So last one I remember was how would you build a product table with coupons. Ok, so two tables right, no big deal. Well, we are going to need to keep a history right? So now I need to update and have datetimes for different products and coupons. And now I should think about how to do indexes on the tables, and gosh my join to get the discounted price is that a good way to do that? Most coupons only allow a person to use them once, how the hell am I going to implement that? They probably just wanted the simple product + coupon table, but let me spin on it for quite a while like a madman.
- blowski 2y ago
  I'd say this is exactly what the interviewers wanted. They're interested in how you break down the problem, the types of solutions you consider, your understanding of the trade-offs involved. For example, I interviewed somebody who was adamant they could prevent double-booking by polling an end-point and storing the state in Redux. Fantastic JavaScript skills, terrible knowledge of databases.
- nerdponx 2y ago
  I don't like these questions. Data warehouse schema design is out of scope for data science, it's data engineering. Yes we have to do a lot of ad-hoc data engineering along the way, but it's such a strange thing to interview for in lieu of the many possible data/math/stats skills and more directly relevant programming skills. It signals a lack of respect for division of labor and specialization, and that lack of respect will be visible in the form of a stretched inefficient team. To be clear, I think the ability to be your own data engineer is a great attribute as a data scientist. I just don't think it's reasonable to expect it: it's not part of the core job description.
  - datadrivenangel 2y ago
    I expect a skilled data scientist to be able to articulate the complexities of the real world in relational data. Otherwise, how the hell will they be able to infer the real world from relational data?
    - nerdponx 2y ago
      That's fair. That's the ad-hoc data engineering I was talking about. I interpreted the question to be asking about "warehousing" moreso than just "laying out data", but I think I might have read too far into it.
petalmind 2y ago
> this should really start with an analysis of the entire problem so as to capture both the static structure of the system (i.e. class model) and its dynamic behavior (i.e. use cases). How does the deliverable for this sort of analysis look? Do you have a link to some example documents (or maybe chapters in a book or whatever)?
- munchler 2y ago
  When I did analysis for a living years ago, we used a subset of Unified Modeling Language (UML) and delivered the result as a Word document containing both the use cases and a class model. That format seems a bit heavy to me now, but the basic idea is still valid. Coincidentally, the guy who taught me this process just published a book about it, which he now calls "Agile with Blueprints": https://www.agilewithblueprints.com/ https://www.agilewithblueprints.com/ You can find a sample deliverable on GitHub here: https://github.com/AgileWithBlueprints/eRecipeBoxSampleApp/blob/V24.1.3/eRecipeBoxSystem/eRecipeBox/Documentation/eRecipeBoxArtifactsFinal.docx https://github.com/AgileWithBlueprints/eRecipeBoxSampleApp/b... Caveat: It's hard to convey the value of this approach because it shines most on complex systems that are hard to wrap your head around, and it feels like overkill when used on the kind of small project that one might want to start with. That said, I learned a lot of valuable lessons from it.

apwheele 2y ago

Data scientist, and I have had a few examples of seemingly simple "how would you build a schema" job interview questions that I had a difficult time with on the spot. So last one I remember was how would you build a product table with coupons. Ok, so two tables right, no big deal. Well, we are going to need to keep a history right? So now I need to update and have datetimes for different products and coupons. And now I should think about how to do indexes on the tables, and gosh my join to get the discounted price is that a good way to do that? Most coupons only allow a person to use them once, how the hell am I going to implement that? They probably just wanted the simple product + coupon table, but let me spin on it for quite a while like a madman.

blowski 2y ago
I'd say this is exactly what the interviewers wanted. They're interested in how you break down the problem, the types of solutions you consider, your understanding of the trade-offs involved. For example, I interviewed somebody who was adamant they could prevent double-booking by polling an end-point and storing the state in Redux. Fantastic JavaScript skills, terrible knowledge of databases.
nerdponx 2y ago
I don't like these questions. Data warehouse schema design is out of scope for data science, it's data engineering. Yes we have to do a lot of ad-hoc data engineering along the way, but it's such a strange thing to interview for in lieu of the many possible data/math/stats skills and more directly relevant programming skills. It signals a lack of respect for division of labor and specialization, and that lack of respect will be visible in the form of a stretched inefficient team. To be clear, I think the ability to be your own data engineer is a great attribute as a data scientist. I just don't think it's reasonable to expect it: it's not part of the core job description.
- datadrivenangel 2y ago
  I expect a skilled data scientist to be able to articulate the complexities of the real world in relational data. Otherwise, how the hell will they be able to infer the real world from relational data?
  - nerdponx 2y ago
    That's fair. That's the ad-hoc data engineering I was talking about. I interpreted the question to be asking about "warehousing" moreso than just "laying out data", but I think I might have read too far into it.

petalmind 2y ago

> this should really start with an analysis of the entire problem so as to capture both the static structure of the system (i.e. class model) and its dynamic behavior (i.e. use cases). How does the deliverable for this sort of analysis look? Do you have a link to some example documents (or maybe chapters in a book or whatever)?

munchler 2y ago
When I did analysis for a living years ago, we used a subset of Unified Modeling Language (UML) and delivered the result as a Word document containing both the use cases and a class model. That format seems a bit heavy to me now, but the basic idea is still valid. Coincidentally, the guy who taught me this process just published a book about it, which he now calls "Agile with Blueprints": https://www.agilewithblueprints.com/ https://www.agilewithblueprints.com/ You can find a sample deliverable on GitHub here: https://github.com/AgileWithBlueprints/eRecipeBoxSampleApp/blob/V24.1.3/eRecipeBoxSystem/eRecipeBox/Documentation/eRecipeBoxArtifactsFinal.docx https://github.com/AgileWithBlueprints/eRecipeBoxSampleApp/b... Caveat: It's hard to convey the value of this approach because it shines most on complex systems that are hard to wrap your head around, and it feels like overkill when used on the kind of small project that one might want to start with. That said, I learned a lot of valuable lessons from it.