13 ms·

Building data infrastructure that will last

162 points by andyjohnson0 2y ago

kerkeslager 2y ago
I work in a different domain (full stack development), but I think the principle here applies broadly. I tend to favor tools that have been around for a long time. A lot of the sites I have built have deployment scripts written in bash with dependencies on apt packages and repos, git to pull application code, and rsync to copy over files. It would probably be okay to update to zsh at this point. ;) I'm constantly shocked by the complexity of deployment infrastructures when I get into new projects: I've spent plenty of time working with Docker and Kubernetes and I have yet to see a case where these simplified things. As a rule, I don't throw out existing infrastructure, but if I'm doing greenfield development I never introduce containers--they simply don't do anything that can't be done more explicitly in a few lines of Bash. One of the sites I still maintain has been running for 15 years. I ported it from Fedora (not my choice) to Debian about 8 years ago, and the only thing that changed in the deployment scripts was the package manager. I switched to DigitalOcean 5 years ago and the deployment script didn't change, period. The deployment script is 81 lines of bash. git blame shows 64 of those lines are from the original commit of the file. The changes are primarily to add new packages and to change the firewall to UFW. And critically: I didn't write this script. This was written by a guy before me who just happened to have a similar deploy philosophy to me. That's a much better maintainability story than having to hire someone with your specific deploy tools on their resume.
- tazu 2y ago
  For infrastructure, I try to keep the Lindy effect [1] in mind. [1]: https://en.wikipedia.org/wiki/Lindy_effect https://en.wikipedia.org/wiki/Lindy_effect
  - deleted 2y ago
    [deleted]
  - kerkeslager 2y ago
    Exactly! Thanks, I didn't know there was a name for that.
- banku_brougham 2y ago
  This is a compelling testimonial. Someone recently posted a 'git ops' package here that was equally simple and bash based. Im interested to try, but im wondering what the rationale will be for avoiding github actions. If you could share script or snippets I would be grateful. One difficulty for me is my datastore and reporo g package is a python application using Prefect, and I manage dependency with Poetry. The 'poetry install' phase isnt always ha ds off and my script fails.
  - ambicapter 2y ago
    > but im wondering what the rationale will be for avoiding github actions. Having used them, I would say lack of test-ability and slightly unhelpful flow-control rules would be my guess. The former you'll find in any managed CI solution.
  - kerkeslager 2y ago
    > Im interested to try, but im wondering what the rationale will be for avoiding github actions. I don't. Github actions are simple enough, and easily run the scripts I'm talking about. > If you could share script or snippets I would be grateful. Maybe I'll pull something together and post it if I get the chance this week. > One difficulty for me is my datastore and reporo g package is a python application using Prefect, and I manage dependency with Poetry. > The 'poetry install' phase isnt always ha ds off and my script fails. I don't use poetry, I use pypi directly with pip, so not sure I can help you there. That sounds like a problem to debug. But notably, it's probably easier to debug than similar problems with a containerized system (which happen).
- itsoktocry 2y ago
  >I tend to favor tools that have been around for a long time If there are no early adopters, then no tools will ever make it to the "been around for a long time" stage. Someone has to use it.
  - kerkeslager 2y ago
    My clients don't pay me to do community service. And frankly, in the vast majority of cases, someone doesn't have to use it, because we simply don't need new solutions to already-solved problems. The proliferation of new solutions made by people who don't understand the existing solutions is a barrier to progress, not a driver of progress. There is a shortage of people willing to do the boring work of iterative improvement to existing tools, not a shortage of people willing to try out the latest exciting new tool. If a tool solves a problem that hasn't already been solved, then there isn't any "tool that has been around for a long time" to prefer. You use the new tool in that case because you don't have a choice. And yes, sometimes there do emerge new solutions that solve the same problems better, but that's actually extremely rare, because if there were an easy, obvious way to do it better, the original tool's developers would have done that. And then it has to be better enough to justify changing everything else that has been built around the existing solutions. This is why, incidentally, I generally support making rare breaking changes to existing tools: the cost of switching to a completely new tool is greater than the cost of making breaking changes to an existing tool.
kkfx 2y ago
Hem... Sorry but... It seems more propaganda for proprietary cloud solutions than a personal statement and actually the conclusion "do not do it yourself" tend to be regularly denied by the facts... Choosing third party, well known, FLOSS infra/open formats it's a thing, not developing their own infra with such tools in house is another.
weego 2y ago
How do you ensure the data infrastructure you’re building doesn’t get replaced as soon as you leave in the future? If this is a core conceit of the thinking then my answer is who cares? Why do you want to try and influence a situation you're not even involed in? Taking it back to the best lesson I was ever given in software engineering "don't code for every future". Do what you're asked to and not get caught up in projecting your own biases into trying to make a "solid base" for the future when you can't know the concerns of said future.
- layer8 2y ago
  You should be interested in building something that others don't want to replace as soon as you leave. That doesn't require predicting the future. It should be obvious that people not caring what comes after them is not a good thing.
  - ipaddr 2y ago
    No matter what you build someone will come along later and try to rewrite. If it is built too well with too many future cases in mind it will be too complex. If you write something simple and basic someone will try to add their complexity. If you write in one language someone will try to use something different. Same goes for framework. Write for your current requirements not some future state because people will say your work was subpar or overkill regardless because you are not around to defend your decisions and putting you down raises them up. Things you learn after 25 years.
    - Closi 2y ago
      Then let someone come along and try to rewrite and improve - but if your solution is so flimsy it forces a rewrite, it's just poorly made to start with.
      - boznz 2y ago
        Or old. Not many technical decisions stand 15+ years in any organisation
pmx 2y ago
> In one example I came in and found a data vault project that wasn’t being used. It actually had pretty good documentation. However, the team had taken so long and hadn’t fully completed the project which led to their dismissal. I feel like this is a major reason things don't get documentation. We don't get judged on it, nobody cares how good the docs are, but they DO care that we're shipping features.
- g4zj 2y ago
  I think I write decent documentation, but my target audience is usually my future self. If I can rely on my own documentation to help quickly reacquaint myself with the project later on, I generally consider it sufficient.
- cmiles74 2y ago
  The most praise I've ever received for my documentation was from the developers forced to read it after I moved on to a position with another company. It's a little unfortunate I didn't hear more about it as I was writing it, but then again it's the kind of stuff other developers find useful. Still, in my opinion, spending the time writing documentation was a net positive: it didn't take me all that long to write up and it clearly made someone else's job easier, so much so that they mentioned it to me. And, of course, I've had to read my own docs more than once.
  - yourapostasy 2y ago
    > It's a little unfortunate I didn't hear more about it as I was writing it... This is usually because most people find it faster and easier to ping the random access memory that is you while you are there instead of serially reading the documentation. I have found a non-linear correlation going in the wrong direction between the comprehensiveness of the documentation written, and the amount of requests for live assistance on the very matters covered in the documentation, when the requestee is the author of the documentation. This is where an effectively-segmented Table Of Contents comes in handy. As the author, instead of rewarding unwanted organizational behavior by giving the requestor the spoon-fed answer they seek even if the requestee knows it off the top of their head, encourage the organization a "teach them to fish" posture by pointing them in the right direction by the chapter section. "It will be somewhere in Chapter Foo" gets the requestors to familiarize themselves with the documentation structure at least in piecemeal fashion. Junior engineers who make a habit of asking questions already addressed in documentation and don't take the "LMGTFY"-type responses as a hint they should search first, ask later, I start helping by asking, "what keywords did you try searching upon in the documentation", and start a conversation about why they used only such keywords. I'm excited by the RAG-family generative AI I am able to start loading documentation into that will let me ask, "what prompts did you try upon the documentation fine-tuned AI", as these AI's are powerfully effective with functionally fuzzy keyword searches. I'll fine-tune the documentation when it looks like various terminology is used over and over by searchers, but in general this approach works great to teach juniors how to search for information. Engineers who don't quickly learn and keep asking for spoon-fed answers, enter an endless loop where they might occasionally help fine-tune the documentation, but otherwise are only pointed to the general area to search, and delegated to their colleagues who have learned to search for peer-led instruction. In over 20 years of applying this strategy, I've yet to run into an engineer who would not eventually learn, though if I did, they probably self select out of this field. In a dog fooding fashion, when I ask someone for help, after my specific ask of them, I tell them what documentation I tried availing myself of first, what terms I searched upon, how I reasoned through to the point where I'm stuck, and what my theories/models are of what I think is happening. This helps rapidly prune the tree of responses, and 80% of the time the response is along the lines of, "oh, yeah the official documentation is out of date, use this technical note instead".
mritchie712 2y ago
(disclaimer: I'm a founder in this space) > the project is either so incomplete or so lacking in a central design that the best thing to do is replace the old system I put a lot of blame here on "the modern data stack". Hundreds[0] of point-solution data tools and very few of them achieve "business outcomes" on their own. You need to stitch together 5 of them to get dashboards, 7 of them to get real time analytics, etc. We're going to see more products that achieve an outcome end-to-end. A lot of companies just want a few dashboards that give a 360 degree view of their data. They want all their data in one spot, an easy way to access it and don't want to spend fortune on it. That's what we're focused on at Definite[1]. We're built on the best open source data projects (e.g. DuckDB, Iceberge, Cube, etc.). If you decide to self host, you can use the same components, but it's generally cheaper to use us than manage all this stuff yourself. 0 - https://mattturck.com/landscape/mad2024.pdf https://mattturck.com/landscape/mad2024.pdf 1 - https://www.definite.app/ https://www.definite.app/ 2 - https://youtu.be/7FAJLc3k2Fo https://youtu.be/7FAJLc3k2Fo
- tomrod 2y ago
  I'm not a founder in this space like yourself, but I do a solid mix of consulting and building on the modern data stack for AI/ML and always appreciate a well constructed data stack.
- steveBK123 2y ago
  The "you need 5-7 different tools glued together to solve anything" is the CORE problem of the "modern data stack". It also ties very closely with Resume Driven Development. It leads to a lot of anti-patterns. For example, the 5-7 different tools are constantly changing, so after hiring some proclaimed expert.. they end up re-inventing the wheel by choosing a new combination of tools than they've used in the past, hitting various unexpected issues as they go. VERY rarely in this space do you see someone come in and go "I used these 5 tools in previous roles, they work great, and I'm going to build the best solution because I have done it before." These guys always think they need to reinvent the wheel, and then end up wrecking the car with some combination of v0.1 untested FOSS, up&coming SaaS, and their own in-house DSL.
  - 2y ago
Narhem 2y ago
[flagged]
cletus 2y ago
All technical problems are organizational problems. Put another way: any technical problem is a symptom of an organizational problem. Even at Google, which has some truly amazing homegrown technical infrastructure, you see what I called Promotion Driven Development ("PDD"). I didn't see this when it came to core technical infrastructure (eg storage, networking) but at a higher level I saw many examples of something being replaced solely (ultimately) because someobody wanted to get promoted and you don't get promoted for maintaining the thing. You get promoted for replacing the thing. The most egregious example was someone getting promoted to Principal Engineer (T8) for being the TL of something that was meant to replace existing core infrastructure before it had even shipped. In the end it didn't ship. The original thing is still there. But wait, "we learned a lot". So this happens because the organization rewards the new thing. So why is your data infrastructure being replaced? Probably because of an organizational failure and it'll have almost nothing to do with technical aspects of that infrastructure. This is true at least 90% of the time (IME). Data infrastructure is particularly bad for this because in any sufficiently large organization you will completely underestimate the impact of changing data dependencies for metrics, dashboards, monitoring, ML training and so on. Those things can be hard to find and map out and generally you only find them when they break. Sometimes they can break for years before anyone notices even when the thing is used by live production systems.
- yks 2y ago
  If your system is not getting replaced after your departure, it is likely your domain within the organization is just not attracting ambitious people.
mkl95 2y ago
> The problem I find is that many data teams get thrown into having to design data infrastructure with little experience actually setting up one in the past. Don’t get me wrong, we all start with no experience. But it can be difficult to assess what all the nuances of different tooling and designs can be. I've been there. Companies can be cheap about training and I was given none before building some sophisticated data stuff that surprisingly worked, but probably could have been simpler. I got a much better job soon after, and hopefully my replacement got some training.
- devjab 2y ago
  I think that one of the biggest issues is that a lot of training is flat out horrible. The author preaches simplicity and working directly with the business, but mean while you have entire teams of developers being taught something like Clean Architecture, SOLID, DRY and all sorts of fancy things that when used poorly leads to extreme over-abstractions. So even the most well meaning engineers with good training can go about building data structures which won’t last very long once the key people leave. Simply because what they were taught doesn’t work.
  - yfontana 2y ago
    Maybe better recognition of data architecture as a specific domain would help. I'm a data architect, I have a specific set of skills to build and maintain data platforms, but when applying for jobs I'm very often finding myself in 1 of 2 situations. 1) I'm interviewed by software developers / architects, who expect me to have strong knowledge of software development patterns (like OOP patterns or microservices design in webapps). Well I know the basics but not at senior level. That's also not what's actually needed to design and implement a data platform. 2) I'm interviewed by data engineers, and I'm expected to show expert knowledge in data pipelines implementation. That's a lot closer to what I can do, but it's still not the core of my skill set, and it's a bit frustrating not to be judged on that.
moltar 2y ago
An easy to maintain stack from my experience that almost anyone can do: - S3 for storage - Glue catalog to describe / define source data shapes - Athena to query the above - dbt for business data modelling (has Athena and glue adapter) The only difficult part I always struggle with is getting partitioning right.
- hipadev23 2y ago
  Everytime I look at S3/Glue/Athena I can’t help but feeling like the Glue layer shouldn’t be necessary and it’s instead just part of athena’s ddl
  - ianburrell 2y ago
    Athena is query engine and can use multiple catalogs. It forwards DDL queries to the catalog. Glue is the default catalog.
  - mulmen 2y ago
    Glue catalogs can be used by other query engines as well. Separating schema from compute is the foundational concept behind a data lake.
- OutOfHere 2y ago
  Is DBT really necessary? (serious question) If so, why? What would go wrong by skipping it?
  - moltar 2y ago
    No, not necessary at all. You can write queries and CTEs and create views in Athena/Glue by hand, if that’s what you prefer.
    - OutOfHere 2y ago
      I mean what does DBT offer me here that makes it worthwhile?
pphysch 2y ago
> Our really smart engineer working over time built amazing custom infrastructure > They quit and no one knows how it works Either the infrastructure wasn't "amazing" in the first place or clueless management is looking for a scapegoat. "Amazing" is an interesting word choice because a non-technical manager will be amazed by any blob of code. "Amazing" doesn't mean a straightforward, robust solution to a difficult problem.
- cwalv 2y ago
  > They quit and no one knows how it works Agreed, this is a very low bar for "amazing"
yobbo 2y ago
Yes, but this article seems to be talking to a business that has no competence in "data" outside of a handful random engineers that have no stake in the business. The advice given amounts to "avoid bad things". Resume-driven engineering etc is the result of engineers with no stake in the business future. Any solution must involve incentives against "bad things" and favour "good things".
fifilura 2y ago
Parquet+iceberg stored on s3 as a base. That is solid enough. After that comes various kinds of caches, maybe postgres for frontend? Or something streaming? But once everything is stored as files you gain the freedom to experiment or refactor.
- yfontana 2y ago
  In many cases, that means gaining the freedom of creating a huge mess (speaking from experience)
fsndz 2y ago
The problem is consultants selling bullshit as expertise are more prevalent than honest consultants. And for these bullshit consultant, selling the most unnecessarily complex solution with all the trendy keywords and making beautiful slides is all that counts. And what is funny is that customers believe all those lies.
Terr_ 2y ago
I've come to believe the opposite, promoting it as "Design for Deletion." I used to think I could make a wonderful work of art which everyone will appreciate for the ages, crafted so that every contingency is planned for, every need met... But nobody predicts future needs that well. Someday whatever I make is going to be That Stupid Thing to somebody, and they're going to be justified demolishing the whole mess, no matter how proud I may feel about it now. So instead, put effort into making it easy to remove. This often ends up reducing coupling, but--crucially--it's not the same as some enthusiastic young developer trying to decouple all the things through a meta-configurable framework. Sometimes a tight coupling is better when it's easier to reason about. The question isn't whether You Ain't Gonna Need It, the question is whether when you do need it will so much have changed that other design-aspects won't be valid anymore. It also means a level of trust towards (or helpless acceptance of) a future steward of your code.
- beardedetim 2y ago
  Cannot agree more to this sentiment. I call it "throw away code" and it's always seemed like the easiest to change in the future, and we all know everything is gonna change in the future.
  - mulmen 2y ago
    Especially if you never solve anything properly.
    - Terr_ 2y ago
      In a lot of business-software contexts, that's a given because stakeholders don't really know what they want/need either.
- deleted 2y ago
  [deleted]
- twic 2y ago
  A colleague of mine preaches this gospel. But we never actually delete anything. So now our life is lived on a shanty-town of under-engineered throwaway codebases. I'm not saying that our life would be better if we gold-plated anything; it would probably just suck differently. But so far, IME, design for deletion hasn't really delivered.
cgio 2y ago
Missing a most important part in my opinion, store data using open standards in an accessible platform to enable rather than anticipate evolution. Options would be e.g. Postgres, Parquet, Avro, Json, even csv. Storage is the foundational, absolutely infrastructural data infrastructure. No one cares if data pipelines infrastructure changes, but if it cannot be done, just because your data is hosed into a vendor locked-in platform, then that is the infrastructure failure you did not want in your conscience.
alexpotato 2y ago
Here is the Big Investment Bank version of "resume driven development": - "Legacy" trading system in place (old but battle tested) - New head of technology for the business comes in - They get the bright idea to retire the old system and roll out a new system - They promise ridiculous deadlines - Roll out a half baked new system and retire AT MOST 60% of the old system - They bounce to the next role at another firm b/c they have "retired legacy trading system / rolled out new system" on their resume Meanwhile, the new system ALWAYS has some giant outage or near miss due to the rushed deadlines.
xiaodai 2y ago
whoever thought spark based on Scala is a good idea should get shot.
mannyv 2y ago
Design so you can replace the design when needed.
4WIW 2y ago
I did designs that overstayed their intended lifespan by 15 years. I did designs that were cancelled even before being fully implemented. Most however had their predictable lifespan of 3-6 years. It seems to me that the key is to make useful product based on sound technical decisions; entropy (a thing you can't control) will handle the rest.

162 points by andyjohnson0 2y ago

I work in a different domain (full stack development), but I think the principle here applies broadly. I tend to favor tools that have been around for a long time. A lot of the sites I have built have deployment scripts written in bash with dependencies on apt packages and repos, git to pull application code, and rsync to copy over files. It would probably be okay to update to zsh at this point. ;) I'm constantly shocked by the complexity of deployment infrastructures when I get into new projects: I've spent plenty of time working with Docker and Kubernetes and I have yet to see a case where these simplified things. As a rule, I don't throw out existing infrastructure, but if I'm doing greenfield development I never introduce containers--they simply don't do anything that can't be done more explicitly in a few lines of Bash. One of the sites I still maintain has been running for 15 years. I ported it from Fedora (not my choice) to Debian about 8 years ago, and the only thing that changed in the deployment scripts was the package manager. I switched to DigitalOcean 5 years ago and the deployment script didn't change, period. The deployment script is 81 lines of bash. git blame shows 64 of those lines are from the original commit of the file. The changes are primarily to add new packages and to change the firewall to UFW. And critically: I didn't write this script. This was written by a guy before me who just happened to have a similar deploy philosophy to me. That's a much better maintainability story than having to hire someone with your specific deploy tools on their resume.

tazu 2y ago
For infrastructure, I try to keep the Lindy effect [1] in mind. [1]: https://en.wikipedia.org/wiki/Lindy_effect https://en.wikipedia.org/wiki/Lindy_effect
- deleted 2y ago
  [deleted]
- kerkeslager 2y ago
  Exactly! Thanks, I didn't know there was a name for that.
banku_brougham 2y ago
This is a compelling testimonial. Someone recently posted a 'git ops' package here that was equally simple and bash based. Im interested to try, but im wondering what the rationale will be for avoiding github actions. If you could share script or snippets I would be grateful. One difficulty for me is my datastore and reporo g package is a python application using Prefect, and I manage dependency with Poetry. The 'poetry install' phase isnt always ha ds off and my script fails.
- ambicapter 2y ago
  > but im wondering what the rationale will be for avoiding github actions. Having used them, I would say lack of test-ability and slightly unhelpful flow-control rules would be my guess. The former you'll find in any managed CI solution.
- kerkeslager 2y ago
  > Im interested to try, but im wondering what the rationale will be for avoiding github actions. I don't. Github actions are simple enough, and easily run the scripts I'm talking about. > If you could share script or snippets I would be grateful. Maybe I'll pull something together and post it if I get the chance this week. > One difficulty for me is my datastore and reporo g package is a python application using Prefect, and I manage dependency with Poetry. > The 'poetry install' phase isnt always ha ds off and my script fails. I don't use poetry, I use pypi directly with pip, so not sure I can help you there. That sounds like a problem to debug. But notably, it's probably easier to debug than similar problems with a containerized system (which happen).
itsoktocry 2y ago
>I tend to favor tools that have been around for a long time If there are no early adopters, then no tools will ever make it to the "been around for a long time" stage. Someone has to use it.
- kerkeslager 2y ago
  My clients don't pay me to do community service. And frankly, in the vast majority of cases, someone doesn't have to use it, because we simply don't need new solutions to already-solved problems. The proliferation of new solutions made by people who don't understand the existing solutions is a barrier to progress, not a driver of progress. There is a shortage of people willing to do the boring work of iterative improvement to existing tools, not a shortage of people willing to try out the latest exciting new tool. If a tool solves a problem that hasn't already been solved, then there isn't any "tool that has been around for a long time" to prefer. You use the new tool in that case because you don't have a choice. And yes, sometimes there do emerge new solutions that solve the same problems better, but that's actually extremely rare, because if there were an easy, obvious way to do it better, the original tool's developers would have done that. And then it has to be better enough to justify changing everything else that has been built around the existing solutions. This is why, incidentally, I generally support making rare breaking changes to existing tools: the cost of switching to a completely new tool is greater than the cost of making breaking changes to an existing tool.

kkfx 2y ago

Hem... Sorry but... It seems more propaganda for proprietary cloud solutions than a personal statement and actually the conclusion "do not do it yourself" tend to be regularly denied by the facts... Choosing third party, well known, FLOSS infra/open formats it's a thing, not developing their own infra with such tools in house is another.

weego 2y ago

How do you ensure the data infrastructure you’re building doesn’t get replaced as soon as you leave in the future? If this is a core conceit of the thinking then my answer is who cares? Why do you want to try and influence a situation you're not even involed in? Taking it back to the best lesson I was ever given in software engineering "don't code for every future". Do what you're asked to and not get caught up in projecting your own biases into trying to make a "solid base" for the future when you can't know the concerns of said future.

layer8 2y ago
You should be interested in building something that others don't want to replace as soon as you leave. That doesn't require predicting the future. It should be obvious that people not caring what comes after them is not a good thing.
- ipaddr 2y ago
  No matter what you build someone will come along later and try to rewrite. If it is built too well with too many future cases in mind it will be too complex. If you write something simple and basic someone will try to add their complexity. If you write in one language someone will try to use something different. Same goes for framework. Write for your current requirements not some future state because people will say your work was subpar or overkill regardless because you are not around to defend your decisions and putting you down raises them up. Things you learn after 25 years.
  - Closi 2y ago
    Then let someone come along and try to rewrite and improve - but if your solution is so flimsy it forces a rewrite, it's just poorly made to start with.
    - boznz 2y ago
      Or old. Not many technical decisions stand 15+ years in any organisation

pmx 2y ago

> In one example I came in and found a data vault project that wasn’t being used. It actually had pretty good documentation. However, the team had taken so long and hadn’t fully completed the project which led to their dismissal. I feel like this is a major reason things don't get documentation. We don't get judged on it, nobody cares how good the docs are, but they DO care that we're shipping features.

g4zj 2y ago
I think I write decent documentation, but my target audience is usually my future self. If I can rely on my own documentation to help quickly reacquaint myself with the project later on, I generally consider it sufficient.
cmiles74 2y ago
The most praise I've ever received for my documentation was from the developers forced to read it after I moved on to a position with another company. It's a little unfortunate I didn't hear more about it as I was writing it, but then again it's the kind of stuff other developers find useful. Still, in my opinion, spending the time writing documentation was a net positive: it didn't take me all that long to write up and it clearly made someone else's job easier, so much so that they mentioned it to me. And, of course, I've had to read my own docs more than once.
- yourapostasy 2y ago
  > It's a little unfortunate I didn't hear more about it as I was writing it... This is usually because most people find it faster and easier to ping the random access memory that is you while you are there instead of serially reading the documentation. I have found a non-linear correlation going in the wrong direction between the comprehensiveness of the documentation written, and the amount of requests for live assistance on the very matters covered in the documentation, when the requestee is the author of the documentation. This is where an effectively-segmented Table Of Contents comes in handy. As the author, instead of rewarding unwanted organizational behavior by giving the requestor the spoon-fed answer they seek even if the requestee knows it off the top of their head, encourage the organization a "teach them to fish" posture by pointing them in the right direction by the chapter section. "It will be somewhere in Chapter Foo" gets the requestors to familiarize themselves with the documentation structure at least in piecemeal fashion. Junior engineers who make a habit of asking questions already addressed in documentation and don't take the "LMGTFY"-type responses as a hint they should search first, ask later, I start helping by asking, "what keywords did you try searching upon in the documentation", and start a conversation about why they used only such keywords. I'm excited by the RAG-family generative AI I am able to start loading documentation into that will let me ask, "what prompts did you try upon the documentation fine-tuned AI", as these AI's are powerfully effective with functionally fuzzy keyword searches. I'll fine-tune the documentation when it looks like various terminology is used over and over by searchers, but in general this approach works great to teach juniors how to search for information. Engineers who don't quickly learn and keep asking for spoon-fed answers, enter an endless loop where they might occasionally help fine-tune the documentation, but otherwise are only pointed to the general area to search, and delegated to their colleagues who have learned to search for peer-led instruction. In over 20 years of applying this strategy, I've yet to run into an engineer who would not eventually learn, though if I did, they probably self select out of this field. In a dog fooding fashion, when I ask someone for help, after my specific ask of them, I tell them what documentation I tried availing myself of first, what terms I searched upon, how I reasoned through to the point where I'm stuck, and what my theories/models are of what I think is happening. This helps rapidly prune the tree of responses, and 80% of the time the response is along the lines of, "oh, yeah the official documentation is out of date, use this technical note instead".

mritchie712 2y ago

(disclaimer: I'm a founder in this space) > the project is either so incomplete or so lacking in a central design that the best thing to do is replace the old system I put a lot of blame here on "the modern data stack". Hundreds[0] of point-solution data tools and very few of them achieve "business outcomes" on their own. You need to stitch together 5 of them to get dashboards, 7 of them to get real time analytics, etc. We're going to see more products that achieve an outcome end-to-end. A lot of companies just want a few dashboards that give a 360 degree view of their data. They want all their data in one spot, an easy way to access it and don't want to spend fortune on it. That's what we're focused on at Definite[1]. We're built on the best open source data projects (e.g. DuckDB, Iceberge, Cube, etc.). If you decide to self host, you can use the same components, but it's generally cheaper to use us than manage all this stuff yourself. 0 - https://mattturck.com/landscape/mad2024.pdf https://mattturck.com/landscape/mad2024.pdf 1 - https://www.definite.app/ https://www.definite.app/ 2 - https://youtu.be/7FAJLc3k2Fo https://youtu.be/7FAJLc3k2Fo

tomrod 2y ago
I'm not a founder in this space like yourself, but I do a solid mix of consulting and building on the modern data stack for AI/ML and always appreciate a well constructed data stack.
steveBK123 2y ago
The "you need 5-7 different tools glued together to solve anything" is the CORE problem of the "modern data stack". It also ties very closely with Resume Driven Development. It leads to a lot of anti-patterns. For example, the 5-7 different tools are constantly changing, so after hiring some proclaimed expert.. they end up re-inventing the wheel by choosing a new combination of tools than they've used in the past, hitting various unexpected issues as they go. VERY rarely in this space do you see someone come in and go "I used these 5 tools in previous roles, they work great, and I'm going to build the best solution because I have done it before." These guys always think they need to reinvent the wheel, and then end up wrecking the car with some combination of v0.1 untested FOSS, up&coming SaaS, and their own in-house DSL.
- 2y ago

Narhem 2y ago

[flagged]

cletus 2y ago

All technical problems are organizational problems. Put another way: any technical problem is a symptom of an organizational problem. Even at Google, which has some truly amazing homegrown technical infrastructure, you see what I called Promotion Driven Development ("PDD"). I didn't see this when it came to core technical infrastructure (eg storage, networking) but at a higher level I saw many examples of something being replaced solely (ultimately) because someobody wanted to get promoted and you don't get promoted for maintaining the thing. You get promoted for replacing the thing. The most egregious example was someone getting promoted to Principal Engineer (T8) for being the TL of something that was meant to replace existing core infrastructure before it had even shipped. In the end it didn't ship. The original thing is still there. But wait, "we learned a lot". So this happens because the organization rewards the new thing. So why is your data infrastructure being replaced? Probably because of an organizational failure and it'll have almost nothing to do with technical aspects of that infrastructure. This is true at least 90% of the time (IME). Data infrastructure is particularly bad for this because in any sufficiently large organization you will completely underestimate the impact of changing data dependencies for metrics, dashboards, monitoring, ML training and so on. Those things can be hard to find and map out and generally you only find them when they break. Sometimes they can break for years before anyone notices even when the thing is used by live production systems.

yks 2y ago
If your system is not getting replaced after your departure, it is likely your domain within the organization is just not attracting ambitious people.

mkl95 2y ago

> The problem I find is that many data teams get thrown into having to design data infrastructure with little experience actually setting up one in the past. Don’t get me wrong, we all start with no experience. But it can be difficult to assess what all the nuances of different tooling and designs can be. I've been there. Companies can be cheap about training and I was given none before building some sophisticated data stuff that surprisingly worked, but probably could have been simpler. I got a much better job soon after, and hopefully my replacement got some training.

devjab 2y ago
I think that one of the biggest issues is that a lot of training is flat out horrible. The author preaches simplicity and working directly with the business, but mean while you have entire teams of developers being taught something like Clean Architecture, SOLID, DRY and all sorts of fancy things that when used poorly leads to extreme over-abstractions. So even the most well meaning engineers with good training can go about building data structures which won’t last very long once the key people leave. Simply because what they were taught doesn’t work.
- yfontana 2y ago
  Maybe better recognition of data architecture as a specific domain would help. I'm a data architect, I have a specific set of skills to build and maintain data platforms, but when applying for jobs I'm very often finding myself in 1 of 2 situations. 1) I'm interviewed by software developers / architects, who expect me to have strong knowledge of software development patterns (like OOP patterns or microservices design in webapps). Well I know the basics but not at senior level. That's also not what's actually needed to design and implement a data platform. 2) I'm interviewed by data engineers, and I'm expected to show expert knowledge in data pipelines implementation. That's a lot closer to what I can do, but it's still not the core of my skill set, and it's a bit frustrating not to be judged on that.

moltar 2y ago

An easy to maintain stack from my experience that almost anyone can do: - S3 for storage - Glue catalog to describe / define source data shapes - Athena to query the above - dbt for business data modelling (has Athena and glue adapter) The only difficult part I always struggle with is getting partitioning right.

hipadev23 2y ago
Everytime I look at S3/Glue/Athena I can’t help but feeling like the Glue layer shouldn’t be necessary and it’s instead just part of athena’s ddl
- ianburrell 2y ago
  Athena is query engine and can use multiple catalogs. It forwards DDL queries to the catalog. Glue is the default catalog.
- mulmen 2y ago
  Glue catalogs can be used by other query engines as well. Separating schema from compute is the foundational concept behind a data lake.
OutOfHere 2y ago
Is DBT really necessary? (serious question) If so, why? What would go wrong by skipping it?
- moltar 2y ago
  No, not necessary at all. You can write queries and CTEs and create views in Athena/Glue by hand, if that’s what you prefer.
  - OutOfHere 2y ago
    I mean what does DBT offer me here that makes it worthwhile?

pphysch 2y ago

> Our really smart engineer working over time built amazing custom infrastructure > They quit and no one knows how it works Either the infrastructure wasn't "amazing" in the first place or clueless management is looking for a scapegoat. "Amazing" is an interesting word choice because a non-technical manager will be amazed by any blob of code. "Amazing" doesn't mean a straightforward, robust solution to a difficult problem.

cwalv 2y ago
> They quit and no one knows how it works Agreed, this is a very low bar for "amazing"

yobbo 2y ago

Yes, but this article seems to be talking to a business that has no competence in "data" outside of a handful random engineers that have no stake in the business. The advice given amounts to "avoid bad things". Resume-driven engineering etc is the result of engineers with no stake in the business future. Any solution must involve incentives against "bad things" and favour "good things".

fifilura 2y ago

Parquet+iceberg stored on s3 as a base. That is solid enough. After that comes various kinds of caches, maybe postgres for frontend? Or something streaming? But once everything is stored as files you gain the freedom to experiment or refactor.

yfontana 2y ago
In many cases, that means gaining the freedom of creating a huge mess (speaking from experience)

fsndz 2y ago

The problem is consultants selling bullshit as expertise are more prevalent than honest consultants. And for these bullshit consultant, selling the most unnecessarily complex solution with all the trendy keywords and making beautiful slides is all that counts. And what is funny is that customers believe all those lies.

Terr_ 2y ago

I've come to believe the opposite, promoting it as "Design for Deletion." I used to think I could make a wonderful work of art which everyone will appreciate for the ages, crafted so that every contingency is planned for, every need met... But nobody predicts future needs that well. Someday whatever I make is going to be That Stupid Thing to somebody, and they're going to be justified demolishing the whole mess, no matter how proud I may feel about it now. So instead, put effort into making it easy to remove. This often ends up reducing coupling, but--crucially--it's not the same as some enthusiastic young developer trying to decouple all the things through a meta-configurable framework. Sometimes a tight coupling is better when it's easier to reason about. The question isn't whether You Ain't Gonna Need It, the question is whether when you do need it will so much have changed that other design-aspects won't be valid anymore. It also means a level of trust towards (or helpless acceptance of) a future steward of your code.

beardedetim 2y ago
Cannot agree more to this sentiment. I call it "throw away code" and it's always seemed like the easiest to change in the future, and we all know everything is gonna change in the future.
- mulmen 2y ago
  Especially if you never solve anything properly.
  - Terr_ 2y ago
    In a lot of business-software contexts, that's a given because stakeholders don't really know what they want/need either.
deleted 2y ago
[deleted]
twic 2y ago
A colleague of mine preaches this gospel. But we never actually delete anything. So now our life is lived on a shanty-town of under-engineered throwaway codebases. I'm not saying that our life would be better if we gold-plated anything; it would probably just suck differently. But so far, IME, design for deletion hasn't really delivered.

cgio 2y ago

Missing a most important part in my opinion, store data using open standards in an accessible platform to enable rather than anticipate evolution. Options would be e.g. Postgres, Parquet, Avro, Json, even csv. Storage is the foundational, absolutely infrastructural data infrastructure. No one cares if data pipelines infrastructure changes, but if it cannot be done, just because your data is hosed into a vendor locked-in platform, then that is the infrastructure failure you did not want in your conscience.

alexpotato 2y ago

Here is the Big Investment Bank version of "resume driven development": - "Legacy" trading system in place (old but battle tested) - New head of technology for the business comes in - They get the bright idea to retire the old system and roll out a new system - They promise ridiculous deadlines - Roll out a half baked new system and retire AT MOST 60% of the old system - They bounce to the next role at another firm b/c they have "retired legacy trading system / rolled out new system" on their resume Meanwhile, the new system ALWAYS has some giant outage or near miss due to the rushed deadlines.

xiaodai 2y ago

whoever thought spark based on Scala is a good idea should get shot.

mannyv 2y ago

Design so you can replace the design when needed.

4WIW 2y ago

I did designs that overstayed their intended lifespan by 15 years. I did designs that were cancelled even before being fully implemented. Most however had their predictable lifespan of 3-6 years. It seems to me that the key is to make useful product based on sound technical decisions; entropy (a thing you can't control) will handle the rest.