19 ms·

In defense of shallow technical knowledge

108 points by swah 1y ago

harrall 1y ago
I think it's critical to have both (1) actual experience working with something and (2) then reading about how it works. You gain natural intuition when you have done both. When I'm listening to someone suggest an idea, you can tell - if they're just working off something that they heard about (i.e. let's implement Redis, I heard it was fast), - if they've done it before but don't really know how it works (i.e. MongoDB worked for me once, don't need to consider Postgres), - or if they did both
nico 1y ago
> requires having reliable shallow intuitions about how things work This is very insightful and applies to many fields, maybe specially within STEM Like for example the water flow analogy of current. It’s a great analogy that works to a great degree to explain and model a lot of things about electricity, but eventually breaks down For 99%+ of the people and cases, the approximate analogy is perfectly useful
palmotea 1y ago
I think the title is a little, I don't know, trollish? Unnecessarily controversial?, but I think the insight is true: it's very good to have some knowledge about things outside of "your area," instead of being too hyperfocused. However, I think that's a broader thing than just "technical" knowledge: you should know a little about what your customers do, what your manager does, what the role is of systems peripheral to yours is, etc.
- dfxm12 1y ago
  On first pass, it seems vague enough to border on click bait. Shallow compared to what? No knowledge? Adept knowledge? Expert knowledge? Is the argument we should intentionally stop at "shallow"? Why isn't this clear? But, with the added context that it's from a personal blog, we should give the benefit of the doubt that the author is just not good at writing headlines and give the article a shot on its own merits...
al_borland 1y ago
Over the last decade or so I’ve seen more and more people dismiss the idea of universities, and more generally, dismissing the idea of learning anything that isn’t explicitly needed for the career they’re studying for. This always felt like a huge mistake to me. Universities have their problems, but getting students to see the value in subjects on the fringe, or completely outside, of their primary field of study is not one of them. These are the places new and novel solutions are born. Even if someone isn’t an expert, knowing enough to bring in an expert and facilitating the conversation can pay dividends. I was once tasked with getting a new team up to speed quickly in a new site we were standing up. The manager at the time wanted to forgo training entirely to just let them figure it out, in the name of speed. I dug my heels in and still ran everyone through it. With some, it was extremely fast, and there was no way they were going to absorb it. However, I wanted them to at least hear it, so if something came up, they may not know what to do, but they will hopefully at least know enough to ask, so we can then dive deeper and show them the right way. The company had its own way of doing almost everyone, so someone doing what they thought was right based on previous experience often led to a mess.
- wagwang 1y ago
  Universities in the US are a giant clusterfuck because they confuse themselves between centers of status/prestige, a place for learning/training, and a semi-professional sports team.
  - senderista 1y ago
    Not sure why you're getting downvotes; I think this is quite accurate.
    - riskassessment 1y ago
      They did not mention research which is like the defining characteristic of a university. They seem to be condescendingly grouping research into the category of "status/prestige"
      - 1y ago
pmarreck 1y ago
AI makes it completely defensible these days, as long as you have some "dev intuition". Example: Someone just handed me a USB key with some old DOS software on it called SAGE made by a company which disappeared in 2011 (which apparently still powers a number of law offices... shocker), and they encountered a date handling issue (any case entered for 2026 borks some views and reports) and I'm facing either a decompilation or some other heavy analysis (none of which I'm familiar with) since the source code is obviously unobtainable (score another for open source? Seriously, when companies sink, they should be required to release their source code...). I'm not doing this for free, of course (I did tell them that failure was a distinct possibility), but I'm going to attack it with AI and rely on dev intuition for the rest and see what happens... (and use some kind of test harness, of course... to prevent regressions and prove the fixes)
- glitchc 1y ago
  "Going to" is an aspiration. "Having done" is an achievement. Let us know how it goes.
- layer8 1y ago
  It’s also a very distinct possibility that this will be a counterexample.
- Karrot_Kream 1y ago
  Would be cool if you document the process, even if it ends in failure.
- wiseowise 1y ago
  Someone just asked me to become world champion arm wrestler, I’m going to attack it with some biceps curls and see what happens.
photochemsyn 1y ago
I'd like to see an expert in the field of machine learning generate a cursed iceberg diagram specific just to machine learning. I suspect an LLM would do OK on generating the upper levels but very poorly on the lowest levels. https://news.ycombinator.com/item?id=26766722 https://news.ycombinator.com/item?id=26766722 (307 pts 60 comments) https://news.ycombinator.com/item?id=32612931 https://news.ycombinator.com/item?id=32612931 (341 pts 39 comments)
tetha 1y ago
This is something I am observing at work, teaching a few people and observing in some other people outside of work: It is good to understand how your stuff touches adjacent stuff, and how adjacent things touch your stuff. This eventually enables communication across layers and teams as well. To pick up one one of his examples, a few people at work understand Postgres very, very well. But some of them have troubles to discuss topics with developers, because they have no knowledge how application servers interact with the Postgres (usually via some pooling, sometimes not), how different kinds of applications have different query patterns (think REST-based applications that are heavily indexed for low-quantity retrievals vs ETL based applications) and so on. I can't write a production ready app in Rails, Spring, Django right now, or a data analysis system in Flink, Spark or whatever, but I tend to have an idea what the dev needs to do so. On the flipside, if you have a motivated customer or provider, I find it very valuable to spend some time to show and teach them how our systems want to be touched, one way or another. Some "idle" or "non productive" time with some senior-ish devs just sharing ideas and knowledge how our Postgres functions, some somewhat unintuitive thoughts like index selectiveness, and wants to work at a somewhat shallow level has paid off a lot at work. Suddenly people ask good questions about the postgres ecosystem before starting their project and such so they don't have to spend time building a postgres extension in a worse way in their application. How silly is that.
FailMore 1y ago
Here here!
hnthrow90348765 1y ago
The depth of knowledge required to work on something tends to scale with how much performance is needed, or how large the impact will be. I feel this should be obvious, but maybe the reason we're having to defend shallow technical knowledge here might be because we're getting pushed to understand things deeply regardless of the performance/impact of the actual jobs. I also think it should be fairly obvious that some jobs can be accomplished by only having shallow or ad-hoc knowledge because they work on low performance/impact, but are still needed by someone, and thus require an employee. What has not been obvious is why we can't differentiate roles that require deep vs. shallow knowledge officially, because there is still quite a lot of ambiguity in the actual work demands of "Software Engineer" (or "Software Developer") which makes this kind of defense in the OP necessary.
- mlinhares 1y ago
  Most of the time you should be working on both, sometimes you do shallow work, sometimes you do deep work and you naturally build a deep understanding of the pieces of the stack you mostly work with. I find it hard to believe anyone does serious development work without some deep understanding of a piece of what they work on, the folks I've met that did that didn't last multiple review cycles.
- kixiQu 1y ago
  Some people think I'm effective at work because I have a deep understanding of the system my team owns. There are projects we take on and decisions we make that do depend on that understanding. However, what's far more important 80% of the time are the wide-ranging shallow understandings I have of other systems that we interact with (sometimes directly, sometimes highly indirectly). I can't imagine being able to be successful in my role without both, but I see folks overindexing on the former pretty frequently.
lubujackson 1y ago
Giving yourselves breadcrumbs to knowledge is a fantastic life skill... but one that may (even more) fall out of favor with AI replacing the value. If I suddenly need to add a caching layer and have learned nothing about it I can ask AI to give me an overview and go ahead and recommend the best solution for my codebase and situation. Yes, you may miss some subtly and have to flail around to ask the right questions, but there is no doubt the value of "shallow technical knowledge" is less important. What will remain important or grow in importance is general curiosity. Connecting completely disparate ideas or ways of thinking will lead you to new creative thoughts or solutions that AI would never produce because everyone else is working from the same standard ideas. I was an English major in college and took classes in politics, philosophy, math, language, etc. based on personal interest. And I ended up as an engineer (with my trusty CS minor). TI've met several developers who have had a similar background and they tend to become the most well-rounded and business-aware ones on the team. I worry that this shift to higher cost/higher stakes/higher competition education is making this approach to learning feel untenable and my approach of 20 years ago comes across as totally irresponsible. But I would argue American education is leading to a factory approach at exactly the time when "structured thinking" is being fully replaced by AI. What is the value of crushing leetcode nowadays? Better to have a dev that has some intuition as to why people aren't clicking that new button.
o3b 1y ago
With the pace of technology progress and the amount of new developments in computer science, it's quite easy today to remain on shallow knowledge, which could still be ok, accepted and further help progress itself. At the end of the day, one of the key point of the progress is to make difficult things easier for a larger audience. This introduces abstraction layers, one on the other. Hence you may not know how the KV Cache works in Attention mechanism, but you might still be able to train a super useful AI model. However I think those unique engineers with vertical and deep knowledge in a tech stack (e.g. C, Java, Maths under NN) are still very needed in the world, because they are capable of building and repairing the fundamentals of everything which gets built upon. So, if you are interested a such fundamental stack, hack it, crash it, it won't be wasted time and world will need you :)
tibbar 1y ago
I think the distinction here is between "I understand that X is a solution to Y", vs. "I understand that X is BASICALLY a mashup of A and B according to scheme C." A lot of times, people reach for X when they have a vaguely Y-ish problem, when that's actually inefficient or overkill, and/or you could easily validate if X would actually work by manually trying out A and B by hand. In addition, there will be times when X cannot directly solve the version of Y you have, but there are simple ways to tweak A or B such that now you do have a solution to the problem. So you can become much, much more effective at solving Y-like problems by understand the the building blocks behind standard solutions.
blindriver 1y ago
The problem with shallow technical knowledge and, even worse, talking with confidence like the author does, is that it can propagate misinformation and loses a ton of nuance. For example, the author talks very confidently about indexes, and makes a few conclusions, but they aren't as correct as his confidence suggests. > That an index is only useful if it matches the actual query your application is making. If you index on “name plus email” and then start querying on “name plus address”, your index won’t be used and you’re back to the full table scan described in (1) Not true. If you have single column indexes on both name and email, it could use the two indexes, though not as efficient as a single two-column index. If you query "name plus email" and the index is "name, email, age" then it could use the index. > That indexes massively speed up reads, but must slow down database writes and updates, because each change must also be made again in the index “dictionary” Must? No. The performance might be imperceptible and not material at all. If you have a ton of indexes, sure but not if you have a reasonable amount. Shallow technical knowledge is fine but you should also have the humility to acknowledge that when you're dispensing said shallow knowledge. It can also lead to pretty bad engineering decisions if you think your shallow knowledge is enough.
- sgarland 1y ago
  For that matter, if you index `(name, email)` and query with `name` and `address` as the predicates, unless there's a better option, or the table is tiny, there's an excellent chance the planner will use that index to narrow down the initial result set to filter.
- tibbar 1y ago
  > Not true. If you have single column indexes on both name and email, it could use the two indexes, though not as efficient as a single two-column index. If you query "name plus email" and the index is "name, email, age" then it could use the index. See, there are databases that implement clever optimizations like this, but those are going to vary widely by database and you would need some domain expertise with that system to know if such optimizations are working. By contrast, this mental model does help you ensure that you can create indexes that are actually helpful in the vast majority of databases. So I think the author's mental model is working out pretty well for him here, honestly.
godelski 1y ago
I'm reminded of a Knuth quote[0] | If you find yourself doing only theory, do some practical work. It'll help with your theory. | If you find yourself doing only practical, do some theory. It'll help with your practical work. The two complement one another. Theory informs you of what is possible while the hands on informs you on what is feasible. They are different kinds of intuitions. Theory gets you thinking in high abstractions, understanding the building blocks and how things are put together. The practice forces you to deal with everything that slips through the cracks, teaching you how to do things efficiently and effectively. It's like making a building. You technically only need the construction skills to make a building. But the engineering designs make you able to build bigger and safer. The theory lets you find new novel designs and materials. You need all of this working together to build something great. I've always found theory and practice to go hand in hand. Maybe because I came to CS from physics, but they have hard divisions there. I focused on experimental physics but I was known for strong math skills which allowed me to bridge these two sides. But the major reason I did it is because I couldn't see how they were different. So when I later heard Knuth, it really resonated with me. But truthfully, I've struggled sometimes working with others, getting pulled in different directions. I left physics to become an engineer and all they saw me as was a theorist despite most of my work for them being practical (running simulations and then building physical devices and the necessary test infrastructure). Now I'm about to defend my PhD in CS (ML) and the biggest struggle I've had is that it feels like any time I spend on theory is seen as wasted. I do so much less than before, but it's been the most helpful tool to my work. So I'm hoping others can help me understand this culture. We definitely see theory differently but I'm not sure why or how. Everything seems hyper focused on making products. To ship as fast as possible. No time to think about what we're making. For me, that slows me down! Just because lines of code aren't appearing on screen doesn't mean I'm not hard at work. A few hours or even a day of thought has saved me weeks worth of work. I'm not the quickest to first result[1] but the result is I can do more with less and quickly adapt to all the changes that happen as the project evolves. New experiments, measurements, features and such get integrated quickly because I expect this to happen. But still, considered slow, and I can't find out why. So I'm wondering how others have dealt with this. I know I'm not alone here. I'm probably not going to change much because the end result works. But why is CS so focused on short term and how do I highlight longer term work when we're just measuring weekly updates. The strategy means more weeks with "not much to show" but other weeks that look like extreme productivity. But it's all the same. Because the truth is, the earlier work is just difficult or impossible to measure. I'm pretty sure this is why CS is the way it is, but I've got to say, I've got a lot of experience with measures and analysis and the fact is not everything is measurable. We only can use proxies and those easily become unaligned. [0] I didn't copy paste it so it might be a little off but the message is right [1] after proof of concept. Early on I try to be quick to fail. I'm talking about once that's going and we're doing more serious building.
4ndrewl 1y ago
Does the description of database indexes even count as _shallow_ knowledge? Sounds like a dictionary description with an example. Not even table stakes to warrant your inclusion in any conversation on the topic. I know that some birds migrate depending on the season and they fly in certain formations for efficiency. I'd never,ever think I could have any serious conversation with a biologist or ornithologist.
- tibbar 1y ago
  I think you underestimate how abysmal many product developers' understanding of databases is:-). It's less about making people respectable experts, more about teaching basic safety principles so they won't do too much damage...
chasil 1y ago
I have absolutely no desire to become an expert in OpenVMS, but I have an account with DCL access that can run the Oracle Rdb sql utility. There have been several times that my exposure has been wrapped into production applications. I know what I need to know on that platform, and try to maintain little more. Is this what everybody does?
sgarland 1y ago
> Database [anything] Honestly, the author couldn't have chosen a worse example if they tried. RDBMS are so absurdly complex and have so many edge cases for everything, that they are my canonical answer to why it's a bad idea to have dev teams managing them, or designing schema. Disclaimer: I'm a DBRE. Look at MySQL's ORDER BY [0], GROUP BY [1], and index merge [2] optimizations. Postgres' ORDER BY optimization [3] doesn't have nearly as many gotchas, but it more than makes up for it by having many more types of indices [4], as well as many types of operator classes within each of those. While you're at it, you should probably have a handle on collations [6], AKA "why did this ORDER BY do something odd?" > You could learn about the data structures that represent the index You mean a B+ tree (for standard types)? I would hope you covered that in DS&A already. > about all the tricks that go into updating them efficiently > But it will very rarely inform the database migration or query you’re writing. I disagree. For example, if you didn't know that updating any indexed column in a Postgres table means the entire row is re-written (hope you haven't done anything silly, like storing medium-sized JSON blobs that aren't quite big enough to trigger TOAST), you would likely be much more cavalier with your indexing decisions. I'm so very tired of the "I don't need to know how the things I use work" attitude. Do you need to be able to write your own database to effectively use one? No, of course not. Should you have read the official docs? Yes. You'll learn far more than from skimming random blogs and YouTube videos. [0]: https://dev.mysql.com/doc/refman/8.0/en/order-by-optimization.html https://dev.mysql.com/doc/refman/8.0/en/order-by-optimizatio... [1]: https://dev.mysql.com/doc/refman/8.0/en/group-by-optimization.html https://dev.mysql.com/doc/refman/8.0/en/group-by-optimizatio... [2]: https://dev.mysql.com/doc/refman/8.0/en/index-merge-optimization.html https://dev.mysql.com/doc/refman/8.0/en/index-merge-optimiza... [3]: https://www.postgresql.org/docs/current/indexes-ordering.html https://www.postgresql.org/docs/current/indexes-ordering.htm... [4]: https://www.postgresql.org/docs/current/indexes-types.html https://www.postgresql.org/docs/current/indexes-types.html [5]: https://www.postgresql.org/docs/current/indexes-opclass.html https://www.postgresql.org/docs/current/indexes-opclass.html [6]: https://www.postgresql.org/docs/current/collation.html https://www.postgresql.org/docs/current/collation.html
exiguus 1y ago
If you are developing (technical) solutions, i would say that shallow knowledge is essential. As long as it allows you to make decisions.

108 points by swah 1y ago

I think it's critical to have both (1) actual experience working with something and (2) then reading about how it works. You gain natural intuition when you have done both. When I'm listening to someone suggest an idea, you can tell - if they're just working off something that they heard about (i.e. let's implement Redis, I heard it was fast), - if they've done it before but don't really know how it works (i.e. MongoDB worked for me once, don't need to consider Postgres), - or if they did both

nico 1y ago

> requires having reliable shallow intuitions about how things work This is very insightful and applies to many fields, maybe specially within STEM Like for example the water flow analogy of current. It’s a great analogy that works to a great degree to explain and model a lot of things about electricity, but eventually breaks down For 99%+ of the people and cases, the approximate analogy is perfectly useful

palmotea 1y ago

I think the title is a little, I don't know, trollish? Unnecessarily controversial?, but I think the insight is true: it's very good to have some knowledge about things outside of "your area," instead of being too hyperfocused. However, I think that's a broader thing than just "technical" knowledge: you should know a little about what your customers do, what your manager does, what the role is of systems peripheral to yours is, etc.

dfxm12 1y ago
On first pass, it seems vague enough to border on click bait. Shallow compared to what? No knowledge? Adept knowledge? Expert knowledge? Is the argument we should intentionally stop at "shallow"? Why isn't this clear? But, with the added context that it's from a personal blog, we should give the benefit of the doubt that the author is just not good at writing headlines and give the article a shot on its own merits...

al_borland 1y ago

Over the last decade or so I’ve seen more and more people dismiss the idea of universities, and more generally, dismissing the idea of learning anything that isn’t explicitly needed for the career they’re studying for. This always felt like a huge mistake to me. Universities have their problems, but getting students to see the value in subjects on the fringe, or completely outside, of their primary field of study is not one of them. These are the places new and novel solutions are born. Even if someone isn’t an expert, knowing enough to bring in an expert and facilitating the conversation can pay dividends. I was once tasked with getting a new team up to speed quickly in a new site we were standing up. The manager at the time wanted to forgo training entirely to just let them figure it out, in the name of speed. I dug my heels in and still ran everyone through it. With some, it was extremely fast, and there was no way they were going to absorb it. However, I wanted them to at least hear it, so if something came up, they may not know what to do, but they will hopefully at least know enough to ask, so we can then dive deeper and show them the right way. The company had its own way of doing almost everyone, so someone doing what they thought was right based on previous experience often led to a mess.

wagwang 1y ago
Universities in the US are a giant clusterfuck because they confuse themselves between centers of status/prestige, a place for learning/training, and a semi-professional sports team.
- senderista 1y ago
  Not sure why you're getting downvotes; I think this is quite accurate.
  - riskassessment 1y ago
    They did not mention research which is like the defining characteristic of a university. They seem to be condescendingly grouping research into the category of "status/prestige"
    - 1y ago

pmarreck 1y ago

AI makes it completely defensible these days, as long as you have some "dev intuition". Example: Someone just handed me a USB key with some old DOS software on it called SAGE made by a company which disappeared in 2011 (which apparently still powers a number of law offices... shocker), and they encountered a date handling issue (any case entered for 2026 borks some views and reports) and I'm facing either a decompilation or some other heavy analysis (none of which I'm familiar with) since the source code is obviously unobtainable (score another for open source? Seriously, when companies sink, they should be required to release their source code...). I'm not doing this for free, of course (I did tell them that failure was a distinct possibility), but I'm going to attack it with AI and rely on dev intuition for the rest and see what happens... (and use some kind of test harness, of course... to prevent regressions and prove the fixes)

glitchc 1y ago
"Going to" is an aspiration. "Having done" is an achievement. Let us know how it goes.
layer8 1y ago
It’s also a very distinct possibility that this will be a counterexample.
Karrot_Kream 1y ago
Would be cool if you document the process, even if it ends in failure.
wiseowise 1y ago
Someone just asked me to become world champion arm wrestler, I’m going to attack it with some biceps curls and see what happens.

photochemsyn 1y ago

I'd like to see an expert in the field of machine learning generate a cursed iceberg diagram specific just to machine learning. I suspect an LLM would do OK on generating the upper levels but very poorly on the lowest levels. https://news.ycombinator.com/item?id=26766722 https://news.ycombinator.com/item?id=26766722 (307 pts 60 comments) https://news.ycombinator.com/item?id=32612931 https://news.ycombinator.com/item?id=32612931 (341 pts 39 comments)

tetha 1y ago

This is something I am observing at work, teaching a few people and observing in some other people outside of work: It is good to understand how your stuff touches adjacent stuff, and how adjacent things touch your stuff. This eventually enables communication across layers and teams as well. To pick up one one of his examples, a few people at work understand Postgres very, very well. But some of them have troubles to discuss topics with developers, because they have no knowledge how application servers interact with the Postgres (usually via some pooling, sometimes not), how different kinds of applications have different query patterns (think REST-based applications that are heavily indexed for low-quantity retrievals vs ETL based applications) and so on. I can't write a production ready app in Rails, Spring, Django right now, or a data analysis system in Flink, Spark or whatever, but I tend to have an idea what the dev needs to do so. On the flipside, if you have a motivated customer or provider, I find it very valuable to spend some time to show and teach them how our systems want to be touched, one way or another. Some "idle" or "non productive" time with some senior-ish devs just sharing ideas and knowledge how our Postgres functions, some somewhat unintuitive thoughts like index selectiveness, and wants to work at a somewhat shallow level has paid off a lot at work. Suddenly people ask good questions about the postgres ecosystem before starting their project and such so they don't have to spend time building a postgres extension in a worse way in their application. How silly is that.

FailMore 1y ago

Here here!

hnthrow90348765 1y ago

The depth of knowledge required to work on something tends to scale with how much performance is needed, or how large the impact will be. I feel this should be obvious, but maybe the reason we're having to defend shallow technical knowledge here might be because we're getting pushed to understand things deeply regardless of the performance/impact of the actual jobs. I also think it should be fairly obvious that some jobs can be accomplished by only having shallow or ad-hoc knowledge because they work on low performance/impact, but are still needed by someone, and thus require an employee. What has not been obvious is why we can't differentiate roles that require deep vs. shallow knowledge officially, because there is still quite a lot of ambiguity in the actual work demands of "Software Engineer" (or "Software Developer") which makes this kind of defense in the OP necessary.

mlinhares 1y ago
Most of the time you should be working on both, sometimes you do shallow work, sometimes you do deep work and you naturally build a deep understanding of the pieces of the stack you mostly work with. I find it hard to believe anyone does serious development work without some deep understanding of a piece of what they work on, the folks I've met that did that didn't last multiple review cycles.
kixiQu 1y ago
Some people think I'm effective at work because I have a deep understanding of the system my team owns. There are projects we take on and decisions we make that do depend on that understanding. However, what's far more important 80% of the time are the wide-ranging shallow understandings I have of other systems that we interact with (sometimes directly, sometimes highly indirectly). I can't imagine being able to be successful in my role without both, but I see folks overindexing on the former pretty frequently.

lubujackson 1y ago

Giving yourselves breadcrumbs to knowledge is a fantastic life skill... but one that may (even more) fall out of favor with AI replacing the value. If I suddenly need to add a caching layer and have learned nothing about it I can ask AI to give me an overview and go ahead and recommend the best solution for my codebase and situation. Yes, you may miss some subtly and have to flail around to ask the right questions, but there is no doubt the value of "shallow technical knowledge" is less important. What will remain important or grow in importance is general curiosity. Connecting completely disparate ideas or ways of thinking will lead you to new creative thoughts or solutions that AI would never produce because everyone else is working from the same standard ideas. I was an English major in college and took classes in politics, philosophy, math, language, etc. based on personal interest. And I ended up as an engineer (with my trusty CS minor). TI've met several developers who have had a similar background and they tend to become the most well-rounded and business-aware ones on the team. I worry that this shift to higher cost/higher stakes/higher competition education is making this approach to learning feel untenable and my approach of 20 years ago comes across as totally irresponsible. But I would argue American education is leading to a factory approach at exactly the time when "structured thinking" is being fully replaced by AI. What is the value of crushing leetcode nowadays? Better to have a dev that has some intuition as to why people aren't clicking that new button.

o3b 1y ago

With the pace of technology progress and the amount of new developments in computer science, it's quite easy today to remain on shallow knowledge, which could still be ok, accepted and further help progress itself. At the end of the day, one of the key point of the progress is to make difficult things easier for a larger audience. This introduces abstraction layers, one on the other. Hence you may not know how the KV Cache works in Attention mechanism, but you might still be able to train a super useful AI model. However I think those unique engineers with vertical and deep knowledge in a tech stack (e.g. C, Java, Maths under NN) are still very needed in the world, because they are capable of building and repairing the fundamentals of everything which gets built upon. So, if you are interested a such fundamental stack, hack it, crash it, it won't be wasted time and world will need you :)

tibbar 1y ago

I think the distinction here is between "I understand that X is a solution to Y", vs. "I understand that X is BASICALLY a mashup of A and B according to scheme C." A lot of times, people reach for X when they have a vaguely Y-ish problem, when that's actually inefficient or overkill, and/or you could easily validate if X would actually work by manually trying out A and B by hand. In addition, there will be times when X cannot directly solve the version of Y you have, but there are simple ways to tweak A or B such that now you do have a solution to the problem. So you can become much, much more effective at solving Y-like problems by understand the the building blocks behind standard solutions.

blindriver 1y ago

The problem with shallow technical knowledge and, even worse, talking with confidence like the author does, is that it can propagate misinformation and loses a ton of nuance. For example, the author talks very confidently about indexes, and makes a few conclusions, but they aren't as correct as his confidence suggests. > That an index is only useful if it matches the actual query your application is making. If you index on “name plus email” and then start querying on “name plus address”, your index won’t be used and you’re back to the full table scan described in (1) Not true. If you have single column indexes on both name and email, it could use the two indexes, though not as efficient as a single two-column index. If you query "name plus email" and the index is "name, email, age" then it could use the index. > That indexes massively speed up reads, but must slow down database writes and updates, because each change must also be made again in the index “dictionary” Must? No. The performance might be imperceptible and not material at all. If you have a ton of indexes, sure but not if you have a reasonable amount. Shallow technical knowledge is fine but you should also have the humility to acknowledge that when you're dispensing said shallow knowledge. It can also lead to pretty bad engineering decisions if you think your shallow knowledge is enough.

sgarland 1y ago
For that matter, if you index `(name, email)` and query with `name` and `address` as the predicates, unless there's a better option, or the table is tiny, there's an excellent chance the planner will use that index to narrow down the initial result set to filter.
tibbar 1y ago
> Not true. If you have single column indexes on both name and email, it could use the two indexes, though not as efficient as a single two-column index. If you query "name plus email" and the index is "name, email, age" then it could use the index. See, there are databases that implement clever optimizations like this, but those are going to vary widely by database and you would need some domain expertise with that system to know if such optimizations are working. By contrast, this mental model does help you ensure that you can create indexes that are actually helpful in the vast majority of databases. So I think the author's mental model is working out pretty well for him here, honestly.

godelski 1y ago

I'm reminded of a Knuth quote[0] | If you find yourself doing only theory, do some practical work. It'll help with your theory. | If you find yourself doing only practical, do some theory. It'll help with your practical work. The two complement one another. Theory informs you of what is possible while the hands on informs you on what is feasible. They are different kinds of intuitions. Theory gets you thinking in high abstractions, understanding the building blocks and how things are put together. The practice forces you to deal with everything that slips through the cracks, teaching you how to do things efficiently and effectively. It's like making a building. You technically only need the construction skills to make a building. But the engineering designs make you able to build bigger and safer. The theory lets you find new novel designs and materials. You need all of this working together to build something great. I've always found theory and practice to go hand in hand. Maybe because I came to CS from physics, but they have hard divisions there. I focused on experimental physics but I was known for strong math skills which allowed me to bridge these two sides. But the major reason I did it is because I couldn't see how they were different. So when I later heard Knuth, it really resonated with me. But truthfully, I've struggled sometimes working with others, getting pulled in different directions. I left physics to become an engineer and all they saw me as was a theorist despite most of my work for them being practical (running simulations and then building physical devices and the necessary test infrastructure). Now I'm about to defend my PhD in CS (ML) and the biggest struggle I've had is that it feels like any time I spend on theory is seen as wasted. I do so much less than before, but it's been the most helpful tool to my work. So I'm hoping others can help me understand this culture. We definitely see theory differently but I'm not sure why or how. Everything seems hyper focused on making products. To ship as fast as possible. No time to think about what we're making. For me, that slows me down! Just because lines of code aren't appearing on screen doesn't mean I'm not hard at work. A few hours or even a day of thought has saved me weeks worth of work. I'm not the quickest to first result[1] but the result is I can do more with less and quickly adapt to all the changes that happen as the project evolves. New experiments, measurements, features and such get integrated quickly because I expect this to happen. But still, considered slow, and I can't find out why. So I'm wondering how others have dealt with this. I know I'm not alone here. I'm probably not going to change much because the end result works. But why is CS so focused on short term and how do I highlight longer term work when we're just measuring weekly updates. The strategy means more weeks with "not much to show" but other weeks that look like extreme productivity. But it's all the same. Because the truth is, the earlier work is just difficult or impossible to measure. I'm pretty sure this is why CS is the way it is, but I've got to say, I've got a lot of experience with measures and analysis and the fact is not everything is measurable. We only can use proxies and those easily become unaligned. [0] I didn't copy paste it so it might be a little off but the message is right [1] after proof of concept. Early on I try to be quick to fail. I'm talking about once that's going and we're doing more serious building.

4ndrewl 1y ago

Does the description of database indexes even count as _shallow_ knowledge? Sounds like a dictionary description with an example. Not even table stakes to warrant your inclusion in any conversation on the topic. I know that some birds migrate depending on the season and they fly in certain formations for efficiency. I'd never,ever think I could have any serious conversation with a biologist or ornithologist.

tibbar 1y ago
I think you underestimate how abysmal many product developers' understanding of databases is:-). It's less about making people respectable experts, more about teaching basic safety principles so they won't do too much damage...

chasil 1y ago

I have absolutely no desire to become an expert in OpenVMS, but I have an account with DCL access that can run the Oracle Rdb sql utility. There have been several times that my exposure has been wrapped into production applications. I know what I need to know on that platform, and try to maintain little more. Is this what everybody does?

sgarland 1y ago

> Database [anything] Honestly, the author couldn't have chosen a worse example if they tried. RDBMS are so absurdly complex and have so many edge cases for everything, that they are my canonical answer to why it's a bad idea to have dev teams managing them, or designing schema. Disclaimer: I'm a DBRE. Look at MySQL's ORDER BY [0], GROUP BY [1], and index merge [2] optimizations. Postgres' ORDER BY optimization [3] doesn't have nearly as many gotchas, but it more than makes up for it by having many more types of indices [4], as well as many types of operator classes within each of those. While you're at it, you should probably have a handle on collations [6], AKA "why did this ORDER BY do something odd?" > You could learn about the data structures that represent the index You mean a B+ tree (for standard types)? I would hope you covered that in DS&A already. > about all the tricks that go into updating them efficiently > But it will very rarely inform the database migration or query you’re writing. I disagree. For example, if you didn't know that updating any indexed column in a Postgres table means the entire row is re-written (hope you haven't done anything silly, like storing medium-sized JSON blobs that aren't quite big enough to trigger TOAST), you would likely be much more cavalier with your indexing decisions. I'm so very tired of the "I don't need to know how the things I use work" attitude. Do you need to be able to write your own database to effectively use one? No, of course not. Should you have read the official docs? Yes. You'll learn far more than from skimming random blogs and YouTube videos. [0]: https://dev.mysql.com/doc/refman/8.0/en/order-by-optimization.html https://dev.mysql.com/doc/refman/8.0/en/order-by-optimizatio... [1]: https://dev.mysql.com/doc/refman/8.0/en/group-by-optimization.html https://dev.mysql.com/doc/refman/8.0/en/group-by-optimizatio... [2]: https://dev.mysql.com/doc/refman/8.0/en/index-merge-optimization.html https://dev.mysql.com/doc/refman/8.0/en/index-merge-optimiza... [3]: https://www.postgresql.org/docs/current/indexes-ordering.html https://www.postgresql.org/docs/current/indexes-ordering.htm... [4]: https://www.postgresql.org/docs/current/indexes-types.html https://www.postgresql.org/docs/current/indexes-types.html [5]: https://www.postgresql.org/docs/current/indexes-opclass.html https://www.postgresql.org/docs/current/indexes-opclass.html [6]: https://www.postgresql.org/docs/current/collation.html https://www.postgresql.org/docs/current/collation.html

exiguus 1y ago

If you are developing (technical) solutions, i would say that shallow knowledge is essential. As long as it allows you to make decisions.