4 ms·

It’s because modern devs by and large have zero concept of latency differences. I had to explain to people yesterday why an on-disk temporary table in MySQL was

It’s because modern devs by and large have zero concept of latency differences. I had to explain to people yesterday why an on-disk temporary table in MySQL was slower than an in-memory one. “But the disk is an SSD,” was an actual rebuttal I got. Never mind the fact that this was Aurora, so the “disk” is a slice of (multiple) SSDs exposed over a SAN…

jonhohle 2y ago
https://gist.github.com/jboner/2841832 https://gist.github.com/jboner/2841832
- hedora 2y ago
  That really should include "sync write 4KB to an SSD". People don't realize it's purely in-RAM on the disk these days (you pay a PCIe round trip, and that's about it).
101008 2y ago
That is what happens when your dev learnt it from an online course instead of going to college or through proper education. "Anyone can code!"
- dns_snek 2y ago
  And I thought we were past this sort of gatekeeping and elitism. I've worked with people who had a master's degree in CS who couldn't code their way out of a wet paper bag. In my experience there's very little correlation between how someone learned programming and how deep their understanding of the entire stack is.
  - bsrkf 2y ago
    Hope this doesn't come off as disrespectful, as in that I don't believe you, but out of personal interest, would you consider expanding on that? I'd love to hear about the particular example you were thinking of, or in what ways self-taught coders surprised you over academically-taught ones, if you've had experience working with both over a meaningful span of time. Also, if the case, in what ways self-taught coders were/are maybe lacking on average. If you've ever given answers to that in another comments on HN or elsewhere, feel free to link.
    - MattPalmer1086 2y ago
      It's certainly true in my experience. The main thing that makes a difference is simply how curious and interested you are. Plenty of graduates simply got into to it to make money, and have no real deep interest. Some of them love the theory but hate the practice. And some of them are good at both of course. By contrast, self taught people tend to have personal interest going for them. But I've also worked with self taught people who had no real understanding (despite their interest), and who were satisfied if something just worked. Even if they are motivated to know more, they are often lacking in deeper theoretical computer science (this is a gap I've had to gradually fill in myself). Anyway, the determining factor is rarely exactly how they acquired their skills, it's the approach they take to the subject and personal motivation and curiosity.
      - bsrkf 2y ago
        Makes sense, out of all the potential differentiators the source of skill attainment simply isn't the necessarily dominant one. Thanks for the answer :)
    - dns_snek 2y ago
      Not disrespectful at all. I agree with the sibling comments, I think what allows someone to become a great software developer, to have great intuition, and to understand systems and abstractions on a really deep level is their curiosity, hunger for knowledge, and a lot of experience. There are many college educated software developers who have that sort of drive (or passion, if you will) and there are just as many who don't, it's not something college can teach you, and the same is true for self-taught developers. At the end of the day "self-taught" is also a spectrum that ranges from people who created their first "hello world" React app 2 months ago to people who have been involved in systems programming since they were 10 years old, and have a wealth of knowledge in multiple related fields like web development, systems administration, and networking. That's why I think it's silly to generalize like that. Software development is extremely broad so depending on the discipline self-taught developers might not be missing anything essential, or they might have to learn algorithms, discrete mathematics, linear algebra, or calculus on their own. I learned all of that in college but I'd probably have to learn most of it again if I really needed it.
  - dbalatero 2y ago
    Yeah, deep understanding I think is a matter of how much time you spend investigating the craft and improving. Maybe the real question is: what motivates that to happen? Maybe it's school type, but maybe it's not. My personal anecdotes, which are music centric but all apply to my software career: 1. I've studied music my whole life, and baked into music is the idea of continual practice & improvement. Because of this experiential input, I believe that I can always improve at things if I actively put a bit of energy into it and show up. I believe it because I've put in so many hours to it and have seen the results. This is deeply ingrained. 2. When I picked up bass in high school, I spent the first year learning tabs in my bedroom. It was ok, but my ability accelerated when I started a band with my friends and had to keep up. I really think the people you surround yourself with can: push you more, teach you things you didn't know, and make the process way more of a fun hang than doing it by yourself. 3. Another outcome from music education was learning that I really love how mastery feels. There's a lot of positive feeling from achieving and growing. As a result, I try to seek it out in other places as well. I imagine sports/etc are the same?
  - aleph_minus_one 2y ago
    > I've worked with people who had a master's degree in CS who couldn't code their way out of a wet paper bag. "Programming" consists of an insanely large number of isolated communities. Assuming the respective person is capable, I would assume that he simply comes from a very different "programming culture". I actually observe a very related phenomenon for myself: the more I learn about some very complicated programming topics, the more "alien" I actually (start to) become to the programming topics that I have to do at work.
  - sgarland 2y ago
    I’ve worked with both (I personally have an unrelated BS and an MS in SWE, which I used purely to get my foot in the door – it worked), and IMO if someone has a BS, not MS, there’s a decent chance they at least understand DS&A, probably took an OS course, etc. That said, I have also worked with brilliant people who had no formal education in the subject whatsoever, they just really, really liked it. I’m biased towards ops because that’s what I do and like, but at least in that field, the single biggest green flag I’ve found is whether or not someone has a homelab. People can cry all they want about “penalizing people for having hobbies outside of their job,” but it’s pretty obvious that if you spend more time doing something – even moreso if you enjoy it – you will learn it at a much faster rate than someone who only does it during business hours.
- jeltz 2y ago
  Maybe you should skip bashing modern devs and people who learnt from online courses when the parent poster is the one in the wrong. Unless InnoDB has a particularly bad implementation of disk backed temporary tables almodt all disk IO should be done in the background so the latency cost should be small. I recommend benchmarking on you particular workload of you really want to know hur big it is. I am an old-school developer with a computer engineering degree but many of the old famous devs were self taught. Yes, if you learn how to code through online courses you will miss some important fundamentals but those are possible to learn later and I know several people who have.
  - sgarland 2y ago
    Please see this comment: https://news.ycombinator.com/item?id=43863885 https://news.ycombinator.com/item?id=43863885 We have excellent metrics between PMM and AWS Perf. Insights / Enhanced Monitoring. I assure you, on-disk temp tables were to blame. To your point, though, MySQL 5.7 does have a sub-optimal implementation in that it kicks a lot of queries out to disk from memory because of the existence of a TEXT column, which internally treated as a BLOB. Schema design is also partially to blame, since most of the tables were unnecessarily denormalized, and TEXT was often used where VARCHAR would have been more appropriate, but still.
- StefanBatory 2y ago
  I wish. I did my undergrad, and now doing my Masters, at a top 10 uni in Poland. Trust me so much, some of the stuff I learned there was actively harmful. Half of subjects were random fillers, and so on. I envy Americans so much with that, their CS education seems to be top notch.
athenot 2y ago
It's also a Product issue: "fast and snappy" is almost never on their list of priorities. So product managers will push devs for more features, which satisfy the roadmap, and only get concerned about speed when it reaches painful levels.
- jMyles 2y ago
  > only get concerned about speed when it reaches painful levels. ...and by then, the requests for performance are somewhere between onerous and ridiculous. I'm as wary of premature optimization as anyone, but I also have a healthy fear of future-proofed sluggishness.
osigurdson 2y ago
Agree. I've heard people say things like "it is slow because it is not C++". When, in reality, the problem was I/O and N^2 algos.
- zahlman 2y ago
  I've been trying to impress upon people from my own research in Python packaging: Pip is slow because it defaults to pre-compiling bytecode (and doesn't invoke multiprocessing for that, although this seems to be in the works from the discussion I've seen); imports literally hundreds of modules even when it ultimately does nothing; creates complex wrappers to set up for connecting to the Internet (and using SSH, of course) even if you tell it to install from a locally downloaded wheel directly; caches things in a way that simulates network sessions instead of just actually having the files... you get the idea. "It's written in Python and not e.g. in Rust" is simply not relevant in that context. (For that matter, when uv is asked to pre-compile, while it does some intelligent setup for multiprocessing, it still ultimately invokes the same bytecode compiler - which is part of the Python implementation itself, written in C unless you're using an alternative implementation like PyPy.)
  - osigurdson 2y ago
    There may be a bit of culture at play sometimes as well. If a language isn't meant to be fast, then perhaps devs using the language do not prioritize performance very much. For some, as long as it is possible to point to some externality ("hey this is Python, what do you expect") this is sufficient. Of course, not always the case. C++ is a good counter example with a massive range of performance "orientedness". On the other hand, I suspect there are few Rust / Zig or C programmers that don't care about performance.
    - zahlman 2y ago
      To me, Python is more of a challenge than an excuse when it comes to performance. On the flip side, I've seen quite a few C developers using their own hand-rolled linked lists where vector-like storage would be more appropriate, without giving it a second thought. Implementing good hash tables from scratch turns out not to be very much fun, either. I'm sure there are off the shelf solutions for that sort of thing, but `#include` and static compilation in C don't exactly encourage module reuse the same way that modern languages with package managers do (even considering all the unique issues with Python package management). For better or worse. (For what it's worth, I worked in J2ME for phones in the North American market in the mid 00s, if you remember what those were like.)
jeltz 2y ago
But is it actually faster by any significant amount on your workload? Did you benchmark? Temporary tables in databases rarely actually do disk IO in any blocking code path but mostly just dirty buffers in the OS or in the database itself and then something writes it to disk in the background. This limits throughput but does not add much extra latency in the common cases. Edit: It still might be a bad idea to waste the IO if you do not have to but the latency of a temporary table is usually RAM latency, not disk latency even for temporary tables on disk.
- sgarland 2y ago
  Didn’t have to, prod benchmarked it for us, twice. If you’re curious, the EBS disks Aurora uses for temporary storage, when faced with a QD of approximately 240, can manage approximately 5000 IOPS. This was an r6i.32xlarge. My hypothesis is currently that the massive context switching the CPU had to do to handle the interrupts slowed down its acceptance of new connections / processing other queries enough such that everything piled up. I’ve no idea what kind of core pinning / isolation AWS does under the hood, but CPU utilization from disk I/O alone, according to Enhanced Monitoring, was about 20%.
sgarland 2y ago
I do want to add that many of the devs I work with and have worked with are genuinely interested in learning this stuff, and I am thrilled by it. What bothers me is when they neither have an interest in learning it, nor in fixing their bad patterns, and leadership shrugs and tells infra teams to make it work.

jonhohle 2y ago

https://gist.github.com/jboner/2841832 https://gist.github.com/jboner/2841832

hedora 2y ago
That really should include "sync write 4KB to an SSD". People don't realize it's purely in-RAM on the disk these days (you pay a PCIe round trip, and that's about it).

101008 2y ago

That is what happens when your dev learnt it from an online course instead of going to college or through proper education. "Anyone can code!"

dns_snek 2y ago
And I thought we were past this sort of gatekeeping and elitism. I've worked with people who had a master's degree in CS who couldn't code their way out of a wet paper bag. In my experience there's very little correlation between how someone learned programming and how deep their understanding of the entire stack is.
- bsrkf 2y ago
  Hope this doesn't come off as disrespectful, as in that I don't believe you, but out of personal interest, would you consider expanding on that? I'd love to hear about the particular example you were thinking of, or in what ways self-taught coders surprised you over academically-taught ones, if you've had experience working with both over a meaningful span of time. Also, if the case, in what ways self-taught coders were/are maybe lacking on average. If you've ever given answers to that in another comments on HN or elsewhere, feel free to link.
  - MattPalmer1086 2y ago
    It's certainly true in my experience. The main thing that makes a difference is simply how curious and interested you are. Plenty of graduates simply got into to it to make money, and have no real deep interest. Some of them love the theory but hate the practice. And some of them are good at both of course. By contrast, self taught people tend to have personal interest going for them. But I've also worked with self taught people who had no real understanding (despite their interest), and who were satisfied if something just worked. Even if they are motivated to know more, they are often lacking in deeper theoretical computer science (this is a gap I've had to gradually fill in myself). Anyway, the determining factor is rarely exactly how they acquired their skills, it's the approach they take to the subject and personal motivation and curiosity.
    - bsrkf 2y ago
      Makes sense, out of all the potential differentiators the source of skill attainment simply isn't the necessarily dominant one. Thanks for the answer :)
  - dns_snek 2y ago
    Not disrespectful at all. I agree with the sibling comments, I think what allows someone to become a great software developer, to have great intuition, and to understand systems and abstractions on a really deep level is their curiosity, hunger for knowledge, and a lot of experience. There are many college educated software developers who have that sort of drive (or passion, if you will) and there are just as many who don't, it's not something college can teach you, and the same is true for self-taught developers. At the end of the day "self-taught" is also a spectrum that ranges from people who created their first "hello world" React app 2 months ago to people who have been involved in systems programming since they were 10 years old, and have a wealth of knowledge in multiple related fields like web development, systems administration, and networking. That's why I think it's silly to generalize like that. Software development is extremely broad so depending on the discipline self-taught developers might not be missing anything essential, or they might have to learn algorithms, discrete mathematics, linear algebra, or calculus on their own. I learned all of that in college but I'd probably have to learn most of it again if I really needed it.
- dbalatero 2y ago
  Yeah, deep understanding I think is a matter of how much time you spend investigating the craft and improving. Maybe the real question is: what motivates that to happen? Maybe it's school type, but maybe it's not. My personal anecdotes, which are music centric but all apply to my software career: 1. I've studied music my whole life, and baked into music is the idea of continual practice & improvement. Because of this experiential input, I believe that I can always improve at things if I actively put a bit of energy into it and show up. I believe it because I've put in so many hours to it and have seen the results. This is deeply ingrained. 2. When I picked up bass in high school, I spent the first year learning tabs in my bedroom. It was ok, but my ability accelerated when I started a band with my friends and had to keep up. I really think the people you surround yourself with can: push you more, teach you things you didn't know, and make the process way more of a fun hang than doing it by yourself. 3. Another outcome from music education was learning that I really love how mastery feels. There's a lot of positive feeling from achieving and growing. As a result, I try to seek it out in other places as well. I imagine sports/etc are the same?
- aleph_minus_one 2y ago
  > I've worked with people who had a master's degree in CS who couldn't code their way out of a wet paper bag. "Programming" consists of an insanely large number of isolated communities. Assuming the respective person is capable, I would assume that he simply comes from a very different "programming culture". I actually observe a very related phenomenon for myself: the more I learn about some very complicated programming topics, the more "alien" I actually (start to) become to the programming topics that I have to do at work.
- sgarland 2y ago
  I’ve worked with both (I personally have an unrelated BS and an MS in SWE, which I used purely to get my foot in the door – it worked), and IMO if someone has a BS, not MS, there’s a decent chance they at least understand DS&A, probably took an OS course, etc. That said, I have also worked with brilliant people who had no formal education in the subject whatsoever, they just really, really liked it. I’m biased towards ops because that’s what I do and like, but at least in that field, the single biggest green flag I’ve found is whether or not someone has a homelab. People can cry all they want about “penalizing people for having hobbies outside of their job,” but it’s pretty obvious that if you spend more time doing something – even moreso if you enjoy it – you will learn it at a much faster rate than someone who only does it during business hours.
jeltz 2y ago
Maybe you should skip bashing modern devs and people who learnt from online courses when the parent poster is the one in the wrong. Unless InnoDB has a particularly bad implementation of disk backed temporary tables almodt all disk IO should be done in the background so the latency cost should be small. I recommend benchmarking on you particular workload of you really want to know hur big it is. I am an old-school developer with a computer engineering degree but many of the old famous devs were self taught. Yes, if you learn how to code through online courses you will miss some important fundamentals but those are possible to learn later and I know several people who have.
- sgarland 2y ago
  Please see this comment: https://news.ycombinator.com/item?id=43863885 https://news.ycombinator.com/item?id=43863885 We have excellent metrics between PMM and AWS Perf. Insights / Enhanced Monitoring. I assure you, on-disk temp tables were to blame. To your point, though, MySQL 5.7 does have a sub-optimal implementation in that it kicks a lot of queries out to disk from memory because of the existence of a TEXT column, which internally treated as a BLOB. Schema design is also partially to blame, since most of the tables were unnecessarily denormalized, and TEXT was often used where VARCHAR would have been more appropriate, but still.
StefanBatory 2y ago
I wish. I did my undergrad, and now doing my Masters, at a top 10 uni in Poland. Trust me so much, some of the stuff I learned there was actively harmful. Half of subjects were random fillers, and so on. I envy Americans so much with that, their CS education seems to be top notch.

athenot 2y ago

It's also a Product issue: "fast and snappy" is almost never on their list of priorities. So product managers will push devs for more features, which satisfy the roadmap, and only get concerned about speed when it reaches painful levels.

jMyles 2y ago
> only get concerned about speed when it reaches painful levels. ...and by then, the requests for performance are somewhere between onerous and ridiculous. I'm as wary of premature optimization as anyone, but I also have a healthy fear of future-proofed sluggishness.

osigurdson 2y ago

Agree. I've heard people say things like "it is slow because it is not C++". When, in reality, the problem was I/O and N^2 algos.

zahlman 2y ago
I've been trying to impress upon people from my own research in Python packaging: Pip is slow because it defaults to pre-compiling bytecode (and doesn't invoke multiprocessing for that, although this seems to be in the works from the discussion I've seen); imports literally hundreds of modules even when it ultimately does nothing; creates complex wrappers to set up for connecting to the Internet (and using SSH, of course) even if you tell it to install from a locally downloaded wheel directly; caches things in a way that simulates network sessions instead of just actually having the files... you get the idea. "It's written in Python and not e.g. in Rust" is simply not relevant in that context. (For that matter, when uv is asked to pre-compile, while it does some intelligent setup for multiprocessing, it still ultimately invokes the same bytecode compiler - which is part of the Python implementation itself, written in C unless you're using an alternative implementation like PyPy.)
- osigurdson 2y ago
  There may be a bit of culture at play sometimes as well. If a language isn't meant to be fast, then perhaps devs using the language do not prioritize performance very much. For some, as long as it is possible to point to some externality ("hey this is Python, what do you expect") this is sufficient. Of course, not always the case. C++ is a good counter example with a massive range of performance "orientedness". On the other hand, I suspect there are few Rust / Zig or C programmers that don't care about performance.
  - zahlman 2y ago
    To me, Python is more of a challenge than an excuse when it comes to performance. On the flip side, I've seen quite a few C developers using their own hand-rolled linked lists where vector-like storage would be more appropriate, without giving it a second thought. Implementing good hash tables from scratch turns out not to be very much fun, either. I'm sure there are off the shelf solutions for that sort of thing, but `#include` and static compilation in C don't exactly encourage module reuse the same way that modern languages with package managers do (even considering all the unique issues with Python package management). For better or worse. (For what it's worth, I worked in J2ME for phones in the North American market in the mid 00s, if you remember what those were like.)

jeltz 2y ago

But is it actually faster by any significant amount on your workload? Did you benchmark? Temporary tables in databases rarely actually do disk IO in any blocking code path but mostly just dirty buffers in the OS or in the database itself and then something writes it to disk in the background. This limits throughput but does not add much extra latency in the common cases. Edit: It still might be a bad idea to waste the IO if you do not have to but the latency of a temporary table is usually RAM latency, not disk latency even for temporary tables on disk.

sgarland 2y ago
Didn’t have to, prod benchmarked it for us, twice. If you’re curious, the EBS disks Aurora uses for temporary storage, when faced with a QD of approximately 240, can manage approximately 5000 IOPS. This was an r6i.32xlarge. My hypothesis is currently that the massive context switching the CPU had to do to handle the interrupts slowed down its acceptance of new connections / processing other queries enough such that everything piled up. I’ve no idea what kind of core pinning / isolation AWS does under the hood, but CPU utilization from disk I/O alone, according to Enhanced Monitoring, was about 20%.

sgarland 2y ago

I do want to add that many of the devs I work with and have worked with are genuinely interested in learning this stuff, and I am thrilled by it. What bothers me is when they neither have an interest in learning it, nor in fixing their bad patterns, and leadership shrugs and tells infra teams to make it work.