6 ms·

It seems like that to me as well, but what is an engineer to do if he feels confused about what he's building on top of and he gets told that he's just feeling

It seems like that to me as well, but what is an engineer to do if he feels confused about what he's building on top of and he gets told that he's just feeling impostor syndrome or needs to stop "trying to understand the universe" or that his boss trusts that he does actually understand things? It has been true my whole career and I can't wait for the moment when I can retire and then have the time to learn enough to feel like I know what I'm doing.

Delk 6y ago
The "universe" is also, in some sense, much more complex than a couple of decades ago. In most cases we don't need to deal with as much low-level complexity or with cumbersome tooling, but "understanding the universe" would mean understanding pretty much all the levels of abstraction we have (and we have a lot), a diverse stack of technologies with rather different paradigms, a security landscape that has become much more complex with all the code we're running in a browser, and so on. I don't know what one is to do, but I somehow feel that "understanding the universe" is much more difficult nowadays than it perhaps was a few decades ago. There were probably fewer levels of abstraction, less diversity of technology, UIs might have been programmed in the same language as the rest of the program so there might not have been a need to juggle in a few languages at the same time, etc. Not to mention that since it's nowadays possible to program reasonably without having all that lower-level understanding, unlike perhaps a few decades ago, lots of people do that and (immediate) productivity expectations naturally grow to match. I feel the pain, though.
- alisonatwork 6y ago
  I agree that there were less levels of abstraction, but I think the domain in which each individual developer operates isn't that much different. In my first "serious" programming job, I was hired to be a C programmer. But just to get C building, you also need to understand Makefiles, and to understand those you need to understand shell script. We printed stuff to console in color, so we needed to understand ANSI escape sequences. We also printed to printers, so we needed to understand HP PCL. Some of our systems needed to talk to dedicated hardware attached on the serial port, so we had to learn proprietary protocols. Newer devices were attached via TCP, so we had to learn sockets and those protocols too. We had our own home-built database, but sometimes we needed SQL. We used Informix, later Postgres. Each had a different stored procedure language and C bindings. We deployed on UNIX, so we needed to understand files and pipes and processes and semaphores. We used a modem to dial our our customer sites, so needed to know about a tool called Kermit. There were many different flavors of UNIX back then, and they all had slightly different behaviors. We had to learn all the quirks that applied to the APIs we were using. Later on we started to build front ends in Java AWT and Python GTK, and that brought in a slew of new tech. Much of this tech was still pre-2000. In my current job, I am mostly working in Java and Python, but now they are backend languages. The services are built using Gradle, which needs knowledge of Groovy, or setuptools which is just more Python. The packages are bundled in containers and pushed to the cloud. There is a lot of engineering that enables this stack, but the only part we need to know about is the format of a Dockerfile (so, basic shell scripting) and various YAMLs for CloudFormation and the CI/CD system. We talk to lots of other services, but we don't need to know anything about TCP or sockets, we just need to use a library to manage the HTTP connections and another library to convert the wire format to objects. We don't care about presentation logic or reporting because someone elsewhere in the stack is going to do that. We define our contracts in Protobuf, JSON or XML and move on. Some services need to keep state, and for almost all use cases the file system is sufficient - in AWS that means S3. Sometimes we send or receive messages - in AWS that means SNS/SQS. We also push metrics and logging and distributed trace context, but that's just more HTTP abstracted away into a library. We build dashboards in Grafana using OpenTSDB query language. We check our logs in Kibana using Lucene query language. There is a lot to learn. I could spend even more time learning about Kubernetes or Kafka or Go or Redis or a hundred other components required by software outside my current domain, but none of that is necessary in my current role. Back in the day I could have spent time learning about OS/390 or Syslog or Perl or Berkeley DB too. So, although I agree that "the universe" in general is much more complex, I haven't found that to be true about the part individual developers are expected to understand.
  - Delk 6y ago
    > So, although I agree that "the universe" in general is much more complex, I haven't found that to be true about the part individual developers are expected to understand. That's probably true. I think the person I was responding to felt like they would want to understand more than they're necessarily expected by others to, though. I kind of share that sentiment, and they also mentioned impostor syndrome, so I wanted to express my feeling that if you want to understand "the universe" (and not just what's necessary in order to fulfill the what's expected of you), that's probably more complex nowadays than it maybe was back when the field was even younger. You have a longer perspective on that development than I do, though (I started in the early 2000's or so), so thanks for sharing.
alisonatwork 6y ago
At a certain point you need to trust that the library you are calling or the framework you are developing inside is doing what it is supposed to do. Going deeper than that can be interesting as a learning experience, and occasionally it's required to figure out really difficult bugs or performance issues, but it's not a prerequisite for being able to understand your own work. Identify the edges of your domain, accept them and treat them as the rules of the game. That said, by the sounds of it you have worked with some colleagues who have been unhelpful in teaching you the rules. This might be because you are building on a poorly-architected thing that cannot easily be explained, or in a system that does not have detailed documentation. Both of those situations are solvable problems, and neither of them is an excuse for the people who do understand to gatekeep their knowledge or discourage curiosity.

Delk 6y ago

The "universe" is also, in some sense, much more complex than a couple of decades ago. In most cases we don't need to deal with as much low-level complexity or with cumbersome tooling, but "understanding the universe" would mean understanding pretty much all the levels of abstraction we have (and we have a lot), a diverse stack of technologies with rather different paradigms, a security landscape that has become much more complex with all the code we're running in a browser, and so on. I don't know what one is to do, but I somehow feel that "understanding the universe" is much more difficult nowadays than it perhaps was a few decades ago. There were probably fewer levels of abstraction, less diversity of technology, UIs might have been programmed in the same language as the rest of the program so there might not have been a need to juggle in a few languages at the same time, etc. Not to mention that since it's nowadays possible to program reasonably without having all that lower-level understanding, unlike perhaps a few decades ago, lots of people do that and (immediate) productivity expectations naturally grow to match. I feel the pain, though.

alisonatwork 6y ago
I agree that there were less levels of abstraction, but I think the domain in which each individual developer operates isn't that much different. In my first "serious" programming job, I was hired to be a C programmer. But just to get C building, you also need to understand Makefiles, and to understand those you need to understand shell script. We printed stuff to console in color, so we needed to understand ANSI escape sequences. We also printed to printers, so we needed to understand HP PCL. Some of our systems needed to talk to dedicated hardware attached on the serial port, so we had to learn proprietary protocols. Newer devices were attached via TCP, so we had to learn sockets and those protocols too. We had our own home-built database, but sometimes we needed SQL. We used Informix, later Postgres. Each had a different stored procedure language and C bindings. We deployed on UNIX, so we needed to understand files and pipes and processes and semaphores. We used a modem to dial our our customer sites, so needed to know about a tool called Kermit. There were many different flavors of UNIX back then, and they all had slightly different behaviors. We had to learn all the quirks that applied to the APIs we were using. Later on we started to build front ends in Java AWT and Python GTK, and that brought in a slew of new tech. Much of this tech was still pre-2000. In my current job, I am mostly working in Java and Python, but now they are backend languages. The services are built using Gradle, which needs knowledge of Groovy, or setuptools which is just more Python. The packages are bundled in containers and pushed to the cloud. There is a lot of engineering that enables this stack, but the only part we need to know about is the format of a Dockerfile (so, basic shell scripting) and various YAMLs for CloudFormation and the CI/CD system. We talk to lots of other services, but we don't need to know anything about TCP or sockets, we just need to use a library to manage the HTTP connections and another library to convert the wire format to objects. We don't care about presentation logic or reporting because someone elsewhere in the stack is going to do that. We define our contracts in Protobuf, JSON or XML and move on. Some services need to keep state, and for almost all use cases the file system is sufficient - in AWS that means S3. Sometimes we send or receive messages - in AWS that means SNS/SQS. We also push metrics and logging and distributed trace context, but that's just more HTTP abstracted away into a library. We build dashboards in Grafana using OpenTSDB query language. We check our logs in Kibana using Lucene query language. There is a lot to learn. I could spend even more time learning about Kubernetes or Kafka or Go or Redis or a hundred other components required by software outside my current domain, but none of that is necessary in my current role. Back in the day I could have spent time learning about OS/390 or Syslog or Perl or Berkeley DB too. So, although I agree that "the universe" in general is much more complex, I haven't found that to be true about the part individual developers are expected to understand.
- Delk 6y ago
  > So, although I agree that "the universe" in general is much more complex, I haven't found that to be true about the part individual developers are expected to understand. That's probably true. I think the person I was responding to felt like they would want to understand more than they're necessarily expected by others to, though. I kind of share that sentiment, and they also mentioned impostor syndrome, so I wanted to express my feeling that if you want to understand "the universe" (and not just what's necessary in order to fulfill the what's expected of you), that's probably more complex nowadays than it maybe was back when the field was even younger. You have a longer perspective on that development than I do, though (I started in the early 2000's or so), so thanks for sharing.

alisonatwork 6y ago

At a certain point you need to trust that the library you are calling or the framework you are developing inside is doing what it is supposed to do. Going deeper than that can be interesting as a learning experience, and occasionally it's required to figure out really difficult bugs or performance issues, but it's not a prerequisite for being able to understand your own work. Identify the edges of your domain, accept them and treat them as the rules of the game. That said, by the sounds of it you have worked with some colleagues who have been unhelpful in teaching you the rules. This might be because you are building on a poorly-architected thing that cannot easily be explained, or in a system that does not have detailed documentation. Both of those situations are solvable problems, and neither of them is an excuse for the people who do understand to gatekeep their knowledge or discourage curiosity.