4 ms·

Ask HN: Want to keep my software engineering skills sharp

19 points by mettamage 2y ago

I currently started work as a data analyst. For a data analyst, the work is quite programming heavy actually (e.g. we don't use Excel but Jupyter). But for a software engineer, the amount of programming feels low. It's more like creating quick scripts. For now, I'm having a blast but I want to keep my actual software engineering skills sharp where I also learn some stuff on a part-time basis. Preferably stuff that I can also put on a resume that will be seen as experience. What would you suggest I should do? I was thinking about joining an open-source project of a big enough scope or something. Perhaps I should insist on fun software, which would be certain web apps or game development in my case.

cpach 2y ago
If you’re interested in security-related self-studies, I can warmly recommend Cryptopals: https://cryptopals.com/ https://cryptopals.com/
- mettamage 2y ago
  That brings up a good point! I could also go the hackthebox.eu route to train my security engineering skills, including cryptopals. PlayerTwo was still a fun insane box :)
rednafi 2y ago
Could you elaborate on this a bit? > But for a software engineer, the amount of programming feels low. It's more like creating quick scripts. I feel like it’s the other way around. When I worked as a data analyst (before transitioning to a data scientist and eventually a full-fledged SWE), my experience was quite different. As a DA, I mostly wrote aggregation queries, built dashboards, and did some Python, Pandas, and Notebook munging. On the other hand, SWE is a different world. I guess it depends on the type of SWE you are and the nature of the organization you work for. At a large-scale organization, as an SWE, I did far more programming work than I ever did as a DA. That said, there are different ways to cope and build skills—it takes time. I do side projects and, more importantly, write about them regularly [1]. Since I have to publish something each week, it forces me to think about different problems and seek out new technologies to try out so I can write about them. I also write about things I’ve figured out at work, which has been immensely helpful. Over the years, this act of actively seeking knowledge has added up and become substantial. More than once, people have reached out with job offers just because they stumbled across my writing on Hacker News or other social media platforms. Your approach might be different. Find something tangential to programming and work on it outside of work hours. Even a few hours per week adds up. For me, it’s side projects and writing. For you, it might be something else. [1]: https://rednafi.com https://rednafi.com
- mettamage 2y ago
  I think I explained myself poorly. > At a large-scale organization, as an SWE, I did far more programming work than I ever did as a DA. Yea, this is what I meant. The amount of programming I do now as a DA feels low (since I'm a software engineer historically, that's what I meant with "but for a software engineer" - a bit of a confusing sentence). Well specifically, the amount of engineering feels low. > That said, there are different ways to cope and build skills—it takes time. I do side projects and, more importantly, write about them regularly [1]. Good point, to what extent do employers see this as experience though? It's my experience they don't care at all. Maybe I'm looking at this wrong. The reason I'd do it is (1) I want to keep on building and (2) I want to keep on building experience as a SWE that employers will be impressed by. I'm not sure if I want to switch back, but for now I want to keep that option open. > More than once, people have reached out with job offers just because they stumbled across my writing on Hacker News or other social media platforms. I guess you answered your question here. I think it's good enough to test it out :) > As a DA, I mostly wrote aggregation queries, built dashboards, and did some Python, Pandas, and Notebook munging. Yea, I see a lot of SQL queries here, notebook stuff (so also Python/Pandas) and dashboarding. It's probably a 50/50 split. I have a suspicion that my manager wants me more on the programming side of things though. So far me as well. I'm allowed to pick any tool I find right for the job. I already wrote a bash script today.
  - rednafi 2y ago
    > Good point. To what extent do employers even see this as experience, though? In my experience, they don’t care at all. Maybe I’m looking at this wrong. I don’t know to what extent this is considered experience either. That said, the hiring process in tech is completely broken. There are companies that ask LeetCode hard questions for positions where you’ll just write HTML templates and React. I work on side projects and write because I enjoy the process—not because I’m fishing for potential employers. In the past half-decade, I’ve written close to 200 articles. Researching for them was immensely helpful in building knowledge. At the start, I didn’t care if anyone read them. But eventually, the site got traction, and now it forces me to try new things and keep my skillset sharp. This works as a positive feedback loop.
petargyurov 2y ago
If you write a lot of "quick scripts", you might be writing a lot of the same boilerplate code over and over again. If that's the case, creating a framework/library that does X/Y/Z (related to your job) might help.
- mettamage 2y ago
  Yea that's fair, just started, so we'll see. Thanks for the tip though!
boricj 2y ago
If you're looking for suggestions outside of work, one important factor will be motivation. You're not paid for it so you'll need something that will hold your attention regardless, or you won't commit to it. Therefore, the answer depends on what you personally find interesting.
- mettamage 2y ago
  Yea I thought about that, probably game dev from the ground (with my own graphics engine) up or a web app that I find useful (currently no thoughts on that).
dagw 2y ago
As someone in a similar situation, my number one piece of advice is to meticulously keep up your software engineering best practices even when doing your day to day data analysis job. All that bread-and-butter stuff like tests and easy deployment and well structured code and reproducible builds and writing reusable tools and packages tends to fall by the wayside when your 'just' hacking one off Jupyter notebooks. As a software engineer you bring a perspective to the job that many of your colleagues may lack. Lean on that and use that background to help build better tools and ways of working. This not only makes your job (and the job of your colleagues) easier in the long run it will also let you keep your software engineering skills sharp.
- aristofun 2y ago
  > even when doing your day to day data analysis job This is a bad advice. Usually coming from an over-engineering mentality (that many engineers suffer from, including myself). Part of the engineering culture is to find the _right_ solution for the job. Not the most _engineered_ one. There is no point to shoot at sparrows from a canon.
  - buffington 2y ago
    How is this bad advice? > keep up your software engineering best practices It's not a best practice to over-engineer solutions, and indeed, it is a best practice to find the right solution.
    - aristofun 2y ago
      When your job is to cobble up disposable scripts - there is no place for “practices”, unit test for example. Thus there is nothing for a software engineer to pay special attention to. Therefore there is no value in suggesting that While some may take that as “turn up your engineering knob” that leads to over engineering
      - mettamage 2y ago
        While I am fairly good at overengineering and optimizing, I have always hated doing it. For the longest time I wasn't able to do it but then I learned about low level hacking, and throwing things into assembly. Then I had a friend enter the HFT field and he taught me a thing or two about optimization. godbolt.org is fun to check out :) But yea, I really like making quick scrappy prototypes that deliver business results quickly. I've always had that mindset rather than crafting good quality code. In software engineering teams I threw that mindset to the wayside because they'd get mad. Sometimes rightly so as tests have saved my life, but quite often I think SWEs focus too much on code and being technical rather than delivering impact. I think in part that's also because it seems that in most places I've worked at product owners are "supposed to" think about that. The dev is only supposed to think about the technical implications, maybe the UX implications (depending on the company) but the moment a dev thinks about business implications, I've gotten quite a few times to "stay in my lane". In most cases the care for code quality isn't needed, sometimes it is. I'm taking a page from Jonathan Blow here (the game dev that created The Witness and Braid). He'd develop everything unoptimized and only later when he'd run into performance issues would he optimize it.
scarface_74 2y ago
I think one of the most toxic things in this industry is that we think we need to continuously be doing coding outside of work instead of living life. So the answer is I don’t. I had been programming continuously from the time I was in 6th grade in 1986 until 2020. As I pivoted into more strategic cloud consulting specializing in cloud architecture + application development in 2020 - “cloud application modernization” - I started coding less and less to the point now that almost any time that I even suggest writing code at work even though my title is “staff software architect”, I get my hand slapped and asked why didn’t I delegate it and focus on “higher impact work”. I keep my knowledge of the industry up to date by continuously learning during the work day. But when I get off work, I shut my computer down and live my life - spending time with my wife and friends, exercise, travel, concerts, etc. We travel - a lot. One of the benefits of remote work. If I do need to pivot back into hands on coding, I have a years worth of expenses in savings in the bank aside from retirement savings to give me time to get back into the groove of things.
- mettamage 2y ago
  My career started really rough, first 1.5 years didn't get a job and I have another 1 year gap. I don't have that luxury. I have experiences in stacks for 6 months tops with most. If it's longer it's an esoteric language. No one cares about my CS degree(s) and the experience I gained there, despite the fact I learned more in some courses than I did in most jobs. This is the first job I can fully showcase all my skills, so that's amazingly good. In that sense I need to seriously reconsider how I can make a career out of being a data analyst. But I'm not that far in my thinking yet and still see myself as a software engineer for my next job and while this job gives me great skills to learn in marketing and soft skills (and some stats and some dev), it doesn't give enough engineering skills. As a SWE I'm in an odd place. I have never been in a place where SOLID is a thing. I don't know design patterns that well (I can recognize them and in many cases they make sense). But I have engineered quite a bit and I have seen enough 1 million LoC or bigger codebases. So I'm still in a bit of a weird in between state and could use some mentoring with regards to good software design and just good ol' engineering, which is why I was thinking to help out with some open-source software or something like that. I agree that the "always be programming" culture is toxic. It is what it is. I'm the type that just rolls with the punches. Much of my life has been on hard mode for a variety of reasons, so a toxic culture like "always be programming" is honestly not the worst thing.
aristofun 2y ago
The only solution I found working is to leave data field and get back to software one (within same org or not). No other reliable and meaningful way. Nothing beats actual job in improving a complex skill. There is no shortcut to imitate that.
- mettamage 2y ago
  Aren't certain open-source projects like a job?
  - aristofun 2y ago
    Only if you work on it more or less full time and have colleagues or someone holds your accountable for the outcomes. Otherwise it's a hobby.
  - gtirloni 2y ago
    It depends. Plenty of developers have burned out due to support and maintenance work they don't appreciate (understandably). There's no silver bullet solution but delegating certain tasks to the community (if there's one) is one way to overcome it. Picking the right license also helps.
- Desafinado 2y ago
  I agree with this. At some point you need to work on a hard project, with minimal support, for an extended period of time. The skills you'll learn will be indispensable. Open source can help, but it's hard to replicate being immersed in a difficult role for 40 hours per week. The more important thing is that you have a genuine interest in the field, which should take you where you need to go. But without at least a small amount of hard software experience you'll be stuck at an intermediate level.
- xplt 2y ago
  This. I've been in a similar situation working as a data scientist for 2.5 years and went back to software engineering a couple of years ago. I'm not trying to say that data science and analytics are necessarily bad environments; I just came to realize that I had different expectations for my work than my organization. While the skills of a software engineer — e.g. quickly comprehending technical references, operating APIs, ability to type more than three lines of code straight — are highly valuable for a data analyst to be productive, I had to realize that my organization did not appreciate the craftsmanship of producing code as much as I still do. Over time, I had to witness my analyses end up on slides or in Excel workbooks, only to be looked at once. They'd done their job and weren't needed anymore. I was effectively "prompt completing" analytic requests from management to understand the organization/business/whatever better — always with the same result for "my work". Providing an organization with the intelligence to understand their business is for sure not a bad motivation and can be fulfilling. It's just not a good fit if — instead of the analytics — you consider the software you create for the analytics as your work.
purple-leafy 2y ago
Do fun software, games especially - you'll learn heaps. Don't do open source for the sake of it, unless its a product you actually use. Perhaps you could fork my simple raycaster and add your own enhancements to it, its a simple project [0] [0] - https://github.com/con-dog/2.5D-raycasting-engine/blob/master/README.md https://github.com/con-dog/2.5D-raycasting-engine/blob/maste...
muzani 2y ago
AI has made some once absurd ideas possible. For example, you can hardcode personal details and have a resume builder export it to PDF or a web portfolio. It'll probably be less work than formatting a resume in Word. Leave AI to do repetitive work like mapping data types, while you figure out the code to make it look pretty.
vismit2000 2y ago
The Missing Semester of Your CS Education: https://missing.csail.mit.edu/ https://missing.csail.mit.edu/ I would say finishing this helped me a lot to learn better and build up on tech stuff I use at my day job. This is a nice short course giving a very useful intro to software engineering tools required upon which you can build up further on the go, otherwise it's very easy to miss those topics and stay an 1x developer.
NotAnOtter 2y ago
Use them

19 points by mettamage 2y ago

cpach 2y ago

If you’re interested in security-related self-studies, I can warmly recommend Cryptopals: https://cryptopals.com/ https://cryptopals.com/

mettamage 2y ago
That brings up a good point! I could also go the hackthebox.eu route to train my security engineering skills, including cryptopals. PlayerTwo was still a fun insane box :)

rednafi 2y ago

Could you elaborate on this a bit? > But for a software engineer, the amount of programming feels low. It's more like creating quick scripts. I feel like it’s the other way around. When I worked as a data analyst (before transitioning to a data scientist and eventually a full-fledged SWE), my experience was quite different. As a DA, I mostly wrote aggregation queries, built dashboards, and did some Python, Pandas, and Notebook munging. On the other hand, SWE is a different world. I guess it depends on the type of SWE you are and the nature of the organization you work for. At a large-scale organization, as an SWE, I did far more programming work than I ever did as a DA. That said, there are different ways to cope and build skills—it takes time. I do side projects and, more importantly, write about them regularly [1]. Since I have to publish something each week, it forces me to think about different problems and seek out new technologies to try out so I can write about them. I also write about things I’ve figured out at work, which has been immensely helpful. Over the years, this act of actively seeking knowledge has added up and become substantial. More than once, people have reached out with job offers just because they stumbled across my writing on Hacker News or other social media platforms. Your approach might be different. Find something tangential to programming and work on it outside of work hours. Even a few hours per week adds up. For me, it’s side projects and writing. For you, it might be something else. [1]: https://rednafi.com https://rednafi.com

mettamage 2y ago
I think I explained myself poorly. > At a large-scale organization, as an SWE, I did far more programming work than I ever did as a DA. Yea, this is what I meant. The amount of programming I do now as a DA feels low (since I'm a software engineer historically, that's what I meant with "but for a software engineer" - a bit of a confusing sentence). Well specifically, the amount of engineering feels low. > That said, there are different ways to cope and build skills—it takes time. I do side projects and, more importantly, write about them regularly [1]. Good point, to what extent do employers see this as experience though? It's my experience they don't care at all. Maybe I'm looking at this wrong. The reason I'd do it is (1) I want to keep on building and (2) I want to keep on building experience as a SWE that employers will be impressed by. I'm not sure if I want to switch back, but for now I want to keep that option open. > More than once, people have reached out with job offers just because they stumbled across my writing on Hacker News or other social media platforms. I guess you answered your question here. I think it's good enough to test it out :) > As a DA, I mostly wrote aggregation queries, built dashboards, and did some Python, Pandas, and Notebook munging. Yea, I see a lot of SQL queries here, notebook stuff (so also Python/Pandas) and dashboarding. It's probably a 50/50 split. I have a suspicion that my manager wants me more on the programming side of things though. So far me as well. I'm allowed to pick any tool I find right for the job. I already wrote a bash script today.
- rednafi 2y ago
  > Good point. To what extent do employers even see this as experience, though? In my experience, they don’t care at all. Maybe I’m looking at this wrong. I don’t know to what extent this is considered experience either. That said, the hiring process in tech is completely broken. There are companies that ask LeetCode hard questions for positions where you’ll just write HTML templates and React. I work on side projects and write because I enjoy the process—not because I’m fishing for potential employers. In the past half-decade, I’ve written close to 200 articles. Researching for them was immensely helpful in building knowledge. At the start, I didn’t care if anyone read them. But eventually, the site got traction, and now it forces me to try new things and keep my skillset sharp. This works as a positive feedback loop.

petargyurov 2y ago

If you write a lot of "quick scripts", you might be writing a lot of the same boilerplate code over and over again. If that's the case, creating a framework/library that does X/Y/Z (related to your job) might help.

mettamage 2y ago
Yea that's fair, just started, so we'll see. Thanks for the tip though!

boricj 2y ago

If you're looking for suggestions outside of work, one important factor will be motivation. You're not paid for it so you'll need something that will hold your attention regardless, or you won't commit to it. Therefore, the answer depends on what you personally find interesting.

mettamage 2y ago
Yea I thought about that, probably game dev from the ground (with my own graphics engine) up or a web app that I find useful (currently no thoughts on that).

dagw 2y ago

As someone in a similar situation, my number one piece of advice is to meticulously keep up your software engineering best practices even when doing your day to day data analysis job. All that bread-and-butter stuff like tests and easy deployment and well structured code and reproducible builds and writing reusable tools and packages tends to fall by the wayside when your 'just' hacking one off Jupyter notebooks. As a software engineer you bring a perspective to the job that many of your colleagues may lack. Lean on that and use that background to help build better tools and ways of working. This not only makes your job (and the job of your colleagues) easier in the long run it will also let you keep your software engineering skills sharp.

aristofun 2y ago
> even when doing your day to day data analysis job This is a bad advice. Usually coming from an over-engineering mentality (that many engineers suffer from, including myself). Part of the engineering culture is to find the _right_ solution for the job. Not the most _engineered_ one. There is no point to shoot at sparrows from a canon.
- buffington 2y ago
  How is this bad advice? > keep up your software engineering best practices It's not a best practice to over-engineer solutions, and indeed, it is a best practice to find the right solution.
  - aristofun 2y ago
    When your job is to cobble up disposable scripts - there is no place for “practices”, unit test for example. Thus there is nothing for a software engineer to pay special attention to. Therefore there is no value in suggesting that While some may take that as “turn up your engineering knob” that leads to over engineering
    - mettamage 2y ago
      While I am fairly good at overengineering and optimizing, I have always hated doing it. For the longest time I wasn't able to do it but then I learned about low level hacking, and throwing things into assembly. Then I had a friend enter the HFT field and he taught me a thing or two about optimization. godbolt.org is fun to check out :) But yea, I really like making quick scrappy prototypes that deliver business results quickly. I've always had that mindset rather than crafting good quality code. In software engineering teams I threw that mindset to the wayside because they'd get mad. Sometimes rightly so as tests have saved my life, but quite often I think SWEs focus too much on code and being technical rather than delivering impact. I think in part that's also because it seems that in most places I've worked at product owners are "supposed to" think about that. The dev is only supposed to think about the technical implications, maybe the UX implications (depending on the company) but the moment a dev thinks about business implications, I've gotten quite a few times to "stay in my lane". In most cases the care for code quality isn't needed, sometimes it is. I'm taking a page from Jonathan Blow here (the game dev that created The Witness and Braid). He'd develop everything unoptimized and only later when he'd run into performance issues would he optimize it.

scarface_74 2y ago

I think one of the most toxic things in this industry is that we think we need to continuously be doing coding outside of work instead of living life. So the answer is I don’t. I had been programming continuously from the time I was in 6th grade in 1986 until 2020. As I pivoted into more strategic cloud consulting specializing in cloud architecture + application development in 2020 - “cloud application modernization” - I started coding less and less to the point now that almost any time that I even suggest writing code at work even though my title is “staff software architect”, I get my hand slapped and asked why didn’t I delegate it and focus on “higher impact work”. I keep my knowledge of the industry up to date by continuously learning during the work day. But when I get off work, I shut my computer down and live my life - spending time with my wife and friends, exercise, travel, concerts, etc. We travel - a lot. One of the benefits of remote work. If I do need to pivot back into hands on coding, I have a years worth of expenses in savings in the bank aside from retirement savings to give me time to get back into the groove of things.

mettamage 2y ago
My career started really rough, first 1.5 years didn't get a job and I have another 1 year gap. I don't have that luxury. I have experiences in stacks for 6 months tops with most. If it's longer it's an esoteric language. No one cares about my CS degree(s) and the experience I gained there, despite the fact I learned more in some courses than I did in most jobs. This is the first job I can fully showcase all my skills, so that's amazingly good. In that sense I need to seriously reconsider how I can make a career out of being a data analyst. But I'm not that far in my thinking yet and still see myself as a software engineer for my next job and while this job gives me great skills to learn in marketing and soft skills (and some stats and some dev), it doesn't give enough engineering skills. As a SWE I'm in an odd place. I have never been in a place where SOLID is a thing. I don't know design patterns that well (I can recognize them and in many cases they make sense). But I have engineered quite a bit and I have seen enough 1 million LoC or bigger codebases. So I'm still in a bit of a weird in between state and could use some mentoring with regards to good software design and just good ol' engineering, which is why I was thinking to help out with some open-source software or something like that. I agree that the "always be programming" culture is toxic. It is what it is. I'm the type that just rolls with the punches. Much of my life has been on hard mode for a variety of reasons, so a toxic culture like "always be programming" is honestly not the worst thing.

aristofun 2y ago

The only solution I found working is to leave data field and get back to software one (within same org or not). No other reliable and meaningful way. Nothing beats actual job in improving a complex skill. There is no shortcut to imitate that.

mettamage 2y ago
Aren't certain open-source projects like a job?
- aristofun 2y ago
  Only if you work on it more or less full time and have colleagues or someone holds your accountable for the outcomes. Otherwise it's a hobby.
- gtirloni 2y ago
  It depends. Plenty of developers have burned out due to support and maintenance work they don't appreciate (understandably). There's no silver bullet solution but delegating certain tasks to the community (if there's one) is one way to overcome it. Picking the right license also helps.
Desafinado 2y ago
I agree with this. At some point you need to work on a hard project, with minimal support, for an extended period of time. The skills you'll learn will be indispensable. Open source can help, but it's hard to replicate being immersed in a difficult role for 40 hours per week. The more important thing is that you have a genuine interest in the field, which should take you where you need to go. But without at least a small amount of hard software experience you'll be stuck at an intermediate level.
xplt 2y ago
This. I've been in a similar situation working as a data scientist for 2.5 years and went back to software engineering a couple of years ago. I'm not trying to say that data science and analytics are necessarily bad environments; I just came to realize that I had different expectations for my work than my organization. While the skills of a software engineer — e.g. quickly comprehending technical references, operating APIs, ability to type more than three lines of code straight — are highly valuable for a data analyst to be productive, I had to realize that my organization did not appreciate the craftsmanship of producing code as much as I still do. Over time, I had to witness my analyses end up on slides or in Excel workbooks, only to be looked at once. They'd done their job and weren't needed anymore. I was effectively "prompt completing" analytic requests from management to understand the organization/business/whatever better — always with the same result for "my work". Providing an organization with the intelligence to understand their business is for sure not a bad motivation and can be fulfilling. It's just not a good fit if — instead of the analytics — you consider the software you create for the analytics as your work.

purple-leafy 2y ago

Do fun software, games especially - you'll learn heaps. Don't do open source for the sake of it, unless its a product you actually use. Perhaps you could fork my simple raycaster and add your own enhancements to it, its a simple project [0] [0] - https://github.com/con-dog/2.5D-raycasting-engine/blob/master/README.md https://github.com/con-dog/2.5D-raycasting-engine/blob/maste...

muzani 2y ago

AI has made some once absurd ideas possible. For example, you can hardcode personal details and have a resume builder export it to PDF or a web portfolio. It'll probably be less work than formatting a resume in Word. Leave AI to do repetitive work like mapping data types, while you figure out the code to make it look pretty.

vismit2000 2y ago

The Missing Semester of Your CS Education: https://missing.csail.mit.edu/ https://missing.csail.mit.edu/ I would say finishing this helped me a lot to learn better and build up on tech stuff I use at my day job. This is a nice short course giving a very useful intro to software engineering tools required upon which you can build up further on the go, otherwise it's very easy to miss those topics and stay an 1x developer.

NotAnOtter 2y ago

Use them