61 ms·

Ask HN: What overlooked class of tools should a self-taught programmer look into

927 points by nathanasmith 7y ago

15 years ago I learned Python by studying some O'Reilly books and I have been a hobbyist programmer ever since. The books went into detail and since reading them I've felt confident writing scripts I needed to scratch an itch. Over time, I grew comfortable believing I had a strong grasp of the practical details and anything I hadn't seen was likely either minor quibble, domain specific, or impractically theoretic. This was until last year when I started working on a trading bot. I felt there should be two distinct parts to the bot, one script getting data then passing that data along to the other script for action. This seemed correct as later I might want multiple scripts serving both roles and passing data all around. Realizing the scripts would need to communicate over a network with minimal latency, I considered named pipes, Unix domain sockets, even writing files to /dev/shm but none of these solutions really fit. Googling, I encountered something I hadn't heard of called a message queue. More specifically, the ZMQ messaging library. Seeing some examples I realized this was important. The step of then plowing through the docs was nothing short of revelatory. Every next chapter introduced another brilliant pattern. While grokking Pub/Sub, Req/Res, Push/Pull and the rest I couldn't help breaking away, staring in space, struck by how this new thing I had just read could have deftly solved some fiendish memorable problem I'd previously struggled against. Later, I pondered the meaning of only now stumbling on something so powerful, so fundamental, so hidden in plain sight, as messaging middleware? What other great tools remain invisible to me for lack of even knowing what to look for? My question: In the spirit of generally yet ridiculously useful things like messaging middleware, what non-obvious tools and classes of tools would you suggest a hobbyist investigate that they otherwise may never encounter?

m_fayer 7y ago
https://dataintensive.net/ https://dataintensive.net/ I can't recommend this book enough. I have a CS background, and still had quite a few "I can't believe this thing has been hiding in plain sight!" moments while reading it.
- copperx 7y ago
  I'm now torn between reading this one first or the Architecture of Enterprise Applications.
  - politician 7y ago
    If it helps, IMO "Designing Data-Intensive Applications" is a better bang-for-the-buck. Enterprise-scale applications are a world unto themselves.
  - jb3689 7y ago
    I loved Designing Data-Intensive Applications. It gives you the reasons why NoSQL databases exist and the problems they solve. Moreover it gives you reasons to select one over another. It's really excellent and one of my top two CS books
    - mleonard 7y ago
      Your other top CS book out of interest?
  - copperx 7y ago
    Edit: I meant Patterns of Enterprise Application Architecture by Fowler in my comment above. Recommended by DHH.
    - louthy 7y ago
      My advice would be to skip it completely. It's just packed full of standard GoF OO dogma.
      - copperx 7y ago
        Thanks. So, what you're saying it is redundant if you've read GoF?
- redisman 7y ago
  It's great. Incredibly dense with useful information and it just blows my mind how much knowledge Martin has about the topic. I recommend watching this talk from him to give a little glimpse of the book: https://www.youtube.com/watch?v=5ZjhNTM8XU8 https://www.youtube.com/watch?v=5ZjhNTM8XU8 This is just about a little part of one of the chapters.
  - mirekrusin 7y ago
    Oh man, this is good, thank you.
- splittingTimes 7y ago
  But this is mainly for distributed (web) systems, right? Are there good books for data intensive desktop apps? Like games or CAD design tools?
nmca 7y ago
SQL
- mfatica 7y ago
  You think a self-taught programmer might never encounter SQL?
  - politician 7y ago
    Speaking as a technical development manager, I can say that many people have far less than adequate exposure to SQL, and that training people to use SQL effectively and safely is all too often a common requirement before letting them loose on the database. There are so many ways people misunderstand and misuse SQL and relational databases, it's honestly staggering. So, I second the OP. Learn SQL, and you'll stand out.
    - deleted 7y ago
      [deleted]
    - beat 7y ago
      Yeah, learning real sql is a very useful skill. At one employer, I became the "sql expert" because I knew the difference between inner and outer joins. Which, if you know sql, means you know next to nothing. But knowing next to nothing was better than knowing nothing at all, so...
kyllo 7y ago
Learning in-depth your various options for persisting data, is very useful since most applications have to deal with persistence in some form, and increasingly in a distributed manner. Go beyond simply skimming the surface of SQL vs. NoSQL and the marketing claims different databases make about their scalability and consistency. Learn what ACID and CAP stand for and the tradeoffs involved in different persistence strategies. Learn SQL really well. Learn how to read a query plan, which is the algorithm your SQL query gets compiled into. Learn about the tradeoffs of row-based vs column-based storage. Learn how indexes work, and what a B-tree is. Learn the MapReduce pattern. Think about the tradeoffs between sending code to run everywhere your data is stored vs. moving your data to where your code is running.
- theossuary 7y ago
  Two great resources I've been going through are - https://dataintensive.net https://dataintensive.net - Really deep dives into different types of data storage solutions, their history, and how they actually work. - http://www.cattell.net/datastores/Datastores.pdf http://www.cattell.net/datastores/Datastores.pdf - Good paper that helps differentiate similar but different datastores. Really helpful when you're trying to pick a modern data solution.
  - hypertexthero 7y ago
    Another good resource that guides one through both the philosophy (why) and technical details (how) of building a web application is Software Engineering for Internet Applications: https://philip.greenspun.com/seia/ https://philip.greenspun.com/seia/
  - barbecue_sauce 7y ago
    Designing Data-Intensive Applications is probably the best O'Reilly (if not overall technology) book of the past decade.
    - btown 7y ago
      The talk on "Turning the database inside-out" [0][1] by the author, Martin Kleppmann, is a fantastic intro to these dynamics, and it's something I'll always recommend to both experienced and inexperienced data modelers and backend developers. It goes pedagogically through the way things are typically done in a relational database in such a clear way that word-for-word it's one of the best tutorials I've seen... but it also weaves a narrative of "how can this be done better/more scalably/more reliably/more flexibly-to-business-needs" in pointing to a streaming/event-sourcing architecture. You may or not need the latter right away, but it's a fantastic tool to have in your toolbox to be able to say "ah, this new requirement feels like it would benefit hugely from this architecture." Especially for OP who's starting to think about the "why" of messaging queues, this could be a fantastically valuable first step. [0] https://www.youtube.com/watch?v=fU9hR3kiOK0 https://www.youtube.com/watch?v=fU9hR3kiOK0 [1] https://www.confluent.io/blog/turning-the-database-inside-out-with-apache-samza/ https://www.confluent.io/blog/turning-the-database-inside-ou...
mattnewport 7y ago
If you're a self taught hobbyist you may not have had much structured exposure to fundamental data structures and algorithms and complexity analysis. I think that type of thing is easier to learn when you already have some experience so you can relate it back to real world problems you have encountered as you describe doing here. Now might be a good point in your development to dig into some of those fundamentals if you have not done so much in the past.
gaze 7y ago
Get good at math. It'll serve you well and never go out of style.
- lostphilosopher 7y ago
  I'd recommend statistics specifically. Comes up everywhere and it's easy to be wrong about if you don't dig into it.
  - wainstead 7y ago
    Second this, if you only work through the Khan Academy stuff on the various things like scatter plots, standard deviation, the normal distribution, etc. you'll be much better off for it.
- ThrustVectoring 7y ago
  Math is a bit too broad. From personal experience, the most relevant topics for programmers are Linear Algebra and Discrete and Combinatorial Algebra.
  - gaze 7y ago
    Eh I mean, I think there's a certain discipline that comes with studying any branch of math to a certain degree of rigor. But yes, linear algebra and discrete math are probably the most useful. I think control theory and optimization are under appreciated amongst programmers though.
quanticle 7y ago
I would recommend learning the SOLID [1] design principles. I've found them to be a very helpful guide when designing software components. [1] https://en.wikipedia.org/wiki/SOLID https://en.wikipedia.org/wiki/SOLID
- deathanatos 7y ago
  Particularly the S — the Single Responsibility Principal. So much messy, convoluted code is convoluted because it lacks a singular, clear purpose, and bundles up multiple responsibilities into one section of code, be that a module, class, or whatever is appropriate to your language. They're all good, and you'll get good insight from them all, but I think that first one is more important and has provided me more value than all the rest. I think Liskov substitutability would be my second pick.
  - collyw 7y ago
    Agree completely.
fancy_pantser 7y ago
The next obvious step after message queues and distributing work would be streams. Here is an excellent introduction to unified logs and stream processing by the author of Kafka at LinkedIn: https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying https://engineering.linkedin.com/distributed-systems/log-wha...
Robin_Message 7y ago
Read the curriculum of an undergraduate computer science course and read up on the things you haven't heard of. Some courses will even have lecture notes available. E.g. these four pages are the university of Cambridge masters in computer science: https://www.cl.cam.ac.uk/teaching/1819/part1a-75.html https://www.cl.cam.ac.uk/teaching/1819/part1a-75.html https://www.cl.cam.ac.uk/teaching/1819/part1b-75.html https://www.cl.cam.ac.uk/teaching/1819/part1b-75.html https://www.cl.cam.ac.uk/teaching/1819/part2-75.html https://www.cl.cam.ac.uk/teaching/1819/part2-75.html https://www.cl.cam.ac.uk/teaching/1819/part3.html https://www.cl.cam.ac.uk/teaching/1819/part3.html (Or a MOOC, but the links above are easy to browse text, syllabuses and lecture notes, not a load of videos.)
- rat9988 7y ago
  Thank you very much. This is very valuable.
- JoeMalt 7y ago
  To clarify: the first three links are for each year of the (three-year) undergrad program, the fourth is for the Masters. The Cambridge course isn't perfect, but they do a very good job of making as much teaching material as possible publicly available.
- learc83 7y ago
  I support this 100%. I worked for years as a self-taught programmer. When I went back for my CS degree, I was shocked at how much I didn't know that I didn't know. Numerous times I'd be sitting in a class and we'd go over a solution to some theoretical problem, and I'd realize that this solved a problem that had taken me days to discover on my own (and this solution was usually better than what I'd come up with). If you are the kind of person who can work through everything on your own (including what may seem like the the boring parts), I highly recommend doing so.
  - 147 7y ago
    Could you give an example of one of the times something theoretical helped you solve a real world problem? I've thought about going back for my CS degree a lot but can't really justify the cost and time investment vs self teaching. But it's something that's always been in the back of my mind.
narag 7y ago
A few decades ago, a new programming environment exploded: the web. Looking for a ridiculously useful tech stack? Look no further: HTML, HTTP architecture, SQL backend... a guide written at the time: http://philip.greenspun.com/panda/ http://philip.greenspun.com/panda/ Paul Graham also wrote about why the web was such a deal, IIRC in the Beating the Averages essay. In particular: you can use whatever tools you want and avoid deploying to client machines.
- cryptonector 7y ago
  PostgreSQL + PostgREST + react-admin == fantastic stack. You can write an entire application in SQL and PgPlSQL but using an HTTP JSON API as the interface with a static and responsive BUI. This allows you to be extremely agile in development and ops (because, e.g., you get to use logical replication). I can't say enough good things about this approach.
  - pandemic_region 7y ago
    indeed if your app is about managing your personal dvd collection, or variants thereof, such anemic UI-to-database tools work very well. Not when there is complicated domain logic involved.
    - cryptonector 7y ago
      Having built such an application (w/ complex business logic), I have to disagree. But I'd like to hear what problems you've run into. I would agree that react-admin is a bit too simple, and this stack really calls for a BUI to be built specifically to fit the PostgREST model.
  - davb 7y ago
    This reminds me of how we used to create thick client desktop software, connecting directly to the database. Doing most of our business logic on the front end and leaning on the database for auth, relational constraint enforcement and (via stored procedures) transactional consistency and validation. It felt weird at the time when the whole world seemed to move to a three layer model with app servers sitting in the middle - now it's hard to imagine it any other way.
mosalarynolife 7y ago
Design Patterns
- sesser 7y ago
  What are some resources outside of gang of four?
  - qazpot 7y ago
    Head First Design patterns
  - pernambucano 7y ago
    https://sourcemaking.com/design_patterns https://sourcemaking.com/design_patterns
  - jimpudar 7y ago
    Patterns of Enterprise Application Architecture Enterprise Integration Patterns Just start reading some Martin Fowler books, you will be up to your ears in patterns in no time :)
- groby_b 7y ago
  Just do yourself a favor and read it the way it's intended: As a dictionary. Design patterns give you a vocabulary to describe design choices you made. They don't give you a set of things that inform design.
- wwweston 7y ago
  I can't remember who, but someone's said that design patterns really would be more appropriately called something like "palliatives for static manifestly typed languages." Knowing idioms that relate to certain expressive / organizational problems is a good thing (especially if you're primarily working in a static manifestly typed language), but there's a weird overcelebrated status to them, and I'm not sure I'd encourage a developer I was training to become familiar with a full catalogue of them.
  - 7y ago
aequitas 7y ago
Makefiles. I always dismissed them as a C compiler thing. Something that could never be useful for Python programming. But nowadays every project I create has a Makefile to bind together all task involved on that project. From bootstrapping the dev environment, running checks/test, starting a devserver, building releases and container images. Makefiles are just such a nice place to put scripts for these common tasks compared to normal shell scripts. The functional approach and dependency resolving of Make allows you to express them with minimal boilerplate and you get tab completion for free. Sometimes I try to take more native solutions (eg. Tox, docker) but I always end up wrapping those tools in a Makefile somewhere forthe road since there are always missing links and because Make is ubiquitous on nearly every Linux and macOS it is just all you need to get a project started. Example: https://gitlab.com/internet-cleanup-foundation/web-security-map/blob/master/Makefile https://gitlab.com/internet-cleanup-foundation/web-security-... Running the 'make' command in this repo will setup all requirements and then run code checks and tests. Running it again will skip the setup part unless one of the dependency files has changed (or the setup env is removed).
- pletnes 7y ago
  One of the worst problems with using windows (in my opinion) is that there’s no native GNU make.
  - aequitas 7y ago
    Isn't non-native development on Windows a solved problem nowadays with WSL(2)?
    - murderfs 7y ago
      WSL is currently horrendously (unusably, IMO) slow. WSL2 promises a 20x speed up, but it was already 100x slower than native Linux at some actually-realistic workloads that happen all the time when you're developing (e.g. `git grep`), so it's probably still too slow to be tolerable. I had the opposite problem of wanting to develop some stuff for Windows from a Linux environment, and I settled on running a linux VM and copying binaries over by scping to WSL, which works reasonably well.
mettamage 7y ago
It depends on what you want to achieve. I can tell you to read about security but that is only relevant if you want to secure your software. I can tell you about pub/sub, but that is only relevant when you need it. The things you’ve learned were mostly covered in my computer science curriculum. So I am going to second Robin_messsage his message. Skim through a CS curriculum.
zoomablemind 7y ago
I'm not sure if 'hoarding' such knowledge would be practical. Sometimes, having a limited toolset would better focus you on the problem at hand. Then, once the challenge is clarified, the search for alternative ways to architecture and implement it would become practical. If just for fun of exploring something new, pick whatever interests you in general, language, framework, a domain. Async processing and idioms?
vosper 7y ago
Your IDE, its refactoring tools, but especially its debugger. VSCode or PyCharm (assuming you are still a Python developer) could be a good place to start. I'm always surprised when I see professional developers coding in Sublime Text and debugging with print statements (or their equivalent). Usually you have better options than that, especially for statically-typed languages - but even for JS and Python.
- collyw 7y ago
  I don't know how people could work on large scale projects without a step through debugger.
danesparza 7y ago
Caching, logging, centralized configuration, Security, and Design Patterns are all probably easy to overlook.
nerpderp82 7y ago
Debuggers and property based testing. It is a select few people that can actually productively (not their own metrics) use print statements for debugging. Learning how to craft repro scenarios and adequately capturing state in a debugging session can enable junior devs to easily surpass senior devs. Property based testing aren't quite formal methods, but I think they are a good stepping stone. And they also somewhat force your code into an algebraic/functional style which also make it amenable to refactoring, better testing and is easier to understand. Design tools like Swagger can help one think through services w/o diving into code. Code itself is a liability and should be thought of as "spending" not creating. Code is a debt. Refactoring and code understanding tools, if you use PyCharm (you should, it is free in all senses), learn how to navigate into your libraries. Read you libraries.
- marktangotango 7y ago
  This x1000 debuggers are seriously undervalued by many developers. It’s like a super power.
  - thetwentyone 7y ago
    What are some good resources to learn about debugging patterns and tips/tricks? My preferred language, Julia, recently introduced a nice set of tools related to debugging. I feel like there's probably things that would make me more productive but I think the techniques would be more broadly applicable than a specific language.
    - drainyard 7y ago
      One thing I always do these days is I step through any new code I've written the first time I run it. This usually weeds out some bugs that might take a while to find because they are easy to miss. It also ensures that you actually go through each line of code you write, doing a forced code review on yourself early on.
brianpgordon 7y ago
Don't go too overboard with message queues. There's nonzero development and operational overhead incurred when part of your application takes its input in a weird binary format, and when the data in your queue is thrown away after processing, and when you need to think about scaling of workers and concurrency. If you're not working with real "big data" – and, let's be honest, almost nobody is – I would advise using an HTTP-based service (REST, SOAP, whatever, take your pick) for communication and a SQL/NoSQL/NewSQL database for state.
- jrockway 7y ago
  ZeroMQ is not really a message queue, it's more of a networking library. It takes TCP sockets and adds other concepts on top, like request/reply or publish/subscribe.
- massaman_yams 7y ago
  Doing a task queue in an RDBMS is not as simple as some might think, though. https://www.2ndquadrant.com/en/blog/what-is-select-skip-locked-for-in-postgresql-9-5/ https://www.2ndquadrant.com/en/blog/what-is-select-skip-lock...
amrox 7y ago
Learning the basics about how programming languages work - parsers, interpreters/compilers. I've heard good things about Writing An Interpreter In Go [1]. Related, I've enjoyed Martin Fowler's DSL book [2]. - [1] https://interpreterbook.com - [2] https://www.martinfowler.com/books/dsl.html
- O_H_E 7y ago
  Just noting that the URLs weren't formatted as links but code
purplezooey 7y ago
Brown nosing.
new_guy 7y ago
'The Imposters Handbook' is good for foundational knowledge, it was written for exactly your use case. https://bigmachine.io/products/the-imposters-handbook/ https://bigmachine.io/products/the-imposters-handbook/
- LeonB 7y ago
  Came here to recommend this too. It’s goal is very much aligned with the OP.
paulgb 7y ago
State machines. They can be a helpful abstraction for UI or business logic, and they also make regular expressions make a lot more sense.
- jimpudar 7y ago
  Yeah, far too often people create horrific spaghetti code of if/else statements which could be neatly coded as a state machine. This is a very important concept to grasp.
deleted 7y ago
[deleted]
lixtra 7y ago
Learn to use a profiler for your programming language. It will tell you what’s worth optimizing in your programs. I.e. if your program spends only 10% of the time talking to others than that may be the maximum you can optimize by choosing a fancy communication pattern.
westonplatter0 7y ago
SQL
throwaway55554 7y ago
Develop a project that has two local processes that have to communicate, one local database, and one remote service that the two processes have to communicate with. Doesn't matter what the project is. Could just be tossing around a string of data that eventually gets dumped into the db and sent to the server. Research different methods of architecting such a system. Code up a few. By doing this, you will actually find the answer to the question you posed. And, arguably, have fun doing it.
- lioeters 7y ago
  I love it. The best way to learn is by doing, and the suggested project idea is simple enough as a starting point, and complex enough to require deep thinking and exploration in various aspects to figure out a good solution. Many of the recommendations in other comments could tie into architecting such a study project too: data structures, algorithms, communication among distributed processes..
hannob 7y ago
If you happen to program in C/C++ you should absolutely familiarize yourself with the sanitizers of gcc and clang, most notably address sanitizer. (However there are good reasons why you shouldn't program in C/C++ to begin with.)
justinholmes 7y ago
Using GitHub trending for each language look at new things each month. Build something and think how would I scale this project to X capacity then rinse and repeat.
joker3 7y ago
https://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-005-software-construction-spring-2016/ https://ocw.mit.edu/courses/electrical-engineering-and-compu... In my experience, most hobbyist programmers are fine writing scripts and other small programs but have no exposure to the sort of ideas that are necessary for making large programs. This course at least exposes you to a lot of those concepts.
- metahost 7y ago
  The latest course is available here: http://web.mit.edu/6.031/www/sp19/ http://web.mit.edu/6.031/www/sp19/
avip 7y ago
The concept of monitoring. This simple, obvious concept changed a lot how I think of software maintenance. I usually use slack but that's an implementation detail.
gwbas1c 7y ago
Unit testing, mocking, and various other testing techniques. Why? Any project of sufficient complexity is very hard to test. If all you're doing is code -> build -> run to debug your code, you can very easily break something that's not in your immediate attention. The problem is that good unit testing is hard, and time consuming. It can be so time consuming, that unless you can really plan in advance how you test, you could spend more time writing test code than real application code. (This is what happens when writing professional, industrial-strength code.) So, when a hobby project becomes sufficiently interesting enough; such that the code will be so complicated that your code -> build -> run loop won't hit most of your code, you should think about how to have automated tests. They don't have to be "pure, by the book" unit tests, but they should be an approach that can hit most of your program in an automated manner. You don't need to do "pure" mocking either. If you're writing something that calls a webservice, you could write a mock webserver and redirect your program to it. If you're writing something that works with pipes, you could have a set of known files with known results, and always compare them. The goal is that you should cover most of your program with code -> build -> tests; and only do code -> build -> run for experimentation.
- wpietri 7y ago
  Let me second this. And in particular, I strongly encourage every developer to try starting a new project in a test-driven fashion (by which I mean that you advance the code by writing a bit of test and making it pass, and then doing that over and over.) There's a qualitative difference between working in a well-tested code base that's very hard to describe convincingly. A lot of my early development experience was in code bases that had little or no testing. Experiencing a well-tested code base totally changed things for me. Instead of work being a death-of-a-thousand-cuts experience, it became pleasant, steady progress.
  - mooreds 7y ago
    > Experiencing a well-tested code base totally changed things for me. Instead of work being a death-of-a-thousand-cuts experience, it became pleasant, steady progress. I had the luxury of taking a well known data process and rewriting it with integration tests (input in, matching output with a golden file). It changed my professional life. Whereas before our deployment process included a 3 day wait and manual data checking on stage, after I was able to do deploys multiple times a day with confidence. Made a believer out of me.
enkiv2 7y ago
I highly recommend learning PROLOG & understanding how to write your own simple planner system. The hairiest real problems are hairy because they're best suited to a declarative style (and programs written declaratively can be made much more efficient through more clever solvers -- given naive code, a clever solver has a much bigger efficiency boost over a dumb solver than an optimizing compiler does over a non-optimizing one -- although PROLOG itself leaks too much abstraction for many of these techniques to be viable in it). I also recommend understanding message routing systems used in file sharing, like CHORD. If you don't have a strong background in the math behind theoretical computer science, you might benefit a lot from an understanding of the formal rules around boolean logic, symbolic logic, & state machines -- especially, rules about when certain kinds of things are equivalent (since stuff like demorgan's law are used for simplifying and optimizing expressions a lot, and rules for state machines are used to prove upper limits on resource usage). If you don't already, learn to use awk. It's a much more powerful language than it seems, and fits extremely well into the gap between command-line prototyping in shell one-liners & porting a prototyped tool to python or perl, and so it's a huge time saver: it is faster to write many kinds of tools in a mix of shell and awk and then rewrite them in python than it is to write them in python in the first place.
- bdamm 7y ago
  I've never used Prolog in the 15 years since I learned it in college. It's an interesting take on programming, for sure, and I appreciated the mind-expanding exercise, but hasn't helped me in my career at all. Totally agree on awk. I use it almost every day for quick little one-liners. Big time saver. Also agree on state machines, because from there it is a short hop to understanding formal grammars and the foundation of compilers and languages, which has been immensely useful in my career.
  - doboyy 7y ago
    Learning about automata theory was one of the most mind-expanding experiences I had in college. It was certainly not something I would've stumbled upon without guidance. I believe that describes most of the value I derived from a degree, being nudged in the right directions toward solutions and problems that a lot of smart people have though about for a while. Understanding how different languages, or inputs more generally, can be transformed into meaningful outputs is pretty satisfying. It's a topic that almost seems to transcend the realm of computer science.
CalChris 7y ago
Godbolt Compiler Explorer: https://godbolt.org/ https://godbolt.org/ DTrace: http://dtrace.org/ http://dtrace.org/
WoodenChair 7y ago
The best tools you can teach yourself are arguably more fundamental than any specific language or library, but instead computer science problem solving techniques. I've written books on this subject specifically targeted towards self-taught programmers: https://classicproblems.com/ https://classicproblems.com/ Sorry for the self-plug, but it's super relevant I think.
jimpudar 7y ago
Learning how to use dtrace / bpftrace [0] is very valuable if you ever need to get into serious systems profiling. There are some really cool data structures out there you might not know about. One of my favorite basic ones that I get a lot of use out of is the trie [1] (a.k.a. prefix tree). Very useful for IP calculations. Also look into probabilistic data structures [2], very amazing things can be done with them. [0] https://en.wikipedia.org/wiki/DTrace https://en.wikipedia.org/wiki/DTrace [1] https://en.wikipedia.org/wiki/Trie https://en.wikipedia.org/wiki/Trie [2] https://en.wikipedia.org/wiki/Category:Probabilistic_data_structures https://en.wikipedia.org/wiki/Category:Probabilistic_data_st...
- cryptonector 7y ago
  DTrace is life-altering. I keep hoping that someone will build a dtrace(1) CLI that transpiles to bpftrace.
- stcredzero 7y ago
  Profiling, period
- sammnaser 7y ago
  Bloom Filters are awesome.
- Const-me 7y ago
  Approximate Windows equivalent of DTrace is sysinternal process monitor, freeware. Very useful sometimes.
  - Birch-san 7y ago
    The Windows equivalent of DTrace is.. DTrace. [0] DTrace is about far, far more than snooping the filesystem. At best, Process Monitor is an equivalent of Brendan Gregg's DTrace utility, opensnoop. The true power of DTrace is to correlate events across subsystem boundaries. Like, graphing the top quartile of latencies from network acceses initiated via a given function in your application. [0] https://techcommunity.microsoft.com/t5/Windows-Kernel-Internals/DTrace-on-Windows/ba-p/362902 https://techcommunity.microsoft.com/t5/Windows-Kernel-Intern...
malvosenior 7y ago
If you're coming from Python you should start looking into how other languages handle concurrency. Python has a GIL (global interpreter lock) that only allows for single threaded execution under normal circumstances. Learn about threads, locking, mutexes, semaphores, green threads, race conditions...
neilv 7y ago
At some point, it helps to broaden your programming language exposure, even if you stick to mostly one language for most of your work. You'll find ways to apply ideas from other languages/communities to your work. Try to spend some time learning idiomatic programming from one of the Lisp family (Scheme, Racket, CL, Emacs Lisp, and Clojure all have different thinking, but a lot of overlap). Play a bit with Smalltalk or a similar descendant, even if you're already doing OOP elsewhere. At some point you should learn a textual expansion language, like one of the Unix shell scripting ones, or Tcl (and learning basic Bash scripting will probably be useful in tech work). Try a logic programming language, like Prolog, or one that's a minilanguage within another, like Mini-Kanren. Maybe buckle down for hardcore functional programming (e.g., Haskell, OCaml, or discipline yourself to do it in a Lisp?). You should also get comfortable with C or at least an assembly language at some point, to have a better idea of what other languages are and aren't giving you, and also C is just a really useful thing to know when you need to write a little fast code, FFI to a native library, or get into languages/IRs for newer target architectures. (Disclosure: I've been especially involved with Racket, an energetic close descendant of Scheme, and have some interest in promoting it, but I'd list a Lisp as one of the first in any case.)
- jimpudar 7y ago
  Racket is a very good Lisp for beginners, DrRacket makes it easy to get up and running with little effort.
  - neilv 7y ago
    Agreed. Racket is from a particular school of thought, and you won't get all Lisp family ideas from it, but it's great. You can start up DrRacket (a simple IDE for students that can also be used for professional work, and has some powerful features in it), or just use your favorite editor and the `racket` command-line program and REPL. There's way too much documentation at: https://docs.racket-lang.org/ https://docs.racket-lang.org/ You can also do the old MIT introductory CS textbook, SICP, using Racket.
    - snazz 7y ago
asdffdsa 7y ago
Books: Operating Systems/Database/Networking/Computer Security/Computer Architecture textbooks, Software Engineering textbooks (Clean Code, Design Patterns, Designing Data Intensive Applications, Domain Driven Design as a short list off the top of my head)
- stevekemp 7y ago
  By the same token "documentation". Today I explained to a colleague how they could enable port-forwarding across their already-open-SSH-session via "~C". Every now and again I pick a tool I use a lot, and read the man-page. Things like "less", "bash", "ssh". Complex enough to contain surprises, but simple enough that you take them for granted. Almost always this has been time well-spent.
- lbrindze 7y ago
  "Designing Data Intensive Applications" is an absolute goldmine for things like message queues but also going beyond understanding the full implications in database selection and other common, distributed-oriented engineering decisions modern software engineers may come across.
OliverJones 7y ago
Event-driven, asynchronous, programming Here's a valuable addition to your toolkit of mental models for programming: Event-driven, asynchronous, programming (in the style of ES6 Javascript or a similar language. ) Some suggestions about how to learn the basics? tutorials on... -- building a so-called "single page web app" with a framework like vue.js or even jQuery. -- node.js to build a complex back-end server without using a threading model) -- React (to build an interactive program to run in a browser)
agentultra 7y ago
Formal methods. It took me nearly a decade of working in distributed systems to be introduced to TLA+ and other tools in this space. Until then my knowledge had been built from textbooks describing the fundamental data structures, algorithms, and protocols... but those texts take an informal approach with the mathematics involved. And since I was self-taught I was reading those texts with an eye for practical applications than for theoretical understanding. I had no idea that a tool existed that would let me specify a design or find potential flaws in systems and protocols, especially concurrent or parallel systems, with such ease. I think type theory and category theory have also been great tools to have... but I think mathematics in general is probably the more useful tool. Being able to think abstractly about systems in a rigorous way has been the single-biggest booster for me as a practitioner.
- pickdenis 7y ago
  > the single-biggest booster for me as a practitioner. Can you instantiate this claim with an example? I'm somewhat knowledgable in both math and computer science theory but have yet to feel as though my math background has helped me in practical CS.
  - agentultra 7y ago
    About 3-4 years ago I was working on an open source cloud platform for a company deploying a public cloud. There was a particular error that sometimes happened in our production environment where a rebooted VM would come up but couldn't connect to the network. We tracked it down to a race between two services in the data plane. It turns out the VM controller wouldn't wait for the network controller to unplug a virtual interface before requesting the interface to be plugged back in. There was a lack of co-ordination happening. However it only happened when the network component was under heavy enough load that it would take too long to respond before the VM service finished rebooting the VM -- usually it was fast enough and the error wouldn't appear. I managed to model this interaction at a high level in TLA+. From there I had suspected that the error was in the mutex locking code in the async library this system depended on so we refined the model pretty close to that implementation. As I recall we found that the mutex code wasn't the culprit -- a fine result. We ended up implementing some light-weight co-ordination mechanism to ensure that the VM service waits to acknowledge the progress of the network service. Since then I've continued to use TLA+. I find that programming languages are insufficiently expressive enough to describe high-level interactions between other processes, network events, and humans.
jpmelos 7y ago
If understanding how Linux works interests you: http://www.linuxfromscratch.org/ http://www.linuxfromscratch.org/
caust1c 7y ago
Heh, funny that you mention that ZMQ is overlooked. While often though of a message passing library or serverless queue, Zero MQ has some pretty severe limitations that implementers fail to consider. To be clear, I think it's an amazing library which is unmatched in its performance, but it comes at a cost: reduced reliability. ZeroMQ will drop messages in a number of situations. The library does not handle delivery guarantees which means that the application must do it themselves. Whether or not this works for you is an application level concern. However, having used it at two companies now: both times it ended up being thrown out for a more reliable queue (kafka). http://zguide.zeromq.org/py:all#Missing-Message-Problem-Solver http://zguide.zeromq.org/py:all#Missing-Message-Problem-Solv... So maybe the reason you haven't stumbled upon it sooner is because it's overhyped? Definitely useful but with a grain of salt.
- aasasd 7y ago
  Yeah, every new user has to learn that ZMQ is not a proper queue, but simply a network library and protocol. How is it even possible to bungle the name so badly and then do nothing about it for twelve years?
- wallstprog 7y ago
  The "MQ" part of the name is unfortunate, but apparently came about because of the original idea to come up with a "better" implementation of AMQP (http://zeromq.org/docs:welcome-from-amqp http://zeromq.org/docs:welcome-from-amqp). But you're right -- ZeroMQ doesn't do queueing (except in some very limited circumstances), and if you need reliable delivery you must implement that yourself "on top of" ZeroMQ. I've done that, and while it's not a simple task, it is certainly possible. You can get reliable delivery "out of the box" with other software, that in fact does do queueing. (kafka may be one, but I don't know enough to say). But what you give up when you do that is performance -- ZeroMQ can easily be orders of magnitude faster than those other solutions, and for some applications (e.g., real-time market data) the work to provide a custom reliability solution on top of ZeroMQ is worthwhile.
Jach 7y ago
If you're strictly asking about things a self-taught hobbyist programmer may have missed, then I second the suggestions here to skim through a CS degree curriculum and dig into anything that's unfamiliar. As one possible example, maybe you're solving a problem trying to parse some text, and you're in over your head with ad-hoc regexes and conditionals and type casts and exceptions all over the place. If you've seen the concept of a grammar (likely to come up in CS programs, though not all) and of generating parsers / validators for them, you can eliminate a whole lot of programming by specifying a grammar in some common format (for instance, ABNF) and running it through a program that generates a parsing program for you. The general category of programs writing programs is worthwhile to look into, and playing with languages where such a feature is first-class (like Common Lisp, which apart from having macros also has a compile-file function you can invoke at runtime) can be enlightening. A lot of the comments here are highlighting things that not even most CS degree holders will know about, nor many professional programmers. Those can be useful for a hobbyist too, so maybe the lesson is that regardless of your current level of knowledge it's important to keep in mind: "There are more things in heaven and earth, Horatio, than are dreamt of in your philosophy." Your comment about feeling comfortable in the practical details reminded me of how I used to feel as a fresh self-taught PHP-wielding teenager after a few years of it. As an "expert" PHP coder, I could do anything! Even an OS if I wanted! Well I learned better eventually. :)
- stormking 7y ago
  "If you've seen the concept of a grammar (likely to come up in CS programs, though not all)" Really? That was one of the very first topics in my first semester. How do you teach CS without grammars and the Chomsky hierarchy?
  - Jach 7y ago
    I wondered if someone would ask about that... It's just my own recollection from years ago when I looked at a bunch of CS curricula from public and private schools. There was a lot of patchy variation. If the topics were covered at all they'd typically be done in optional electives like a compilers course, or "theory of computation" course, taken in junior or senior years. Also depending on the school such an elective might not actually be available anymore (I looked at current-semester/last-semester offerings to try and identify those) and is only part of the catalog for historical reasons. Maybe it was offered once, but not anymore, in part because students having the choice would rather take the new data science course or Advanced Networking or whatever to fulfill that elective instead. I do think the problem is mostly a function of having lots of choice. Everyone's got their own list of what a CS degree absolutely must cover, but there are going to be differences, and schools are incentivized to let students carve their own path. Let's look at this list: http://matt.might.net/articles/what-cs-majors-should-know/ http://matt.might.net/articles/what-cs-majors-should-know/ I've looked at a lot of intern resumes over the past few years and broadly they've been impressive from a hire-to-BigCo perspective but I would be surprised if many of them knew much if anything about a large majority of topics from that list. Even narrowing to the "better" schools. I also don't even agree with that list, I just don't think it's realistic to cram all that into a 4 year program on top of all the other STEM courses, humanities courses, and project courses... Going back further there's the continuing problem of "dumbing down". It ties into a school's incentive to offer more choices to fulfill the graduation requirements (creating an "easy" path), but as a problem in itself it's worth considering. https://www.joelonsoftware.com/2005/12/29/the-perils-of-javaschools-2/ https://www.joelonsoftware.com/2005/12/29/the-perils-of-java... is a classic rant about it, not a very good one but it points at one consequence of the problem. I would bet there's still a sizable bunch of current-year CS graduates who never had to deal with pointers. Maybe I'll do some data exploring some time with the resumes I still have to see what fraction put C or C++ somewhere to make a weak proxy measure.
jedberg 7y ago
If you're into python, look up decorators and coroutines. They will blow your mind.
- Jach 7y ago
  This is still my favorite mind-blowing introduction to them, from 2009: http://www.dabeaz.com/coroutines/Coroutines.pdf http://www.dabeaz.com/coroutines/Coroutines.pdf Don't have hardware interrupts or threads but still want a multi-tasking operating system? Not A Problem.
crimsonalucard 7y ago
Nobody mentions what self taught programmers miss the most. Theory. And not just algorithm theory.
- Zealotux 7y ago
  Could you expand on that if you don't mind? I'm a self-taught programmer currently in the process of realising just how much I miss from theory, my list so far on theory fundamentals includes: • Mathematics and probabilities, and their applications to CS (e.g. formal methods) • Design patterns (OOP, functional programming) • Data structures • Algorithms, time complexity • System architectures • Software strategies: CI, CD • Database principles: SQL It may sound naive, but I'm kind of overwhelmed by all this and it's not helping my impostor syndrome, I may make a repo of the list with links to ressources I've identified for learning as it seems like a common struggle, maybe it'll help someone.
  - crimsonalucard 7y ago
    None of what you mentioned is theory expect for Mathematics, probability, formal methods and algorithms. Stuff like design patterns, system architectures and software strategies are like flavor of the week stuff. Opinions basically. Patterns like microservices are bad or good depending on opinion, but theory is always correct. Theory gets less bang for the buck but it's always what many programmers especially self taught ones are missing. Theory is so hard that it will be hard to see applicability until you're a more seasoned programmer. Many seasoned programmers get by without ever knowing theory. But you will be a better programmer if you know it. If I were to recommend one theory to study it would be category theory. If there was any true axiomatic theory for design patterns or how to design programs... categories are it. The study of morphisms is the study of the simplest form of a compose-able module. Knowing this theory you will begin to understand why Some design patterns don't work and why it's sometimes hard to reuse patterns in code that was that not properly designed. Theory doesn't answer all questions but for the questions it does answer you will get a definitive answer and not an opinionated one.
  - sn9 7y ago
    The canonical undergrad CS theory text is Sipser's Introduction to the Theory of Computation. The prereqs are a comfort with discrete mathematics and proofs and a course in algorithms and their analysis.
gamegod 7y ago
IDEs - A proper work-grade IDE like Visual Studio will have some tools built into it that will save you tons of time. Features like "Edit & Continue" have saved my bacon many times by making intractably difficult bugs in algorithms much easier to understand because you can experiment with your code on the fly. There's also some more common features in most IDEs like being able to jump between symbols that saves a lot of time day-to-day. If I'm interviewing you and you say you don't like using an IDE or a debugger, that speaks to your work experience, your productivity, your self awareness about your productivity, and really puts an upper limit on the difficulty of the problems you've had to solve.
- skummetmaelk 7y ago
  I find that it is usually the opposite. Devs who don't use big IDEs like visual studio are much more likely to know why things fail in a build pipeline etc. If you can only build a project by bumbling through menus and pressing a big green button at the end, that is worrying. If you can only debug by immediately jumping into the debugger and single stepping that is also worrying. Devs who reach for the tools appropriate to a given situation inspire much more confidence in their abilities. Knowing the time saving features of your editor is a huge boost. However, the old editors have much more of these ;)
  - snazz 7y ago
    IDEs aren’t useless, you just don’t want them to be a crutch. For this reason, I usually recommend new developers use Nano or a notepad-esque editor until they understand why they might want vi keybindings, then use Vim until they understand why they might want an IDE or something like Emacs. Starting with the IDE hides layers and layers of both junk and useful tools, while experienced developers know which layer of the stack to work on at which time.
- clarry 7y ago
  > If I'm interviewing you and you say you don't like using an IDE or a debugger, that speaks to your work experience, your productivity, your self awareness about your productivity, and really puts an upper limit on the difficulty of the problems you've had to solve. Or you have surpassed the limit on the difficulty of the problems your debugger & IDE can help with.
augbog 7y ago
Honestly I still find a lot of engineers don't know git properly. Like they know enough to commit and push but that's about it. It really helps to understand everything git has to offer.
- 333c 7y ago
  The three most important basic git operations to know (in my opinion) git checkout -b git log git rebase -i
  - jimpudar 7y ago
    I also find git blame extremely useful, along with code exploration tools like DeepGit [0] [0] https://www.syntevo.com/deepgit/ https://www.syntevo.com/deepgit/
  - dwoot 7y ago
    I'll have to add `git reflog` for instances where you've felt that you've completely screwed up something as one can always move back to a previous state. I think this is essential. A useful one that I'll add is what I call the sword command: `git log -S<word>` This one allows one to list commits that contain a particular change. This has been useful in tracking down old changes
  - btilly 7y ago
    I strongly prefer git merge over git rebase. Using rebase results in a cleaner history and simplified workflow in many cases. However it also means that when you have a disaster, it can be truly unrecoverable. I hope you have an old backup because you told your source control system to scramble its history, and you don't have any good way to back it out later. For those who don't know what I mean, the funny commit ids that git produces are a hash signifying the current state of the repository AND the complete history of how you got there. Every time you rebase you take the other repository, and its history, and then replay your new commits as happening now, one after the other. Now suppose that you rebased off of a repository. Then the repository is rebased by someone else. Now there is no way to merge your code back except to --force it. And that means that if your codebase is messed up, you're now screwed up with history screwed up and no good way to sort it out. That result is impossible if you're using a merge based result. The cost is, though, that the history is accurately complicated. And the existence of a complex history is a huge problem for useful tools like git bisect.
jacksnipe 7y ago
Not nearly as big a deal as some of the other tools and techniques being mentioned, but tmux/screen are lifesavers when you need them.
niklasjansson 7y ago
Excel. Seriously. For more things than you imagine Let's say you want to make a property assignment for some class; this.a=x.a this.b=x.b ... While you probably would want to do this some other way to start with, and while you of course can solve it using some emacs wizardry, I can whip that up in Excel using some formulas in a matter of a minute. Moreover, I can keep adding to it when I realize something was missing. I can make a diff, join or union of the results of two queries from different databases (or even database engines). Not to mention calculation and design mockups.
- hsitz 7y ago
  I'm not following. What good is it to have equivalent of "this.a=x.a this.b=x.b ..." in Excel, when what I need is to have this in code in my editor? Are you saying you use Excel to create the code, then copy/paste it into your editor? Or what?
  - JonathonW 7y ago
    I think the parent's using Excel essentially to do repeated template expansion: e.g. for a given set of member variable names ([a, b, c...]), give me the assignment statements I'd need to use those in a constructor. Which I could do pretty trivially in Excel... but could also do trivially in about two lines of Python: vars = ["a", "b", "c"] statements = ["this.{0} = x.{0}".format(var) for var in vars] print(statements) Spreadsheets can be an incredible tool-- as an interactive environment that allows non-programmers to express domain knowledge and quickly automate parts of their workflow, they're really unparalleled. But this isn't a very good example of something that Excel is particularly well-suited for.
    - UnFleshedOne 7y ago
      Those can often be done in a decent text editor with regexp-like search and replace.
      - kragen 7y ago
mnemotronic 7y ago
* Regular expressions. The most valuable "not a language" that I know. * SQL. For me, it was a trip coming from a procedural language background. I kept trying to figure out how to do loops. * Command line - DOS and unix * Batch languages for those. * HTML / CSS / Java (please don't kill me) Script and the DOM
- JamesBarney 7y ago
  Second regular expressions. I use it all the time to edit large bundles of code.
hsitz 7y ago
A new version of "The Pragmatic Programmer" recently came out. [EDIT: not available yet, only preorder at amazon, beta version available at pragprog.com.] That book is all about tools and methods that a self-taught programmer should look into: https://www.amazon.com/Pragmatic-Programmer-journey-mastery-Anniversary/dp/0135957052 https://www.amazon.com/Pragmatic-Programmer-journey-mastery-...
- colomon 7y ago
  For me, that Amazon page is listing it as a pre-order, without any release date. And all the other versions (Kindle, Paperback) are the 1st edition instead of the 2nd. Very frustrating, as I considered the first edition to be essential and upon reading your comment, instantly went to purchase the 2nd edition. Edited to add: Found a date, Amazon is listing it as October 21, 2019.
  - hsitz 7y ago
    Sorry, I thought I'd read a review of it already, so just didn't look closely to see it wasn't available yet. It looks like you can get a DRM-free beta version of the ebook on their website, with free upgrades to published version once it's finalized: https://pragprog.com/book/tpp20/the-pragmatic-programmer-20th-anniversary-edition https://pragprog.com/book/tpp20/the-pragmatic-programmer-20t...
  - deleted 7y ago
    [deleted]
SonOfLilit 7y ago
More important than any specific technique or tool is that you get an experienced mentor to look at your work once in a while and point out the obvious things relevant to the specific project that you don't know you don't know. Seriously, you'll probably save a few months of work in the first ten minutes of talking to a senior developer about your hobby project. Talk to someone with many years of experience, though, otherwise you'll most likely get sent on a quest for beauty of implementation that has nothing to do with the goals you're trying to achieve. Sadly, that is a lesson that takes a long time to internalize.
- SonOfLilit 7y ago
  (Learning all the things recommended in this thread would take a few years, and maybe half of them would be useful given you know them, a lot fewer useful enough to justify the price of learning them)
cameronbrown 7y ago
I'm surprised nobody's mentioned spreadsheets - specifically Google Sheets or something scriptable and hosted. Recently I've built up a small system which sucks in data from a few places (fitness, task management, calendar,..) and analyses it against several goals I've set. This means I can see how I'm progressing towards what I want without even touching it anymore. They're a really nice UI for bootstraping projects, and even some small databases. A current project of mine for a client uses sheets as a backing store for an email collection list. Since there's only a few hundred rows, it makes sense at this scale and works really well for non-technical users.
- Forge36 7y ago
  Similarly Microsoft Access. I learned Access for my first job (100% of my job was moving tasks into Access, 3 years later I helped some interns transition to a SQL server+web application). 10 years later I'm still using Access for some aspects of my new job. The ability to quickly prototype something and receive almost immediate value is vastly underestimated (I know of one company which runs all of their tasks out of Access).
  - cameronbrown 7y ago
    I suppose what I've described is just a little more of an unstructured version of this. Mixing data and computation in such a way is definitely a tech debt tradeoff though.
- thecupisblue 7y ago
  Excel & Google Sheets are probably the best enterprise software development platform ever made.
theonemind 7y ago
* Parser generator tools like ANTLR https://www.antlr.org/ https://www.antlr.org/, or even lex and yacc, useful for parsing languages/config files and probably generating a better/more robust parser than you'd cobble together by hand * Dynamic programming https://en.wikipedia.org/wiki/ https://en.wikipedia.org/wiki/ -- great for relatively-quickly (in computation time) coming up with good-enough solutions to some hard (like NP hard) recursive problem that would take forever (almost literally) to technically find the absolute best solution to, but not something you'll use every day.
- sgillen 7y ago
  I was under the impression DP was usually used for getting the exact answer to certain classes of problems faster than exhaustive search. I usually see approximations compared to the ground truth dynamic programming answer. Do you see the reverse in your work?
- oneturkmen 7y ago
  That's weird. DP, though better than naive brute force, is still very slow since it looks only for an exact, optimal solution.
wpietri 7y ago
For me a lot of programming involves some amount of data wrangling. E.g., getting input data ready. Or generating and understanding results of some technical experiment. I recently came across VisiData and adore it. It has a steep learning curve, but I've found it very much worth it: http://visidata.org/ http://visidata.org/
nickjj 7y ago
Shell scripting for processing text. You can often get so much done with so little code and effort. Also on a semi-related note, I think as a self taught programmer, it's easy to get stuck on things that seem cool but are just procrastination enablers (I know, I've been guilty of it for 20 years). Like, if you're about to start a new project and you want to flesh out what it's about, you really don't need to spend 5 hours researching which mind map tool to use. Just open a text document and start writing, or get a piece of paper and a pen. It won't even take that long. I spent about 1.5 hours the other day planning a substantially sized web app. All I did was open a text file and type what came into my head. For fun I decided to record the whole process too[0]. I wish more people recorded their process for things like that because I find the journey more interesting than the destination most of the time. Like your journey of eventually finding message queues must have been quite fun and you probably learned a ton (after all, it lead you to message queues, so it was certainly time well spent). [0]: https://nickjanetakis.com/blog/live-demo-of-planning-a-real-world-web-application-from-scratch https://nickjanetakis.com/blog/live-demo-of-planning-a-real-...
- Draiken 7y ago
  This 1000x. I put of just getting by with shell script for some years and when I finally decided to get deeper into it, it's magical. A good series of piped commands with tools available basically everywhere can solve problems you had no idea could be so simple to solve.
  - 0x262d 7y ago
    have any good resources for this?
    - fancyfish 7y ago
      Perhaps these are a good start: awk: - GNU Awk User Guide - https://www.gnu.org/software/gawk/manual/gawk.html#Getting-Started https://www.gnu.org/software/gawk/manual/gawk.html#Getting-S... - Grymoire guide - http://www.grymoire.com/Unix/Awk.html http://www.grymoire.com/Unix/Awk.html sed: - Grymoire guide - https://www.grymoire.com/Unix/Sed.html https://www.grymoire.com/Unix/Sed.html - Official docs - http://sed.sourceforge.net/#docs http://sed.sourceforge.net/#docs
namank 7y ago
What you want to do is ponder over how common functionality around you must work - if you can't come up with a reasonable solution, it's time to google it and learn it. If you can come up with a reasonable solution, google to check if you're correct. This is the most pragmatic way I've found to learn and stay in touch.
LeonB 7y ago
So many of the problems are “people problems” so in addition to the excellent technical suggestions in this thread I’d add books like “Thanks for the Feedback”, “How to Win Friends and Influence People”, “Getting to Yes”
- yyx 7y ago
  "start with no"
Const-me 7y ago
I don’t think there’re many ridiculously useful things that are universally useful. If some stuff is domain specific or have much theory involved doesn’t mean it’s not considered a fundamental concept in some areas of work. If some other stuff was very useful for you doesn’t mean it’s universally applicable. For example, if you had worked on a high-frequency trading bot, I don’t think you would have used neither Python, nor ZMQ or other general-purpose messaging middleware, nor even OS-provided TCP/IP stack — they all cause too much latency. I’ve been programming for living since 2000, worked in a lot of different stuff from web dev and enterprise to videogames, embedded, robotics and GPGPU. Yet I can name many huge areas which I hadn’t seen close enough, or at all, along with libraries and tools used by people working there. Every time I start working in a new area, or when I resume working in an area after a long (years) pause, I read a lot of relevant stuff. Continuous learning is the key to stay good, IMO.
RangerScience 7y ago
At the job I'm at, I've picked up three tools either for the first time, or in a very new way: 1. Makefiles. See @aequitas' comment for more. 2. Terraform. Seriously, just using this tool taught me [a lot of] devops. It's fantastic! 3. Docker (as a tool!) I'm going to go into the third one a bit - I feel like Docker is mostly thought of as useful for deploying things to the 'net (kubernetes, ECS, etc), but I think it's also amazing for local development and build pipelines. I actually have no idea one way or the other how much other people use it this way as well, so maybe it's just me that's finding it unexpectedly awesome for this. Put together the right CLI command + Dockerfile, and you can hand someone a repo and they can launch a complete, reliable development environment in a single command without any other system prep. No more worrying about which dependencies need to be installed in what way; it's like `rvm` + `bundle exec` but for EVERYTHING. No more dealing with whatever custom system modifications someone has going on. `git clone`, `make dev` move on with life. And then you can also have Dockerfiles that are specifically for producing your build artifacts, and then completely ignore the container for execution. This is how I'm using both AWS EMR and Lambda.
- akx 7y ago
  I'm also occasionally using Docker t generate build artifacts (so +1 for that) - how do you pull the built blob out of the image? I've used `docker exec` plus `docker cp`, but it feels a little clunky.
  - victornomad 7y ago
    You can just mount a volume and write the blob into it, very easy and convinient!
    - akx 7y ago
      Ooh, right, you can mount in `docker build`? That'd solve it, thanks!
      - RangerScience 7y ago
pankajk1 7y ago
Here is a list of technologies/tools that I have found quite enabling (in addition to what has already been covered): 1. Earlier use of VMs and now Docker containers -- It was around 2004 that I got introduced to VMware tools to create, configure and run VMs. Life was never the same after that. No more fretting about installing pre-beta software for the fear of hosing my perfectly working system. There was a time when I had multiple servers running VMWare hypervisor, each running multiple systems. Then around 2014 I switched to using Docker containers for similar purposes and haven't had a need to use VMs. 2. Jupyter notebooks/pandas/plotly -- Can't imagine how one can explore data without this. 3. SQLite -- Perfect for writing unit tests for code that deals with SQL databases.
Khelavaster 7y ago
Visual Studio, especially its visual UI designer, build & reference management, and scaffolded website development (ASP.Net MVC with Entity Framework from scaffolding). Night and freaking day. Also, WCF for all your inter-computer communication needs. I wouldn't be developing software today if I weren't introduced to low-trouble application development with Visual Studio
Khelavaster 7y ago
It's absolutely critical to learn object-oriented programming in a mature language like C#. (Java's OO model is slightly broken, as well as fundamentally underdeveloped; it's better tackled after learning a correct, functional object0oriented language like C#.)
lowken10 7y ago
Anki!!!
sycdan 7y ago
I use ack (https://beyondgrep.com/ https://beyondgrep.com/) many times per day, and it has saved me a great deal of time. That said, if you use IDEs (I don't), it may be somewhat redundant.
azhenley 7y ago
Usability engineering: https://en.wikipedia.org/wiki/Usability_engineering https://en.wikipedia.org/wiki/Usability_engineering
bradford 7y ago
I've extensively used Diff tools as an analytical aid across many different domains. It has uses far beyond code change tracking! Specifically, I like vim's interactive diff'ing capabilities (although any interactive diff tool with sufficiently powerful text-editing capabilities should suffice). So much of the troubleshooting that we do in programming is asking "this thing used to work, what changed?". Don't rely on your eyes to find the changes, let the computer do the work for you. The ability to load up two different log files in a diff session, regex-substitute the noisy portions away (dates, process/thread id), and view the key functional changes really helps me go from noise to signal in an optimal way.
- reificator 7y ago
  You touched on it briefly but I'd like to highlight regular expressions in day to day editing and log delving. It's a massive time saver in my experience at least. Coworkers often come to me to help them write a quick regex for something, or to have me double check their work. If you need a playground to get comfortable, https://regex101.com/ https://regex101.com/ is a great resource. Dump some examples you'd like to match and some you don't in the bottom section, and try to write a regex that matches in the top. It will dynamically match as you type, and the right side shows a token by token breakdown of what your regex does.
6nf 7y ago
SQL Stored procedures and triggers. It's some of the most useful stuff you may not have considered if you're just a hobbyist. It's not for every project but if you do need a proper database for some reason, there's often great ways to use stored procs or database triggers that greatly simplifies your system.
i_feel_great 7y ago
Tools you can use to recover from the loss or destruction of your laptop/desktop as fast as possible. Or having to hand over your password to the authorities. This involves compressing all of the state and data and removing and/or sending it somewhere, and then being able to recover it quickly. And it must be simple, robust and automated.
glangdale 7y ago
Satisfiability Modulo Theory (SMT) and a nice friendly implementation like Z3. When you have a difficult algorithmic problem and can't be bothered to dream up a good solution, why not try chucking it into a solver and see if a computer can solve it for you? Surprising what it can find - not just Sudoku puzzles and "Mr Green lives next door to Mr Black, the baker lives across the street to the butcher, ... " type stuff either.
- travisjungroth 7y ago
  Here’s a great talk by Raymond Hettinger on those things and more: https://m.youtube.com/watch?v=_GP9OpZPUYc https://m.youtube.com/watch?v=_GP9OpZPUYc
keyle 7y ago
The design of everyday things. It's a design book that will make you a better programmer too.
new4thaccount 7y ago
SQL (even if just SQLite) as databases open up a lot of power. Vim or Emacs for powerful text editing. A low level language. Sometimes Python doesn't cut it, or it is pretty suboptimal. If you're writing a trading bot, is speed of execution not important? Operating System knowledge can be helpful at times. I bought one of the No Starch books "How Linux Works" and it is very helpful. The command line and by that I guess you should know the common Linux commands (cat, grep, sort, uniq, head, tail, ls, top) if you use Linux and how to chain them together via pipes. To give some context, I can write one command which would require 8 lines of Python (saves you valuable time). If you use Windows, learn enough Powershell to be comfortable with it. On occasion I'll use Powershell over Python even though it is dirt slow for reading files.
- afarrell 7y ago
  I thought this course did a good job at getting SQL to stick in my brain, largely due to the relational algebra section. https://lagunita.stanford.edu/courses/DB/2014/SelfPaced/about https://lagunita.stanford.edu/courses/DB/2014/SelfPaced/abou...
  - new4thaccount 7y ago
    The W3Schools site is what helped me https://www.w3schools.com/sql/ https://www.w3schools.com/sql/ It has a database you can query and add to and what not.
  - artimaeis 7y ago
    Firmly agree -- I'm pretty sure I've said it elsewhere, this was a transformational course for me. I got my start in product support where a lot of our problems could be solved in SQL. We were encouraged to learn it, but most people were happy with the most basic syntax for DML. I worked through the entirety of Dr. Widom's course and it gave me the fundamental data understanding to be a real value to the teams I've worked for, and I used that opportunity to transition into being a full time developer.
HocusLocus 7y ago
Perl one-liners, the ones that overflow to several lines of code in your terminal.
funkjunky 7y ago
When I went from amateur Python programmer to Google Cloud developer support, I remember being completely blown away by technology and design patterns I've never heard of that goes into modern web/enterprise architecture. I had to learn it all the hard way, but these days there are great free (or mostly free) courses you can take to learn this stuff. For Google, check out the study guides for their certifications, specifically Cloud Architect (basic overview), Cloud Developer, and Data Engineer. https://cloud.google.com/certification/ https://cloud.google.com/certification/ Be sure to follow along with the recommended Coursera and Qwiklabs tutorials and do the exercises. You'll learn about all kinds of neat stuff, like scalable application design, container technology, monitoring+metrics, various types of database technologies, data pipelines (including Pub Sub messaging), SRE best practices, networking+security, and machine learning. I currently work on AWS, and don't find it a good starting point for diving in to these things quickly, but most companies use it so it wouldn't hurt to learn I guess. I still recommend GCP over AWS to start with, as their technology is far more interesting and focused, and quicker/easier to work with.
hammerbrostime 7y ago
One minute ago I discovered that HN comment threads are collapsible by tapping/clicking the "[-]" on the right side of a comment header. I think that could fall under the class of non-obvious tools (maybe not for a hobbyist, but as a long time HN reader I'm shocked I never noticed it before)
vast 7y ago
Automated deployment. Even for smaller projects. Ansible for example is a python product. Online schema change tools if you use RDBMS in production.
Zeroverse 7y ago
Debuggers, probably. I've always had so much admiration for other programmers who have taken time to learn their debugger and just jump into breakpoint-land when something happens. It's how we're meant to think!
GuB-42 7y ago
Not one in particular besides the obvious. If it is really fundamental, it is not overlooked. Every programmer has their favorite tools. Some will use debuggers, others will prefer logging. Some will use class diagrams, others will grep. Some will use IDEs and GUIs, others will use text editors and shells. Same thing for programming languages, techniques, libraries, frameworks, etc... So you are very likely to get a lot of answers. It seems that you got an epiphany when you learned about message passing. All programmers had similar experience as they discovered the one thing they really needed. In reality it depends on the project. Great programmers simply have a lot of experience with many, many things. My suggestion: continue what you are doing. Try new things, don't blindly follow other people lists. In the process that led you towards that messaging library, you learned named pipes, unix sockets and shared memories, all useful, and maybe they will be essential to your next project. Had someone else served you that library on a platter, you would have lacked that insight.
AaronFriel 7y ago
Devops; being able to stand up your entire stack in an automated way. Some programmers dismiss this as "yaml/bash engineering", and that's true, but it also challenges all of your assumptions in your stack. If you can't stand up a complete duplicate programmatically, you almost certainly have implicit assumptions that you haven't verified, which will make it much harder to recover from disaster or scale. Put another way, devops is a mix of declarative programming (YAML) and writing idempotent, imperative code in the presence of large side effects (bash). Learning to handle both is very educational.
mooreds 7y ago
I would say studying any of the AWS paas offerings. Not saying you have to use them, but they cover a large segment of the system component space. They can help you answer questions like "When would I use an in memory cache vs a rdbms vs a key value store?"
AaronFriel 7y ago
Value/message/actor/event oriented programming like you mentioned is useful for building distributed systems. I am a huge fan of going a step further and learning this model: Imperative shell, functional core. The external shell of code in a project is responsible for network connections, console IO, etc. But the internal guts of a program should be largely functional, that is, instead of mutating (changing) values, consider returning different forms of the same value. Decisions (branches, logic) are made at one level, data dependencies at another. The talk Boundaries by Gary Bernhardt describes this model in detail: https://www.destroyallsoftware.com/talks/boundaries https://www.destroyallsoftware.com/talks/boundaries
- btown 7y ago
  I'm also a fan of "imperative shell, functional core, imperative implementation of that core," where within a function that you promise to have no side effects or very specific side effects (say, a component's render function, or a transformation from one complex data representation to another), you should still feel free to use loops, imperative-style control flow, even network requests, etc. I find this puts folks used to "imperative everything" at ease, while still maintaining almost all of the benefits of functional-core provided that execution is scheduled properly.
  - lenticular 7y ago
    It's really easy to do this with pure functional programming in impure languages like Scala. You can use arbitrary impure code within lazy IO values. Outside, the functional purity makes reasoning easy. If, as recommended, there is only a single point in your program where the IO values are actually executed, then the execution order can be reasoned about statically.
gitgud 7y ago
Design patterns were enlightening to me. The idea of expressing concepts in software design which can be reused was what got me out of the stone age of programming. You can describe Pub/Sub or the Builder Pattern and quickly implement it into a project and know it's advantages and limitations, definitely worth checking out if you haven't already. http://wiki.c2.com/?CategoryPattern http://wiki.c2.com/?CategoryPattern
codr7 7y ago
Macros, interpreters and embedded languages [0]. [0] https://github.com/codr7/g-fu/tree/master/v1 https://github.com/codr7/g-fu/tree/master/v1
greenyouse 7y ago
This might not be for a hobbyist since the setup is difficult but validating your frontend work with automation is very interesting. I didn't realize for a long time that I could use visual regression tools and Selenium at the end of development to catch bugs. I'm surprised that there aren't more tools in this space targeted for developers. Selenium Grid with WebdriverIO is incredibly helpful. If you put together a visual regression system for validating your daily work you could save some serious time.
eranation 7y ago
Not tools but more of a mindset: Application Security, OWASP top 10, Web Security. Learn what is XSS, CSRF, what is a CSP, why HSTS exists, why CORS allowing * can be dangerous. Also what is OAuth, OIDC, SAML, etc. How to store passwords securely, how to add security to your devops cycle. Having a good security mindset can be a great asset to any team.
ww520 7y ago
Learn Emacs. Stick with it to get comfortable. Text editing is one transferable skill that you can reuse again and reuse in different projects and on different languages. Org-mode in Emacs. My poor man's project management for a number of projects consists of a todo-<project>.org file listing all the planned features, the pending TODO items, the DONE items for the current working release, and release notes describing features and changes for each of the released version. In one place, I have the future features, immediate todo items, current completed item, and the history of all past releases in an org file, making things simple to access and manage. For theoretical stuff, learn about transaction.
jtolmar 7y ago
1. Profiler. There's a standard tool that tells you what part of your code is slow. Over half the time it'll find something dumb and easy to fix instead of whatever you expected. 2. SQL / relational database schemas. Persistence opens up a lot of capabilities. And databases themselves are very well-optimized; if you do any nontrivial data manipulation it's likely that whatever the query planner comes up with will be faster than your first idea of how to do it by hand. 3. Graph searches. An awful lot of problems can be solved by knowing how to turn problem into a graph search. Make sure not to fall into the trap of thinking a graph search is limited to paths through space - you can solve problems like "get through this dungeon with keys and doors" by adding duplicate nodes for the different states. 4. Sequential Bayesian Filters. Are almost as useful as graphs, but aren't in a standard CS curriculum so you'll look like a wizard. These solve the problem of "I want to know a thing and I know how it changes over time, but I only can get rough estimates for its current state." Kalman Filters are simple and give great results when applicable. Particle Filters have lower quality but are applicable to more problems and dirt simple to code.
- enangel 7y ago
  > Particle Filters have lower quality but are applicable to more problems and dirt simple to code. Particle Filters have lower quality than Kalman Filters?
- rassibassi 7y ago
  Support for 4! Yet, my understanding is that particle filters are superior but computational more demanding. For nonlinear problems, the extended Kalman Filter linearizes the task, whereas particle filters don't and work with many point estimates instead. I loved this book: https://users.aalto.fi/~ssarkka/pub/cup_book_online_20131111.pdf https://users.aalto.fi/~ssarkka/pub/cup_book_online_20131111... and also Thomas Schoen group does great work on Sequential Monte Carlo (SMC), MCMC for sequential data :) http://user.it.uu.se/~thosc112/index.html http://user.it.uu.se/~thosc112/index.html They are also building a probabilistic programming language for sequential data! https://github.com/lawmurray/Birch https://github.com/lawmurray/Birch
  - 7y ago
achenatx 7y ago
There are so many specialized areas that are useful, like common design patterns, basic data structures, various algorithms, transactions, multithreading, parallel processing, etc. Too many things to list, but I agree with a poster below that suggested looking at the course offerings for a computer science degree. You might be able to find outlines of the courses to get specific topics. On a day to day basis, code coverage is something that more people should use when writing unit tests to ensure they have tests that execute all their code.
quickthrower2 7y ago
Tools that have changed how I think about things, and yes some are hyped things you have probably heard of. But the opposite effect of people ignoring it because it's too hyped might come into play: Docker - figuring out what you really need to get something running and having a reproducible way of doing it. I did a blog post on it: https://stackabuse.com/how-docker-can-make-your-life-easier-as-a-developer/ https://stackabuse.com/how-docker-can-make-your-life-easier-... Node JS - much derided, but nothing compares in terms of the speed you can quickly hack up a tool in Node JS and the ecosystem you have access to. I don't use it as production server, but just as a quick way to hack up tools. Pandoc - being using this tool to convert markdown to PDF, and it does a nice job. Uses Latex & friends so you can find a nice template somewhere and base off that. Markdown - I love using markdown formats. Especially with the tooling, things like Pandoc and Github are enough to justify learning a bit of MD. Touchtyping - Makes typing a bit nicer and a bit faster. I could type without looking at the keyboard before, but now I don't lose my hands, just feel the bumps!
jasonhansel 7y ago
Shell scripting (trust me, this is way more useful than you might think).
alphaoide 7y ago
This was recommended in a blog post I read. I haven't read it myself but the table of content looks promising https://www.educative.io/collection/5668639101419520/5649050225344512 https://www.educative.io/collection/5668639101419520/5649050... Designing a URL Shortening service like TinyURL Designing Pastebin Designing Instagram Designing Dropbox ... Key Characteristics of Distributed Systems Load Balancing Caching Sharding or Data Partitioning ...
ilaksh 7y ago
The premise that self-taught programmers necessarily have core holes in their knowledge and skills that would have been filled if they had a CS degree is entirely false. Start with the example you gave of messaging middleware. There are many BS CS curricula that do not address this at all. Also he mentioned that he had already learned about named pipes on his own. For many applications, named pipes could be a perfectly valid alternative to some external message queue system. Looking at the items submitted, the vast majority are core skills that would necessarily be picked up by people who need to work with them. The idea that someone would not know about Makefiles or debugging or profiling or SQL just because they were a hobbyist or self-taught is ludicrous. If you are serious about C programming, whether it's a job or your hobby, you are going to learn about Makefiles. Likewise anyone seriously working on a data-centric application is likely to become well-versed in some database technology, up until a few years ago that would have automatically been relational. And one other thing. Some of the most important skills in programming are in the domain of software engineering. Software engineering is very poorly addressed by many BS CS programs. So again, whether they have good SE skills is often not going to be determined by whether they have a BS degree or not. It's not even necessarily determined by whether they are working in a professional environment. It's mainly going to be a factor of their motivation to self learn and above all practical experience.
- djhaskin987 7y ago
  My experience has been quite different. True that some of the most technically skilled programmers I know of had no degree, but the polished ones, the ones I find easier to work with, tend to have one. Further it's pretty easy for me to tell if a person's degree was a CS degree or not just by talking to someone about the problems they have and how such problems might be solved with code. That's not to say it's required; some of the best professionals I know have non-CS degrees (one in fine art -- painting) or no degree. But if you're still young, I submit that a CS degree is totally worth your time.
  - ilaksh 7y ago
    The argument that I was making was clearly stated at the beginning of my comment. It is significantly different from the supposed argument that you seem to be refuting. Notice how I did not say that "the developers that certain degree holders find to be most 'polished' and easiest to work with will on average not have a degree". Notice how I also did not say that a CS degree was not worth people's time.
jdavis703 7y ago
I’m not sure if I count as self taught or not since I first learned programming outside of formal education, only later getting a degree in IT (for which I really didn’t learn anything new except graph theory). But we never talked about queues in school, in fact I only learned about them when I started working professionally. If you want to be exposed to all kinds of interesting problems, I would suggest working at some mid-stage startup (i.e. around the B or C rounds). They’ll be starting to get in good people who are fixing up the mess the early employees made* and can help you learn why certain patterns are anti-patterns and how to fix them. At this stage they’re still small enough that if you pay attention, you can just learn by paying attention to what everyone else is working on. * Before you downvote me I’ve contracted for pre-seed startups, am currently working at a seed stage startup and have been at companies all the way from a series A to a series E. So yeah I’ve been the one making a mess (because of the whole minimum part of MVP) and cleaning up said mess.
whalesalad 7y ago
Low level unix tools. Learning how to read a file, tail a file as it grows (like a logfile), or look at just the beginning, or just the number of lines in the file, or merge a bunch of files into one, etc... How to get a file from one machine to another with scp. How to use more advanced features of SSH like agents, forwarding, your config file, SOCKS proxying, etc..., How to debug system issues, find where config files should be, find out why your app wont compile. Learn how to install code from source, using configure and make. Learn how to operate your own basic network services like HTTP servers, mail servers, local file sharing with NFS or SMB, etc...
smoe 7y ago
Monitoring From the beginning of my career in web development I worked in companies that had good to great monitoring capabilities in place to log and measure performance, resource usage, availability, errors, etc. in all parts of the system. so I took it as something self-evident that you would add that first thing, before deploying a new project. Only recently as I started mentoring people in other companies did I realize that many developers or not aware of what exists in this space. In two cases I got asked for help to deal with performance issues, and in neither case did they have any idea what was causing them, because they had pretty much no tooling to tracking it down, so they resorted to speculation. We installed a application monitoring solutions and they were able to fix the problem in no time once it was identifiable.
- alphaoide 7y ago
  Can you recommend books/tools? Thanks!
  - smoe 7y ago
    This chapter of the Googles SRE book gives you a good overview. https://landing.google.com/sre/sre-book/chapters/monitoring-distributed-systems/ https://landing.google.com/sre/sre-book/chapters/monitoring-... But I think it is less abut what specific tool to use, but more to just get started with one and learn how to understand your production system behavior and dependencies from metrics and graphs. An easy way to get going for monitoring an application is using a hosted solution like newrelic.com Their free version should be sufficient for a very long time. If you want to run it yourself there are opensource solutions like prometheus.io or riemann.io among many others.
ajflores1604 7y ago
tldr: redis Hey man, I went down an eerily similar path as you. Seld taught and building a trading system. Wading thru sockets and pipes before discovering zmq and having my whole paradigm for programming completely shifted. Absolutely love zmq, and enjoy thinking of new projects to use with it. More towards your actual question, I don't think it's as ground shifting of a discovery as zmq, but Redis has also helped me a lot with the trading bot. I use Interactive Brokers, and how their api examples are set up, I haven't figured out how to design my system using only message requests like you see in micro-service style programs. Also when requesting a price for any given contract, I don't like the idea of having to request it once I need the price and then waiting on the network for the brokerage to receive my request and then the price to come back to me. By that point the price or whatever data you're requesting can get stale depending on the speed and timefrime you're wanting to play. So what I did was just request ahead of time streaming data for contracts I'm interested in, and I write that continuously to a localhost redis server. And then whenever any of my bots need information, they check to pull the latest value from that redis server first before going out to the brokerage platform to request the data directly. Basically cuts the full round trip time in half if it can find the data locally first. I believe this is a similar programming paradigm as using redux if you have experience with that, but I've never personally used redux so don't quote me on that.
- nathanasmith 7y ago
  Absolutely agreed. Redis is a godsend for small time traders like us. I've scaled up to where I'm trading on about 20 crypto exchanges in addition to dabbling in stocks and prior to redis I was using flat json files to store all my data so every time a new price came over the wire the bot would read the file, parse the json, append the price then write it all back out. That was quick and easy to put together and it worked great at first but later when I had all the exchanges going even with an 8700k and Optane drive the computer would just fall over when volume got high. Got sick of that, converted it all over to redis with rejson, added another 32 gigs of RAM and problem is utterly solved. Redis' built-in queue has been extremely useful too as a job system.
xs83 7y ago
DEBUGGING! And I don't just mean echo'ing out statements to check the values, I mean an actual attached debugger to your code. GDB for C / compiled languages XDEBUG for PHP etc The actual act of stepping through your code and looking at the values, datatypes and their transitions massively increases your productivity if something is tricky!
lixtra 7y ago
For learning what’s going on in compiled programs strace can be very helpful. https://www.tecmint.com/strace-commands-for-troubleshooting-and-debugging-linux/ https://www.tecmint.com/strace-commands-for-troubleshooting-...
croo 7y ago
Did you know there are document storage/management systems out there? Its useful when one needs to save a lot of documents(duh) or other too-large-to-save-in-rdbms type of content with metadata. Examples are FileNet or Alfresco. Did you know there is a protocol called CMIS [1] which you can use to query content from these systems? It's similar to a subset of SQL. [1] https://en.wikipedia.org/wiki/Content_Management_Interoperability_Services https://en.wikipedia.org/wiki/Content_Management_Interoperab...
contingencies 7y ago
Anything that automatically generates pictures from text: mscgen, imagemagick, graphviz, SVG, povray, gnuplot, R, etc.
nprateem 7y ago
I used to read High Scalability [1] which has write ups of real-life architectures of lots of the largest web sites. I found it great for learning about different architectures. I'd definintely recommend it as a way to find out about thing like messaging queues, asynchronous vs synchronous processing, etc. On a similar note but more theoretical, my life changed when I learned about design patterns. Martin Fowler's books/site are a good place to start [2]. [1] http://highscalability.com http://highscalability.com [2] https://www.martinfowler.com/eaaCatalog/index.html https://www.martinfowler.com/eaaCatalog/index.html
olingern 7y ago
Just a few that I've found liberating over the past few years: 1. In memory caches for small projects / one-offs. Redis and memcache are sometimes not needed. Wrapping your cache in an class whose storage is dependency injected is a nice way to keep moving along. --- 2. grep Not much to say here other than it's a great search tool. Admittedly, I'm still learning working on wielding it even better --- 3. Piping output: [CMD] > myoutput.txt Lifesaver for large output.
jondubois 7y ago
Corporations have been pushing hard on marketing their vendor-lock-in as-a-service solutions so you don't hear about simple free open source tools anymore; all developer channels are saturated with corporate marketing. So much so that you don't even realize that it's marketing.
shiloa 7y ago
An IT automation framework like Ansible (my favorite), Chef, Puppet, Salt. Highly recommended if your work includes doing repetitive remote server tasks or any sort of infrastructure / devops stuff.
fa 7y ago
Is this trading bot you’re writing available online?
geirman 7y ago
Learn to use Proxy Servers for Troubleshooting Whenever your web app, mobile app, etc depends on a network response, Charles Proxy (https://www.charlesproxy.com/ https://www.charlesproxy.com/) can be super helpful. It sits in between and captures all the HTTP request/responses and allows you to manipulate them. So, you could capture an API response (or request), manipulate it, then let it continue. It also let's you interrogate the requests easily.
anonu 7y ago
> Googling, I encountered something I hadn't heard of called a message queue. Former high frequency trader here. Messaging middleware is a god-send for distributed systems. It used to be quite commonplace in the late 1990s/early 2000s for creating trading systems. By mid-to-late 2000s - and especially post GFC (great financial crisis), volatility lowered, technology improved and middleware systems were never again put in the critical path. The important measure here is "tick-to-trade" - from the moment a market-data message comes off the wire to the moment you send an action to buy/sell/cancel back on the wire. A middleware system just slows this down considerably. As a result, you want "tick-to-trade" to be in the same process, preferably single-threaded and boost your thread affinity to minimize context switching. To answer your actual question: I would say learn about the concepts that the big cloud providers are pushing. If you open the AWS app drop down - there are dozens of concepts that have been encapsulated in managed or serverless frameworks. They are all worth learning IMHO - as they define the current and next generation of computing.
- tynpeddler 7y ago
  Message queues are really cool, but to paraphrase an old joke, "Some people see a distributed systems problem and think, "I know, I'll use a message queue." Now they have otw problems.
tosh 7y ago
* relational databases (e.g. sqlite, postgres) * regular expressions * lisp / scheme / clojure * emacs * mini kanren / datalog / prolog * neural networks and deep learning * state machines * caches
mntmoss 7y ago
Some simple things with good ROI: 1. Ergonomics tooling: Set up your screen at an appropriate height and your keyboard at an appropriate distance, so that you aren't in pain. If you use a laptop often, try setting it on a shoebox. Use a work timer to take breaks(I've been using Workrave on its default settings lately, which is quite harsh but has worked for my productivity). 2. Diary keeping: Note what you did, what you plan to do, and the date and time. Note as often as you feel necessary. Record notes both in source comments or in general purpose diaries. Use the date to eliminate or rewrite notes that are stale. 3. File management, process management and editing. Take a little time to learn things about your operating system and editor. You don't have to master it, or do elaborate configuration(in fact, having a complex config makes it hard to transfer to other environments). What you want are little things like knowing a few handy shortcut keys or a few built-in tools. 4. Gaining familiarity with writing simple "skeleton" or mock-up code, and waiting patiently for it to mature. When first building a system it may be tempting to apply the biggest algorithmic hammer or design pattern you know of. This is the kind of trick you are learning in learning about message queues. But the most likely outcome of trying to use a trick, no matter how well intentioned, is that this will get you a wrong result more slowly, because the full shape of any problem tends to come into view slowly, progressing from blurry and unclear with a rapidly changing design into something sharp, with well-defined boundaries. As such every new system demands a beginner's mindset, and some ability to refuse engineering in-depth until you absolutely must do so to progress. Done properly, you build a system that leverages existing tools well, has some kind of value now(even if it's limited or lacking a critical feature), and then can harvest the learned lessons into a more complete form later. Trying to get all the features into one pass creates a messy soup: iterating on a subset of them naturally leads towards development of tricks like the message queuing architecture, without any prompting. (edit) 5. Read c2 wiki when bored - it covers a lot of recurring discussions in programming: http://wiki.c2.com http://wiki.c2.com If you know the stuff in it you'll be reasonably prepared to think about any unusual ideas and compare them with existing examples.
nojvek 7y ago
As a programmer, learning how to use a user analytics tool really shift my perspective on things. Profilers and debuggers are tools that tell you where to fix the code. A user analytics tool tells you where to fix the product. Being able to hypothesize ideas, measure conversion, engagement, retention, A/B test, slice by cohorts and validate whether an experiment validates your hypothesis. It makes you tackle problems very systematically. Being able to measure both impact and effort. Always evaluating whether the impact was worth the effort and fine tuning it. The product version or tree falling in the forest joke: If you build something and no one uses it, does it really exist?

927 points by nathanasmith 7y ago

m_fayer 7y ago

https://dataintensive.net/ https://dataintensive.net/ I can't recommend this book enough. I have a CS background, and still had quite a few "I can't believe this thing has been hiding in plain sight!" moments while reading it.

copperx 7y ago
I'm now torn between reading this one first or the Architecture of Enterprise Applications.
- politician 7y ago
  If it helps, IMO "Designing Data-Intensive Applications" is a better bang-for-the-buck. Enterprise-scale applications are a world unto themselves.
- jb3689 7y ago
  I loved Designing Data-Intensive Applications. It gives you the reasons why NoSQL databases exist and the problems they solve. Moreover it gives you reasons to select one over another. It's really excellent and one of my top two CS books
  - mleonard 7y ago
    Your other top CS book out of interest?
- copperx 7y ago
  Edit: I meant Patterns of Enterprise Application Architecture by Fowler in my comment above. Recommended by DHH.
  - louthy 7y ago
    My advice would be to skip it completely. It's just packed full of standard GoF OO dogma.
    - copperx 7y ago
      Thanks. So, what you're saying it is redundant if you've read GoF?
redisman 7y ago
It's great. Incredibly dense with useful information and it just blows my mind how much knowledge Martin has about the topic. I recommend watching this talk from him to give a little glimpse of the book: https://www.youtube.com/watch?v=5ZjhNTM8XU8 https://www.youtube.com/watch?v=5ZjhNTM8XU8 This is just about a little part of one of the chapters.
- mirekrusin 7y ago
  Oh man, this is good, thank you.
splittingTimes 7y ago
But this is mainly for distributed (web) systems, right? Are there good books for data intensive desktop apps? Like games or CAD design tools?

nmca 7y ago

SQL

mfatica 7y ago
You think a self-taught programmer might never encounter SQL?
- politician 7y ago
  Speaking as a technical development manager, I can say that many people have far less than adequate exposure to SQL, and that training people to use SQL effectively and safely is all too often a common requirement before letting them loose on the database. There are so many ways people misunderstand and misuse SQL and relational databases, it's honestly staggering. So, I second the OP. Learn SQL, and you'll stand out.
  - deleted 7y ago
    [deleted]
  - beat 7y ago
    Yeah, learning real sql is a very useful skill. At one employer, I became the "sql expert" because I knew the difference between inner and outer joins. Which, if you know sql, means you know next to nothing. But knowing next to nothing was better than knowing nothing at all, so...

kyllo 7y ago

Learning in-depth your various options for persisting data, is very useful since most applications have to deal with persistence in some form, and increasingly in a distributed manner. Go beyond simply skimming the surface of SQL vs. NoSQL and the marketing claims different databases make about their scalability and consistency. Learn what ACID and CAP stand for and the tradeoffs involved in different persistence strategies. Learn SQL really well. Learn how to read a query plan, which is the algorithm your SQL query gets compiled into. Learn about the tradeoffs of row-based vs column-based storage. Learn how indexes work, and what a B-tree is. Learn the MapReduce pattern. Think about the tradeoffs between sending code to run everywhere your data is stored vs. moving your data to where your code is running.

theossuary 7y ago
Two great resources I've been going through are - https://dataintensive.net https://dataintensive.net - Really deep dives into different types of data storage solutions, their history, and how they actually work. - http://www.cattell.net/datastores/Datastores.pdf http://www.cattell.net/datastores/Datastores.pdf - Good paper that helps differentiate similar but different datastores. Really helpful when you're trying to pick a modern data solution.
- hypertexthero 7y ago
  Another good resource that guides one through both the philosophy (why) and technical details (how) of building a web application is Software Engineering for Internet Applications: https://philip.greenspun.com/seia/ https://philip.greenspun.com/seia/
- barbecue_sauce 7y ago
  Designing Data-Intensive Applications is probably the best O'Reilly (if not overall technology) book of the past decade.
  - btown 7y ago
    The talk on "Turning the database inside-out" [0][1] by the author, Martin Kleppmann, is a fantastic intro to these dynamics, and it's something I'll always recommend to both experienced and inexperienced data modelers and backend developers. It goes pedagogically through the way things are typically done in a relational database in such a clear way that word-for-word it's one of the best tutorials I've seen... but it also weaves a narrative of "how can this be done better/more scalably/more reliably/more flexibly-to-business-needs" in pointing to a streaming/event-sourcing architecture. You may or not need the latter right away, but it's a fantastic tool to have in your toolbox to be able to say "ah, this new requirement feels like it would benefit hugely from this architecture." Especially for OP who's starting to think about the "why" of messaging queues, this could be a fantastically valuable first step. [0] https://www.youtube.com/watch?v=fU9hR3kiOK0 https://www.youtube.com/watch?v=fU9hR3kiOK0 [1] https://www.confluent.io/blog/turning-the-database-inside-out-with-apache-samza/ https://www.confluent.io/blog/turning-the-database-inside-ou...

mattnewport 7y ago

If you're a self taught hobbyist you may not have had much structured exposure to fundamental data structures and algorithms and complexity analysis. I think that type of thing is easier to learn when you already have some experience so you can relate it back to real world problems you have encountered as you describe doing here. Now might be a good point in your development to dig into some of those fundamentals if you have not done so much in the past.

gaze 7y ago

Get good at math. It'll serve you well and never go out of style.

lostphilosopher 7y ago
I'd recommend statistics specifically. Comes up everywhere and it's easy to be wrong about if you don't dig into it.
- wainstead 7y ago
  Second this, if you only work through the Khan Academy stuff on the various things like scatter plots, standard deviation, the normal distribution, etc. you'll be much better off for it.
ThrustVectoring 7y ago
Math is a bit too broad. From personal experience, the most relevant topics for programmers are Linear Algebra and Discrete and Combinatorial Algebra.
- gaze 7y ago
  Eh I mean, I think there's a certain discipline that comes with studying any branch of math to a certain degree of rigor. But yes, linear algebra and discrete math are probably the most useful. I think control theory and optimization are under appreciated amongst programmers though.

quanticle 7y ago

I would recommend learning the SOLID [1] design principles. I've found them to be a very helpful guide when designing software components. [1] https://en.wikipedia.org/wiki/SOLID https://en.wikipedia.org/wiki/SOLID

deathanatos 7y ago
Particularly the S — the Single Responsibility Principal. So much messy, convoluted code is convoluted because it lacks a singular, clear purpose, and bundles up multiple responsibilities into one section of code, be that a module, class, or whatever is appropriate to your language. They're all good, and you'll get good insight from them all, but I think that first one is more important and has provided me more value than all the rest. I think Liskov substitutability would be my second pick.
- collyw 7y ago
  Agree completely.

fancy_pantser 7y ago

The next obvious step after message queues and distributing work would be streams. Here is an excellent introduction to unified logs and stream processing by the author of Kafka at LinkedIn: https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying https://engineering.linkedin.com/distributed-systems/log-wha...

Robin_Message 7y ago

Read the curriculum of an undergraduate computer science course and read up on the things you haven't heard of. Some courses will even have lecture notes available. E.g. these four pages are the university of Cambridge masters in computer science: https://www.cl.cam.ac.uk/teaching/1819/part1a-75.html https://www.cl.cam.ac.uk/teaching/1819/part1a-75.html https://www.cl.cam.ac.uk/teaching/1819/part1b-75.html https://www.cl.cam.ac.uk/teaching/1819/part1b-75.html https://www.cl.cam.ac.uk/teaching/1819/part2-75.html https://www.cl.cam.ac.uk/teaching/1819/part2-75.html https://www.cl.cam.ac.uk/teaching/1819/part3.html https://www.cl.cam.ac.uk/teaching/1819/part3.html (Or a MOOC, but the links above are easy to browse text, syllabuses and lecture notes, not a load of videos.)

rat9988 7y ago
Thank you very much. This is very valuable.
JoeMalt 7y ago
To clarify: the first three links are for each year of the (three-year) undergrad program, the fourth is for the Masters. The Cambridge course isn't perfect, but they do a very good job of making as much teaching material as possible publicly available.
learc83 7y ago
I support this 100%. I worked for years as a self-taught programmer. When I went back for my CS degree, I was shocked at how much I didn't know that I didn't know. Numerous times I'd be sitting in a class and we'd go over a solution to some theoretical problem, and I'd realize that this solved a problem that had taken me days to discover on my own (and this solution was usually better than what I'd come up with). If you are the kind of person who can work through everything on your own (including what may seem like the the boring parts), I highly recommend doing so.
- 147 7y ago
  Could you give an example of one of the times something theoretical helped you solve a real world problem? I've thought about going back for my CS degree a lot but can't really justify the cost and time investment vs self teaching. But it's something that's always been in the back of my mind.

narag 7y ago

A few decades ago, a new programming environment exploded: the web. Looking for a ridiculously useful tech stack? Look no further: HTML, HTTP architecture, SQL backend... a guide written at the time: http://philip.greenspun.com/panda/ http://philip.greenspun.com/panda/ Paul Graham also wrote about why the web was such a deal, IIRC in the Beating the Averages essay. In particular: you can use whatever tools you want and avoid deploying to client machines.

cryptonector 7y ago
PostgreSQL + PostgREST + react-admin == fantastic stack. You can write an entire application in SQL and PgPlSQL but using an HTTP JSON API as the interface with a static and responsive BUI. This allows you to be extremely agile in development and ops (because, e.g., you get to use logical replication). I can't say enough good things about this approach.
- pandemic_region 7y ago
  indeed if your app is about managing your personal dvd collection, or variants thereof, such anemic UI-to-database tools work very well. Not when there is complicated domain logic involved.
  - cryptonector 7y ago
    Having built such an application (w/ complex business logic), I have to disagree. But I'd like to hear what problems you've run into. I would agree that react-admin is a bit too simple, and this stack really calls for a BUI to be built specifically to fit the PostgREST model.
- davb 7y ago
  This reminds me of how we used to create thick client desktop software, connecting directly to the database. Doing most of our business logic on the front end and leaning on the database for auth, relational constraint enforcement and (via stored procedures) transactional consistency and validation. It felt weird at the time when the whole world seemed to move to a three layer model with app servers sitting in the middle - now it's hard to imagine it any other way.

mosalarynolife 7y ago

Design Patterns

sesser 7y ago
What are some resources outside of gang of four?
- qazpot 7y ago
  Head First Design patterns
- pernambucano 7y ago
  https://sourcemaking.com/design_patterns https://sourcemaking.com/design_patterns
- jimpudar 7y ago
  Patterns of Enterprise Application Architecture Enterprise Integration Patterns Just start reading some Martin Fowler books, you will be up to your ears in patterns in no time :)
groby_b 7y ago
Just do yourself a favor and read it the way it's intended: As a dictionary. Design patterns give you a vocabulary to describe design choices you made. They don't give you a set of things that inform design.
wwweston 7y ago
I can't remember who, but someone's said that design patterns really would be more appropriately called something like "palliatives for static manifestly typed languages." Knowing idioms that relate to certain expressive / organizational problems is a good thing (especially if you're primarily working in a static manifestly typed language), but there's a weird overcelebrated status to them, and I'm not sure I'd encourage a developer I was training to become familiar with a full catalogue of them.
- 7y ago

aequitas 7y ago

Makefiles. I always dismissed them as a C compiler thing. Something that could never be useful for Python programming. But nowadays every project I create has a Makefile to bind together all task involved on that project. From bootstrapping the dev environment, running checks/test, starting a devserver, building releases and container images. Makefiles are just such a nice place to put scripts for these common tasks compared to normal shell scripts. The functional approach and dependency resolving of Make allows you to express them with minimal boilerplate and you get tab completion for free. Sometimes I try to take more native solutions (eg. Tox, docker) but I always end up wrapping those tools in a Makefile somewhere forthe road since there are always missing links and because Make is ubiquitous on nearly every Linux and macOS it is just all you need to get a project started. Example: https://gitlab.com/internet-cleanup-foundation/web-security-map/blob/master/Makefile https://gitlab.com/internet-cleanup-foundation/web-security-... Running the 'make' command in this repo will setup all requirements and then run code checks and tests. Running it again will skip the setup part unless one of the dependency files has changed (or the setup env is removed).

pletnes 7y ago
One of the worst problems with using windows (in my opinion) is that there’s no native GNU make.
- aequitas 7y ago
  Isn't non-native development on Windows a solved problem nowadays with WSL(2)?
  - murderfs 7y ago
    WSL is currently horrendously (unusably, IMO) slow. WSL2 promises a 20x speed up, but it was already 100x slower than native Linux at some actually-realistic workloads that happen all the time when you're developing (e.g. `git grep`), so it's probably still too slow to be tolerable. I had the opposite problem of wanting to develop some stuff for Windows from a Linux environment, and I settled on running a linux VM and copying binaries over by scping to WSL, which works reasonably well.

mettamage 7y ago

It depends on what you want to achieve. I can tell you to read about security but that is only relevant if you want to secure your software. I can tell you about pub/sub, but that is only relevant when you need it. The things you’ve learned were mostly covered in my computer science curriculum. So I am going to second Robin_messsage his message. Skim through a CS curriculum.

zoomablemind 7y ago

I'm not sure if 'hoarding' such knowledge would be practical. Sometimes, having a limited toolset would better focus you on the problem at hand. Then, once the challenge is clarified, the search for alternative ways to architecture and implement it would become practical. If just for fun of exploring something new, pick whatever interests you in general, language, framework, a domain. Async processing and idioms?

vosper 7y ago

Your IDE, its refactoring tools, but especially its debugger. VSCode or PyCharm (assuming you are still a Python developer) could be a good place to start. I'm always surprised when I see professional developers coding in Sublime Text and debugging with print statements (or their equivalent). Usually you have better options than that, especially for statically-typed languages - but even for JS and Python.

collyw 7y ago
I don't know how people could work on large scale projects without a step through debugger.

danesparza 7y ago

Caching, logging, centralized configuration, Security, and Design Patterns are all probably easy to overlook.

nerpderp82 7y ago

Debuggers and property based testing. It is a select few people that can actually productively (not their own metrics) use print statements for debugging. Learning how to craft repro scenarios and adequately capturing state in a debugging session can enable junior devs to easily surpass senior devs. Property based testing aren't quite formal methods, but I think they are a good stepping stone. And they also somewhat force your code into an algebraic/functional style which also make it amenable to refactoring, better testing and is easier to understand. Design tools like Swagger can help one think through services w/o diving into code. Code itself is a liability and should be thought of as "spending" not creating. Code is a debt. Refactoring and code understanding tools, if you use PyCharm (you should, it is free in all senses), learn how to navigate into your libraries. Read you libraries.

marktangotango 7y ago
This x1000 debuggers are seriously undervalued by many developers. It’s like a super power.
- thetwentyone 7y ago
  What are some good resources to learn about debugging patterns and tips/tricks? My preferred language, Julia, recently introduced a nice set of tools related to debugging. I feel like there's probably things that would make me more productive but I think the techniques would be more broadly applicable than a specific language.
  - drainyard 7y ago
    One thing I always do these days is I step through any new code I've written the first time I run it. This usually weeds out some bugs that might take a while to find because they are easy to miss. It also ensures that you actually go through each line of code you write, doing a forced code review on yourself early on.

brianpgordon 7y ago

Don't go too overboard with message queues. There's nonzero development and operational overhead incurred when part of your application takes its input in a weird binary format, and when the data in your queue is thrown away after processing, and when you need to think about scaling of workers and concurrency. If you're not working with real "big data" – and, let's be honest, almost nobody is – I would advise using an HTTP-based service (REST, SOAP, whatever, take your pick) for communication and a SQL/NoSQL/NewSQL database for state.

jrockway 7y ago
ZeroMQ is not really a message queue, it's more of a networking library. It takes TCP sockets and adds other concepts on top, like request/reply or publish/subscribe.
massaman_yams 7y ago
Doing a task queue in an RDBMS is not as simple as some might think, though. https://www.2ndquadrant.com/en/blog/what-is-select-skip-locked-for-in-postgresql-9-5/ https://www.2ndquadrant.com/en/blog/what-is-select-skip-lock...

amrox 7y ago

Learning the basics about how programming languages work - parsers, interpreters/compilers. I've heard good things about Writing An Interpreter In Go [1]. Related, I've enjoyed Martin Fowler's DSL book [2]. - [1] https://interpreterbook.com - [2] https://www.martinfowler.com/books/dsl.html

O_H_E 7y ago
Just noting that the URLs weren't formatted as links but code

purplezooey 7y ago

Brown nosing.

new_guy 7y ago

'The Imposters Handbook' is good for foundational knowledge, it was written for exactly your use case. https://bigmachine.io/products/the-imposters-handbook/ https://bigmachine.io/products/the-imposters-handbook/

LeonB 7y ago
Came here to recommend this too. It’s goal is very much aligned with the OP.

paulgb 7y ago

State machines. They can be a helpful abstraction for UI or business logic, and they also make regular expressions make a lot more sense.

jimpudar 7y ago
Yeah, far too often people create horrific spaghetti code of if/else statements which could be neatly coded as a state machine. This is a very important concept to grasp.

deleted 7y ago

[deleted]

lixtra 7y ago

Learn to use a profiler for your programming language. It will tell you what’s worth optimizing in your programs. I.e. if your program spends only 10% of the time talking to others than that may be the maximum you can optimize by choosing a fancy communication pattern.

westonplatter0 7y ago

SQL

throwaway55554 7y ago

Develop a project that has two local processes that have to communicate, one local database, and one remote service that the two processes have to communicate with. Doesn't matter what the project is. Could just be tossing around a string of data that eventually gets dumped into the db and sent to the server. Research different methods of architecting such a system. Code up a few. By doing this, you will actually find the answer to the question you posed. And, arguably, have fun doing it.

lioeters 7y ago
I love it. The best way to learn is by doing, and the suggested project idea is simple enough as a starting point, and complex enough to require deep thinking and exploration in various aspects to figure out a good solution. Many of the recommendations in other comments could tie into architecting such a study project too: data structures, algorithms, communication among distributed processes..

hannob 7y ago

If you happen to program in C/C++ you should absolutely familiarize yourself with the sanitizers of gcc and clang, most notably address sanitizer. (However there are good reasons why you shouldn't program in C/C++ to begin with.)

justinholmes 7y ago

Using GitHub trending for each language look at new things each month. Build something and think how would I scale this project to X capacity then rinse and repeat.

joker3 7y ago

https://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-005-software-construction-spring-2016/ https://ocw.mit.edu/courses/electrical-engineering-and-compu... In my experience, most hobbyist programmers are fine writing scripts and other small programs but have no exposure to the sort of ideas that are necessary for making large programs. This course at least exposes you to a lot of those concepts.

metahost 7y ago
The latest course is available here: http://web.mit.edu/6.031/www/sp19/ http://web.mit.edu/6.031/www/sp19/

avip 7y ago

The concept of monitoring. This simple, obvious concept changed a lot how I think of software maintenance. I usually use slack but that's an implementation detail.

gwbas1c 7y ago

Unit testing, mocking, and various other testing techniques. Why? Any project of sufficient complexity is very hard to test. If all you're doing is code -> build -> run to debug your code, you can very easily break something that's not in your immediate attention. The problem is that good unit testing is hard, and time consuming. It can be so time consuming, that unless you can really plan in advance how you test, you could spend more time writing test code than real application code. (This is what happens when writing professional, industrial-strength code.) So, when a hobby project becomes sufficiently interesting enough; such that the code will be so complicated that your code -> build -> run loop won't hit most of your code, you should think about how to have automated tests. They don't have to be "pure, by the book" unit tests, but they should be an approach that can hit most of your program in an automated manner. You don't need to do "pure" mocking either. If you're writing something that calls a webservice, you could write a mock webserver and redirect your program to it. If you're writing something that works with pipes, you could have a set of known files with known results, and always compare them. The goal is that you should cover most of your program with code -> build -> tests; and only do code -> build -> run for experimentation.

wpietri 7y ago
Let me second this. And in particular, I strongly encourage every developer to try starting a new project in a test-driven fashion (by which I mean that you advance the code by writing a bit of test and making it pass, and then doing that over and over.) There's a qualitative difference between working in a well-tested code base that's very hard to describe convincingly. A lot of my early development experience was in code bases that had little or no testing. Experiencing a well-tested code base totally changed things for me. Instead of work being a death-of-a-thousand-cuts experience, it became pleasant, steady progress.
- mooreds 7y ago
  > Experiencing a well-tested code base totally changed things for me. Instead of work being a death-of-a-thousand-cuts experience, it became pleasant, steady progress. I had the luxury of taking a well known data process and rewriting it with integration tests (input in, matching output with a golden file). It changed my professional life. Whereas before our deployment process included a 3 day wait and manual data checking on stage, after I was able to do deploys multiple times a day with confidence. Made a believer out of me.

enkiv2 7y ago

I highly recommend learning PROLOG & understanding how to write your own simple planner system. The hairiest real problems are hairy because they're best suited to a declarative style (and programs written declaratively can be made much more efficient through more clever solvers -- given naive code, a clever solver has a much bigger efficiency boost over a dumb solver than an optimizing compiler does over a non-optimizing one -- although PROLOG itself leaks too much abstraction for many of these techniques to be viable in it). I also recommend understanding message routing systems used in file sharing, like CHORD. If you don't have a strong background in the math behind theoretical computer science, you might benefit a lot from an understanding of the formal rules around boolean logic, symbolic logic, & state machines -- especially, rules about when certain kinds of things are equivalent (since stuff like demorgan's law are used for simplifying and optimizing expressions a lot, and rules for state machines are used to prove upper limits on resource usage). If you don't already, learn to use awk. It's a much more powerful language than it seems, and fits extremely well into the gap between command-line prototyping in shell one-liners & porting a prototyped tool to python or perl, and so it's a huge time saver: it is faster to write many kinds of tools in a mix of shell and awk and then rewrite them in python than it is to write them in python in the first place.

bdamm 7y ago
I've never used Prolog in the 15 years since I learned it in college. It's an interesting take on programming, for sure, and I appreciated the mind-expanding exercise, but hasn't helped me in my career at all. Totally agree on awk. I use it almost every day for quick little one-liners. Big time saver. Also agree on state machines, because from there it is a short hop to understanding formal grammars and the foundation of compilers and languages, which has been immensely useful in my career.
- doboyy 7y ago
  Learning about automata theory was one of the most mind-expanding experiences I had in college. It was certainly not something I would've stumbled upon without guidance. I believe that describes most of the value I derived from a degree, being nudged in the right directions toward solutions and problems that a lot of smart people have though about for a while. Understanding how different languages, or inputs more generally, can be transformed into meaningful outputs is pretty satisfying. It's a topic that almost seems to transcend the realm of computer science.

CalChris 7y ago

Godbolt Compiler Explorer: https://godbolt.org/ https://godbolt.org/ DTrace: http://dtrace.org/ http://dtrace.org/

WoodenChair 7y ago

The best tools you can teach yourself are arguably more fundamental than any specific language or library, but instead computer science problem solving techniques. I've written books on this subject specifically targeted towards self-taught programmers: https://classicproblems.com/ https://classicproblems.com/ Sorry for the self-plug, but it's super relevant I think.

jimpudar 7y ago

Learning how to use dtrace / bpftrace [0] is very valuable if you ever need to get into serious systems profiling. There are some really cool data structures out there you might not know about. One of my favorite basic ones that I get a lot of use out of is the trie [1] (a.k.a. prefix tree). Very useful for IP calculations. Also look into probabilistic data structures [2], very amazing things can be done with them. [0] https://en.wikipedia.org/wiki/DTrace https://en.wikipedia.org/wiki/DTrace [1] https://en.wikipedia.org/wiki/Trie https://en.wikipedia.org/wiki/Trie [2] https://en.wikipedia.org/wiki/Category:Probabilistic_data_structures https://en.wikipedia.org/wiki/Category:Probabilistic_data_st...

cryptonector 7y ago
DTrace is life-altering. I keep hoping that someone will build a dtrace(1) CLI that transpiles to bpftrace.
stcredzero 7y ago
Profiling, period
sammnaser 7y ago
Bloom Filters are awesome.
Const-me 7y ago
Approximate Windows equivalent of DTrace is sysinternal process monitor, freeware. Very useful sometimes.
- Birch-san 7y ago
  The Windows equivalent of DTrace is.. DTrace. [0] DTrace is about far, far more than snooping the filesystem. At best, Process Monitor is an equivalent of Brendan Gregg's DTrace utility, opensnoop. The true power of DTrace is to correlate events across subsystem boundaries. Like, graphing the top quartile of latencies from network acceses initiated via a given function in your application. [0] https://techcommunity.microsoft.com/t5/Windows-Kernel-Internals/DTrace-on-Windows/ba-p/362902 https://techcommunity.microsoft.com/t5/Windows-Kernel-Intern...

malvosenior 7y ago

If you're coming from Python you should start looking into how other languages handle concurrency. Python has a GIL (global interpreter lock) that only allows for single threaded execution under normal circumstances. Learn about threads, locking, mutexes, semaphores, green threads, race conditions...

neilv 7y ago

At some point, it helps to broaden your programming language exposure, even if you stick to mostly one language for most of your work. You'll find ways to apply ideas from other languages/communities to your work. Try to spend some time learning idiomatic programming from one of the Lisp family (Scheme, Racket, CL, Emacs Lisp, and Clojure all have different thinking, but a lot of overlap). Play a bit with Smalltalk or a similar descendant, even if you're already doing OOP elsewhere. At some point you should learn a textual expansion language, like one of the Unix shell scripting ones, or Tcl (and learning basic Bash scripting will probably be useful in tech work). Try a logic programming language, like Prolog, or one that's a minilanguage within another, like Mini-Kanren. Maybe buckle down for hardcore functional programming (e.g., Haskell, OCaml, or discipline yourself to do it in a Lisp?). You should also get comfortable with C or at least an assembly language at some point, to have a better idea of what other languages are and aren't giving you, and also C is just a really useful thing to know when you need to write a little fast code, FFI to a native library, or get into languages/IRs for newer target architectures. (Disclosure: I've been especially involved with Racket, an energetic close descendant of Scheme, and have some interest in promoting it, but I'd list a Lisp as one of the first in any case.)

jimpudar 7y ago
Racket is a very good Lisp for beginners, DrRacket makes it easy to get up and running with little effort.
- neilv 7y ago
  Agreed. Racket is from a particular school of thought, and you won't get all Lisp family ideas from it, but it's great. You can start up DrRacket (a simple IDE for students that can also be used for professional work, and has some powerful features in it), or just use your favorite editor and the `racket` command-line program and REPL. There's way too much documentation at: https://docs.racket-lang.org/ https://docs.racket-lang.org/ You can also do the old MIT introductory CS textbook, SICP, using Racket.
  - snazz 7y ago

asdffdsa 7y ago

Books: Operating Systems/Database/Networking/Computer Security/Computer Architecture textbooks, Software Engineering textbooks (Clean Code, Design Patterns, Designing Data Intensive Applications, Domain Driven Design as a short list off the top of my head)

stevekemp 7y ago
By the same token "documentation". Today I explained to a colleague how they could enable port-forwarding across their already-open-SSH-session via "~C". Every now and again I pick a tool I use a lot, and read the man-page. Things like "less", "bash", "ssh". Complex enough to contain surprises, but simple enough that you take them for granted. Almost always this has been time well-spent.
lbrindze 7y ago
"Designing Data Intensive Applications" is an absolute goldmine for things like message queues but also going beyond understanding the full implications in database selection and other common, distributed-oriented engineering decisions modern software engineers may come across.

OliverJones 7y ago

Event-driven, asynchronous, programming Here's a valuable addition to your toolkit of mental models for programming: Event-driven, asynchronous, programming (in the style of ES6 Javascript or a similar language. ) Some suggestions about how to learn the basics? tutorials on... -- building a so-called "single page web app" with a framework like vue.js or even jQuery. -- node.js to build a complex back-end server without using a threading model) -- React (to build an interactive program to run in a browser)

agentultra 7y ago

Formal methods. It took me nearly a decade of working in distributed systems to be introduced to TLA+ and other tools in this space. Until then my knowledge had been built from textbooks describing the fundamental data structures, algorithms, and protocols... but those texts take an informal approach with the mathematics involved. And since I was self-taught I was reading those texts with an eye for practical applications than for theoretical understanding. I had no idea that a tool existed that would let me specify a design or find potential flaws in systems and protocols, especially concurrent or parallel systems, with such ease. I think type theory and category theory have also been great tools to have... but I think mathematics in general is probably the more useful tool. Being able to think abstractly about systems in a rigorous way has been the single-biggest booster for me as a practitioner.

pickdenis 7y ago
> the single-biggest booster for me as a practitioner. Can you instantiate this claim with an example? I'm somewhat knowledgable in both math and computer science theory but have yet to feel as though my math background has helped me in practical CS.
- agentultra 7y ago
  About 3-4 years ago I was working on an open source cloud platform for a company deploying a public cloud. There was a particular error that sometimes happened in our production environment where a rebooted VM would come up but couldn't connect to the network. We tracked it down to a race between two services in the data plane. It turns out the VM controller wouldn't wait for the network controller to unplug a virtual interface before requesting the interface to be plugged back in. There was a lack of co-ordination happening. However it only happened when the network component was under heavy enough load that it would take too long to respond before the VM service finished rebooting the VM -- usually it was fast enough and the error wouldn't appear. I managed to model this interaction at a high level in TLA+. From there I had suspected that the error was in the mutex locking code in the async library this system depended on so we refined the model pretty close to that implementation. As I recall we found that the mutex code wasn't the culprit -- a fine result. We ended up implementing some light-weight co-ordination mechanism to ensure that the VM service waits to acknowledge the progress of the network service. Since then I've continued to use TLA+. I find that programming languages are insufficiently expressive enough to describe high-level interactions between other processes, network events, and humans.

jpmelos 7y ago

If understanding how Linux works interests you: http://www.linuxfromscratch.org/ http://www.linuxfromscratch.org/

caust1c 7y ago

Heh, funny that you mention that ZMQ is overlooked. While often though of a message passing library or serverless queue, Zero MQ has some pretty severe limitations that implementers fail to consider. To be clear, I think it's an amazing library which is unmatched in its performance, but it comes at a cost: reduced reliability. ZeroMQ will drop messages in a number of situations. The library does not handle delivery guarantees which means that the application must do it themselves. Whether or not this works for you is an application level concern. However, having used it at two companies now: both times it ended up being thrown out for a more reliable queue (kafka). http://zguide.zeromq.org/py:all#Missing-Message-Problem-Solver http://zguide.zeromq.org/py:all#Missing-Message-Problem-Solv... So maybe the reason you haven't stumbled upon it sooner is because it's overhyped? Definitely useful but with a grain of salt.

aasasd 7y ago
Yeah, every new user has to learn that ZMQ is not a proper queue, but simply a network library and protocol. How is it even possible to bungle the name so badly and then do nothing about it for twelve years?
wallstprog 7y ago
The "MQ" part of the name is unfortunate, but apparently came about because of the original idea to come up with a "better" implementation of AMQP (http://zeromq.org/docs:welcome-from-amqp http://zeromq.org/docs:welcome-from-amqp). But you're right -- ZeroMQ doesn't do queueing (except in some very limited circumstances), and if you need reliable delivery you must implement that yourself "on top of" ZeroMQ. I've done that, and while it's not a simple task, it is certainly possible. You can get reliable delivery "out of the box" with other software, that in fact does do queueing. (kafka may be one, but I don't know enough to say). But what you give up when you do that is performance -- ZeroMQ can easily be orders of magnitude faster than those other solutions, and for some applications (e.g., real-time market data) the work to provide a custom reliability solution on top of ZeroMQ is worthwhile.

Jach 7y ago

If you're strictly asking about things a self-taught hobbyist programmer may have missed, then I second the suggestions here to skim through a CS degree curriculum and dig into anything that's unfamiliar. As one possible example, maybe you're solving a problem trying to parse some text, and you're in over your head with ad-hoc regexes and conditionals and type casts and exceptions all over the place. If you've seen the concept of a grammar (likely to come up in CS programs, though not all) and of generating parsers / validators for them, you can eliminate a whole lot of programming by specifying a grammar in some common format (for instance, ABNF) and running it through a program that generates a parsing program for you. The general category of programs writing programs is worthwhile to look into, and playing with languages where such a feature is first-class (like Common Lisp, which apart from having macros also has a compile-file function you can invoke at runtime) can be enlightening. A lot of the comments here are highlighting things that not even most CS degree holders will know about, nor many professional programmers. Those can be useful for a hobbyist too, so maybe the lesson is that regardless of your current level of knowledge it's important to keep in mind: "There are more things in heaven and earth, Horatio, than are dreamt of in your philosophy." Your comment about feeling comfortable in the practical details reminded me of how I used to feel as a fresh self-taught PHP-wielding teenager after a few years of it. As an "expert" PHP coder, I could do anything! Even an OS if I wanted! Well I learned better eventually. :)

stormking 7y ago
"If you've seen the concept of a grammar (likely to come up in CS programs, though not all)" Really? That was one of the very first topics in my first semester. How do you teach CS without grammars and the Chomsky hierarchy?
- Jach 7y ago
  I wondered if someone would ask about that... It's just my own recollection from years ago when I looked at a bunch of CS curricula from public and private schools. There was a lot of patchy variation. If the topics were covered at all they'd typically be done in optional electives like a compilers course, or "theory of computation" course, taken in junior or senior years. Also depending on the school such an elective might not actually be available anymore (I looked at current-semester/last-semester offerings to try and identify those) and is only part of the catalog for historical reasons. Maybe it was offered once, but not anymore, in part because students having the choice would rather take the new data science course or Advanced Networking or whatever to fulfill that elective instead. I do think the problem is mostly a function of having lots of choice. Everyone's got their own list of what a CS degree absolutely must cover, but there are going to be differences, and schools are incentivized to let students carve their own path. Let's look at this list: http://matt.might.net/articles/what-cs-majors-should-know/ http://matt.might.net/articles/what-cs-majors-should-know/ I've looked at a lot of intern resumes over the past few years and broadly they've been impressive from a hire-to-BigCo perspective but I would be surprised if many of them knew much if anything about a large majority of topics from that list. Even narrowing to the "better" schools. I also don't even agree with that list, I just don't think it's realistic to cram all that into a 4 year program on top of all the other STEM courses, humanities courses, and project courses... Going back further there's the continuing problem of "dumbing down". It ties into a school's incentive to offer more choices to fulfill the graduation requirements (creating an "easy" path), but as a problem in itself it's worth considering. https://www.joelonsoftware.com/2005/12/29/the-perils-of-javaschools-2/ https://www.joelonsoftware.com/2005/12/29/the-perils-of-java... is a classic rant about it, not a very good one but it points at one consequence of the problem. I would bet there's still a sizable bunch of current-year CS graduates who never had to deal with pointers. Maybe I'll do some data exploring some time with the resumes I still have to see what fraction put C or C++ somewhere to make a weak proxy measure.

jedberg 7y ago

If you're into python, look up decorators and coroutines. They will blow your mind.

Jach 7y ago
This is still my favorite mind-blowing introduction to them, from 2009: http://www.dabeaz.com/coroutines/Coroutines.pdf http://www.dabeaz.com/coroutines/Coroutines.pdf Don't have hardware interrupts or threads but still want a multi-tasking operating system? Not A Problem.

crimsonalucard 7y ago

Nobody mentions what self taught programmers miss the most. Theory. And not just algorithm theory.

Zealotux 7y ago
Could you expand on that if you don't mind? I'm a self-taught programmer currently in the process of realising just how much I miss from theory, my list so far on theory fundamentals includes: • Mathematics and probabilities, and their applications to CS (e.g. formal methods) • Design patterns (OOP, functional programming) • Data structures • Algorithms, time complexity • System architectures • Software strategies: CI, CD • Database principles: SQL It may sound naive, but I'm kind of overwhelmed by all this and it's not helping my impostor syndrome, I may make a repo of the list with links to ressources I've identified for learning as it seems like a common struggle, maybe it'll help someone.
- crimsonalucard 7y ago
  None of what you mentioned is theory expect for Mathematics, probability, formal methods and algorithms. Stuff like design patterns, system architectures and software strategies are like flavor of the week stuff. Opinions basically. Patterns like microservices are bad or good depending on opinion, but theory is always correct. Theory gets less bang for the buck but it's always what many programmers especially self taught ones are missing. Theory is so hard that it will be hard to see applicability until you're a more seasoned programmer. Many seasoned programmers get by without ever knowing theory. But you will be a better programmer if you know it. If I were to recommend one theory to study it would be category theory. If there was any true axiomatic theory for design patterns or how to design programs... categories are it. The study of morphisms is the study of the simplest form of a compose-able module. Knowing this theory you will begin to understand why Some design patterns don't work and why it's sometimes hard to reuse patterns in code that was that not properly designed. Theory doesn't answer all questions but for the questions it does answer you will get a definitive answer and not an opinionated one.
- sn9 7y ago
  The canonical undergrad CS theory text is Sipser's Introduction to the Theory of Computation. The prereqs are a comfort with discrete mathematics and proofs and a course in algorithms and their analysis.

gamegod 7y ago

IDEs - A proper work-grade IDE like Visual Studio will have some tools built into it that will save you tons of time. Features like "Edit & Continue" have saved my bacon many times by making intractably difficult bugs in algorithms much easier to understand because you can experiment with your code on the fly. There's also some more common features in most IDEs like being able to jump between symbols that saves a lot of time day-to-day. If I'm interviewing you and you say you don't like using an IDE or a debugger, that speaks to your work experience, your productivity, your self awareness about your productivity, and really puts an upper limit on the difficulty of the problems you've had to solve.

skummetmaelk 7y ago
I find that it is usually the opposite. Devs who don't use big IDEs like visual studio are much more likely to know why things fail in a build pipeline etc. If you can only build a project by bumbling through menus and pressing a big green button at the end, that is worrying. If you can only debug by immediately jumping into the debugger and single stepping that is also worrying. Devs who reach for the tools appropriate to a given situation inspire much more confidence in their abilities. Knowing the time saving features of your editor is a huge boost. However, the old editors have much more of these ;)
- snazz 7y ago
  IDEs aren’t useless, you just don’t want them to be a crutch. For this reason, I usually recommend new developers use Nano or a notepad-esque editor until they understand why they might want vi keybindings, then use Vim until they understand why they might want an IDE or something like Emacs. Starting with the IDE hides layers and layers of both junk and useful tools, while experienced developers know which layer of the stack to work on at which time.
clarry 7y ago
> If I'm interviewing you and you say you don't like using an IDE or a debugger, that speaks to your work experience, your productivity, your self awareness about your productivity, and really puts an upper limit on the difficulty of the problems you've had to solve. Or you have surpassed the limit on the difficulty of the problems your debugger & IDE can help with.

augbog 7y ago

Honestly I still find a lot of engineers don't know git properly. Like they know enough to commit and push but that's about it. It really helps to understand everything git has to offer.

333c 7y ago
The three most important basic git operations to know (in my opinion) git checkout -b git log git rebase -i
- jimpudar 7y ago
  I also find git blame extremely useful, along with code exploration tools like DeepGit [0] [0] https://www.syntevo.com/deepgit/ https://www.syntevo.com/deepgit/
- dwoot 7y ago
  I'll have to add `git reflog` for instances where you've felt that you've completely screwed up something as one can always move back to a previous state. I think this is essential. A useful one that I'll add is what I call the sword command: `git log -S<word>` This one allows one to list commits that contain a particular change. This has been useful in tracking down old changes
- btilly 7y ago
  I strongly prefer git merge over git rebase. Using rebase results in a cleaner history and simplified workflow in many cases. However it also means that when you have a disaster, it can be truly unrecoverable. I hope you have an old backup because you told your source control system to scramble its history, and you don't have any good way to back it out later. For those who don't know what I mean, the funny commit ids that git produces are a hash signifying the current state of the repository AND the complete history of how you got there. Every time you rebase you take the other repository, and its history, and then replay your new commits as happening now, one after the other. Now suppose that you rebased off of a repository. Then the repository is rebased by someone else. Now there is no way to merge your code back except to --force it. And that means that if your codebase is messed up, you're now screwed up with history screwed up and no good way to sort it out. That result is impossible if you're using a merge based result. The cost is, though, that the history is accurately complicated. And the existence of a complex history is a huge problem for useful tools like git bisect.

jacksnipe 7y ago

Not nearly as big a deal as some of the other tools and techniques being mentioned, but tmux/screen are lifesavers when you need them.

niklasjansson 7y ago

Excel. Seriously. For more things than you imagine Let's say you want to make a property assignment for some class; this.a=x.a this.b=x.b ... While you probably would want to do this some other way to start with, and while you of course can solve it using some emacs wizardry, I can whip that up in Excel using some formulas in a matter of a minute. Moreover, I can keep adding to it when I realize something was missing. I can make a diff, join or union of the results of two queries from different databases (or even database engines). Not to mention calculation and design mockups.

hsitz 7y ago
I'm not following. What good is it to have equivalent of "this.a=x.a this.b=x.b ..." in Excel, when what I need is to have this in code in my editor? Are you saying you use Excel to create the code, then copy/paste it into your editor? Or what?
- JonathonW 7y ago
  I think the parent's using Excel essentially to do repeated template expansion: e.g. for a given set of member variable names ([a, b, c...]), give me the assignment statements I'd need to use those in a constructor. Which I could do pretty trivially in Excel... but could also do trivially in about two lines of Python: vars = ["a", "b", "c"] statements = ["this.{0} = x.{0}".format(var) for var in vars] print(statements) Spreadsheets can be an incredible tool-- as an interactive environment that allows non-programmers to express domain knowledge and quickly automate parts of their workflow, they're really unparalleled. But this isn't a very good example of something that Excel is particularly well-suited for.
  - UnFleshedOne 7y ago
    Those can often be done in a decent text editor with regexp-like search and replace.
    - kragen 7y ago

mnemotronic 7y ago

* Regular expressions. The most valuable "not a language" that I know. * SQL. For me, it was a trip coming from a procedural language background. I kept trying to figure out how to do loops. * Command line - DOS and unix * Batch languages for those. * HTML / CSS / Java (please don't kill me) Script and the DOM

JamesBarney 7y ago
Second regular expressions. I use it all the time to edit large bundles of code.

hsitz 7y ago

A new version of "The Pragmatic Programmer" recently came out. [EDIT: not available yet, only preorder at amazon, beta version available at pragprog.com.] That book is all about tools and methods that a self-taught programmer should look into: https://www.amazon.com/Pragmatic-Programmer-journey-mastery-Anniversary/dp/0135957052 https://www.amazon.com/Pragmatic-Programmer-journey-mastery-...

colomon 7y ago
For me, that Amazon page is listing it as a pre-order, without any release date. And all the other versions (Kindle, Paperback) are the 1st edition instead of the 2nd. Very frustrating, as I considered the first edition to be essential and upon reading your comment, instantly went to purchase the 2nd edition. Edited to add: Found a date, Amazon is listing it as October 21, 2019.
- hsitz 7y ago
  Sorry, I thought I'd read a review of it already, so just didn't look closely to see it wasn't available yet. It looks like you can get a DRM-free beta version of the ebook on their website, with free upgrades to published version once it's finalized: https://pragprog.com/book/tpp20/the-pragmatic-programmer-20th-anniversary-edition https://pragprog.com/book/tpp20/the-pragmatic-programmer-20t...
- deleted 7y ago
  [deleted]

SonOfLilit 7y ago

More important than any specific technique or tool is that you get an experienced mentor to look at your work once in a while and point out the obvious things relevant to the specific project that you don't know you don't know. Seriously, you'll probably save a few months of work in the first ten minutes of talking to a senior developer about your hobby project. Talk to someone with many years of experience, though, otherwise you'll most likely get sent on a quest for beauty of implementation that has nothing to do with the goals you're trying to achieve. Sadly, that is a lesson that takes a long time to internalize.

SonOfLilit 7y ago
(Learning all the things recommended in this thread would take a few years, and maybe half of them would be useful given you know them, a lot fewer useful enough to justify the price of learning them)

cameronbrown 7y ago

I'm surprised nobody's mentioned spreadsheets - specifically Google Sheets or something scriptable and hosted. Recently I've built up a small system which sucks in data from a few places (fitness, task management, calendar,..) and analyses it against several goals I've set. This means I can see how I'm progressing towards what I want without even touching it anymore. They're a really nice UI for bootstraping projects, and even some small databases. A current project of mine for a client uses sheets as a backing store for an email collection list. Since there's only a few hundred rows, it makes sense at this scale and works really well for non-technical users.

Forge36 7y ago
Similarly Microsoft Access. I learned Access for my first job (100% of my job was moving tasks into Access, 3 years later I helped some interns transition to a SQL server+web application). 10 years later I'm still using Access for some aspects of my new job. The ability to quickly prototype something and receive almost immediate value is vastly underestimated (I know of one company which runs all of their tasks out of Access).
- cameronbrown 7y ago
  I suppose what I've described is just a little more of an unstructured version of this. Mixing data and computation in such a way is definitely a tech debt tradeoff though.
thecupisblue 7y ago
Excel & Google Sheets are probably the best enterprise software development platform ever made.

theonemind 7y ago

* Parser generator tools like ANTLR https://www.antlr.org/ https://www.antlr.org/, or even lex and yacc, useful for parsing languages/config files and probably generating a better/more robust parser than you'd cobble together by hand * Dynamic programming https://en.wikipedia.org/wiki/ https://en.wikipedia.org/wiki/ -- great for relatively-quickly (in computation time) coming up with good-enough solutions to some hard (like NP hard) recursive problem that would take forever (almost literally) to technically find the absolute best solution to, but not something you'll use every day.

sgillen 7y ago
I was under the impression DP was usually used for getting the exact answer to certain classes of problems faster than exhaustive search. I usually see approximations compared to the ground truth dynamic programming answer. Do you see the reverse in your work?
oneturkmen 7y ago
That's weird. DP, though better than naive brute force, is still very slow since it looks only for an exact, optimal solution.

wpietri 7y ago

For me a lot of programming involves some amount of data wrangling. E.g., getting input data ready. Or generating and understanding results of some technical experiment. I recently came across VisiData and adore it. It has a steep learning curve, but I've found it very much worth it: http://visidata.org/ http://visidata.org/

nickjj 7y ago

Shell scripting for processing text. You can often get so much done with so little code and effort. Also on a semi-related note, I think as a self taught programmer, it's easy to get stuck on things that seem cool but are just procrastination enablers (I know, I've been guilty of it for 20 years). Like, if you're about to start a new project and you want to flesh out what it's about, you really don't need to spend 5 hours researching which mind map tool to use. Just open a text document and start writing, or get a piece of paper and a pen. It won't even take that long. I spent about 1.5 hours the other day planning a substantially sized web app. All I did was open a text file and type what came into my head. For fun I decided to record the whole process too[0]. I wish more people recorded their process for things like that because I find the journey more interesting than the destination most of the time. Like your journey of eventually finding message queues must have been quite fun and you probably learned a ton (after all, it lead you to message queues, so it was certainly time well spent). [0]: https://nickjanetakis.com/blog/live-demo-of-planning-a-real-world-web-application-from-scratch https://nickjanetakis.com/blog/live-demo-of-planning-a-real-...

Draiken 7y ago
This 1000x. I put of just getting by with shell script for some years and when I finally decided to get deeper into it, it's magical. A good series of piped commands with tools available basically everywhere can solve problems you had no idea could be so simple to solve.
- 0x262d 7y ago
  have any good resources for this?
  - fancyfish 7y ago
    Perhaps these are a good start: awk: - GNU Awk User Guide - https://www.gnu.org/software/gawk/manual/gawk.html#Getting-Started https://www.gnu.org/software/gawk/manual/gawk.html#Getting-S... - Grymoire guide - http://www.grymoire.com/Unix/Awk.html http://www.grymoire.com/Unix/Awk.html sed: - Grymoire guide - https://www.grymoire.com/Unix/Sed.html https://www.grymoire.com/Unix/Sed.html - Official docs - http://sed.sourceforge.net/#docs http://sed.sourceforge.net/#docs

namank 7y ago

What you want to do is ponder over how common functionality around you must work - if you can't come up with a reasonable solution, it's time to google it and learn it. If you can come up with a reasonable solution, google to check if you're correct. This is the most pragmatic way I've found to learn and stay in touch.

LeonB 7y ago

So many of the problems are “people problems” so in addition to the excellent technical suggestions in this thread I’d add books like “Thanks for the Feedback”, “How to Win Friends and Influence People”, “Getting to Yes”

yyx 7y ago
"start with no"

Const-me 7y ago

I don’t think there’re many ridiculously useful things that are universally useful. If some stuff is domain specific or have much theory involved doesn’t mean it’s not considered a fundamental concept in some areas of work. If some other stuff was very useful for you doesn’t mean it’s universally applicable. For example, if you had worked on a high-frequency trading bot, I don’t think you would have used neither Python, nor ZMQ or other general-purpose messaging middleware, nor even OS-provided TCP/IP stack — they all cause too much latency. I’ve been programming for living since 2000, worked in a lot of different stuff from web dev and enterprise to videogames, embedded, robotics and GPGPU. Yet I can name many huge areas which I hadn’t seen close enough, or at all, along with libraries and tools used by people working there. Every time I start working in a new area, or when I resume working in an area after a long (years) pause, I read a lot of relevant stuff. Continuous learning is the key to stay good, IMO.

RangerScience 7y ago

At the job I'm at, I've picked up three tools either for the first time, or in a very new way: 1. Makefiles. See @aequitas' comment for more. 2. Terraform. Seriously, just using this tool taught me [a lot of] devops. It's fantastic! 3. Docker (as a tool!) I'm going to go into the third one a bit - I feel like Docker is mostly thought of as useful for deploying things to the 'net (kubernetes, ECS, etc), but I think it's also amazing for local development and build pipelines. I actually have no idea one way or the other how much other people use it this way as well, so maybe it's just me that's finding it unexpectedly awesome for this. Put together the right CLI command + Dockerfile, and you can hand someone a repo and they can launch a complete, reliable development environment in a single command without any other system prep. No more worrying about which dependencies need to be installed in what way; it's like `rvm` + `bundle exec` but for EVERYTHING. No more dealing with whatever custom system modifications someone has going on. `git clone`, `make dev` move on with life. And then you can also have Dockerfiles that are specifically for producing your build artifacts, and then completely ignore the container for execution. This is how I'm using both AWS EMR and Lambda.

akx 7y ago
I'm also occasionally using Docker t generate build artifacts (so +1 for that) - how do you pull the built blob out of the image? I've used `docker exec` plus `docker cp`, but it feels a little clunky.
- victornomad 7y ago
  You can just mount a volume and write the blob into it, very easy and convinient!
  - akx 7y ago
    Ooh, right, you can mount in `docker build`? That'd solve it, thanks!
    - RangerScience 7y ago

pankajk1 7y ago

Here is a list of technologies/tools that I have found quite enabling (in addition to what has already been covered): 1. Earlier use of VMs and now Docker containers -- It was around 2004 that I got introduced to VMware tools to create, configure and run VMs. Life was never the same after that. No more fretting about installing pre-beta software for the fear of hosing my perfectly working system. There was a time when I had multiple servers running VMWare hypervisor, each running multiple systems. Then around 2014 I switched to using Docker containers for similar purposes and haven't had a need to use VMs. 2. Jupyter notebooks/pandas/plotly -- Can't imagine how one can explore data without this. 3. SQLite -- Perfect for writing unit tests for code that deals with SQL databases.

Khelavaster 7y ago

Visual Studio, especially its visual UI designer, build & reference management, and scaffolded website development (ASP.Net MVC with Entity Framework from scaffolding). Night and freaking day. Also, WCF for all your inter-computer communication needs. I wouldn't be developing software today if I weren't introduced to low-trouble application development with Visual Studio

Khelavaster 7y ago

It's absolutely critical to learn object-oriented programming in a mature language like C#. (Java's OO model is slightly broken, as well as fundamentally underdeveloped; it's better tackled after learning a correct, functional object0oriented language like C#.)

lowken10 7y ago

Anki!!!

sycdan 7y ago

I use ack (https://beyondgrep.com/ https://beyondgrep.com/) many times per day, and it has saved me a great deal of time. That said, if you use IDEs (I don't), it may be somewhat redundant.

azhenley 7y ago

Usability engineering: https://en.wikipedia.org/wiki/Usability_engineering https://en.wikipedia.org/wiki/Usability_engineering

bradford 7y ago

I've extensively used Diff tools as an analytical aid across many different domains. It has uses far beyond code change tracking! Specifically, I like vim's interactive diff'ing capabilities (although any interactive diff tool with sufficiently powerful text-editing capabilities should suffice). So much of the troubleshooting that we do in programming is asking "this thing used to work, what changed?". Don't rely on your eyes to find the changes, let the computer do the work for you. The ability to load up two different log files in a diff session, regex-substitute the noisy portions away (dates, process/thread id), and view the key functional changes really helps me go from noise to signal in an optimal way.

reificator 7y ago
You touched on it briefly but I'd like to highlight regular expressions in day to day editing and log delving. It's a massive time saver in my experience at least. Coworkers often come to me to help them write a quick regex for something, or to have me double check their work. If you need a playground to get comfortable, https://regex101.com/ https://regex101.com/ is a great resource. Dump some examples you'd like to match and some you don't in the bottom section, and try to write a regex that matches in the top. It will dynamically match as you type, and the right side shows a token by token breakdown of what your regex does.

6nf 7y ago

SQL Stored procedures and triggers. It's some of the most useful stuff you may not have considered if you're just a hobbyist. It's not for every project but if you do need a proper database for some reason, there's often great ways to use stored procs or database triggers that greatly simplifies your system.

i_feel_great 7y ago

Tools you can use to recover from the loss or destruction of your laptop/desktop as fast as possible. Or having to hand over your password to the authorities. This involves compressing all of the state and data and removing and/or sending it somewhere, and then being able to recover it quickly. And it must be simple, robust and automated.

glangdale 7y ago

Satisfiability Modulo Theory (SMT) and a nice friendly implementation like Z3. When you have a difficult algorithmic problem and can't be bothered to dream up a good solution, why not try chucking it into a solver and see if a computer can solve it for you? Surprising what it can find - not just Sudoku puzzles and "Mr Green lives next door to Mr Black, the baker lives across the street to the butcher, ... " type stuff either.

travisjungroth 7y ago
Here’s a great talk by Raymond Hettinger on those things and more: https://m.youtube.com/watch?v=_GP9OpZPUYc https://m.youtube.com/watch?v=_GP9OpZPUYc

keyle 7y ago

The design of everyday things. It's a design book that will make you a better programmer too.

new4thaccount 7y ago

SQL (even if just SQLite) as databases open up a lot of power. Vim or Emacs for powerful text editing. A low level language. Sometimes Python doesn't cut it, or it is pretty suboptimal. If you're writing a trading bot, is speed of execution not important? Operating System knowledge can be helpful at times. I bought one of the No Starch books "How Linux Works" and it is very helpful. The command line and by that I guess you should know the common Linux commands (cat, grep, sort, uniq, head, tail, ls, top) if you use Linux and how to chain them together via pipes. To give some context, I can write one command which would require 8 lines of Python (saves you valuable time). If you use Windows, learn enough Powershell to be comfortable with it. On occasion I'll use Powershell over Python even though it is dirt slow for reading files.

afarrell 7y ago
I thought this course did a good job at getting SQL to stick in my brain, largely due to the relational algebra section. https://lagunita.stanford.edu/courses/DB/2014/SelfPaced/about https://lagunita.stanford.edu/courses/DB/2014/SelfPaced/abou...
- new4thaccount 7y ago
  The W3Schools site is what helped me https://www.w3schools.com/sql/ https://www.w3schools.com/sql/ It has a database you can query and add to and what not.
- artimaeis 7y ago
  Firmly agree -- I'm pretty sure I've said it elsewhere, this was a transformational course for me. I got my start in product support where a lot of our problems could be solved in SQL. We were encouraged to learn it, but most people were happy with the most basic syntax for DML. I worked through the entirety of Dr. Widom's course and it gave me the fundamental data understanding to be a real value to the teams I've worked for, and I used that opportunity to transition into being a full time developer.

HocusLocus 7y ago

Perl one-liners, the ones that overflow to several lines of code in your terminal.

funkjunky 7y ago

When I went from amateur Python programmer to Google Cloud developer support, I remember being completely blown away by technology and design patterns I've never heard of that goes into modern web/enterprise architecture. I had to learn it all the hard way, but these days there are great free (or mostly free) courses you can take to learn this stuff. For Google, check out the study guides for their certifications, specifically Cloud Architect (basic overview), Cloud Developer, and Data Engineer. https://cloud.google.com/certification/ https://cloud.google.com/certification/ Be sure to follow along with the recommended Coursera and Qwiklabs tutorials and do the exercises. You'll learn about all kinds of neat stuff, like scalable application design, container technology, monitoring+metrics, various types of database technologies, data pipelines (including Pub Sub messaging), SRE best practices, networking+security, and machine learning. I currently work on AWS, and don't find it a good starting point for diving in to these things quickly, but most companies use it so it wouldn't hurt to learn I guess. I still recommend GCP over AWS to start with, as their technology is far more interesting and focused, and quicker/easier to work with.

hammerbrostime 7y ago

One minute ago I discovered that HN comment threads are collapsible by tapping/clicking the "[-]" on the right side of a comment header. I think that could fall under the class of non-obvious tools (maybe not for a hobbyist, but as a long time HN reader I'm shocked I never noticed it before)

vast 7y ago

Automated deployment. Even for smaller projects. Ansible for example is a python product. Online schema change tools if you use RDBMS in production.

Zeroverse 7y ago

Debuggers, probably. I've always had so much admiration for other programmers who have taken time to learn their debugger and just jump into breakpoint-land when something happens. It's how we're meant to think!

GuB-42 7y ago

Not one in particular besides the obvious. If it is really fundamental, it is not overlooked. Every programmer has their favorite tools. Some will use debuggers, others will prefer logging. Some will use class diagrams, others will grep. Some will use IDEs and GUIs, others will use text editors and shells. Same thing for programming languages, techniques, libraries, frameworks, etc... So you are very likely to get a lot of answers. It seems that you got an epiphany when you learned about message passing. All programmers had similar experience as they discovered the one thing they really needed. In reality it depends on the project. Great programmers simply have a lot of experience with many, many things. My suggestion: continue what you are doing. Try new things, don't blindly follow other people lists. In the process that led you towards that messaging library, you learned named pipes, unix sockets and shared memories, all useful, and maybe they will be essential to your next project. Had someone else served you that library on a platter, you would have lacked that insight.

AaronFriel 7y ago

Devops; being able to stand up your entire stack in an automated way. Some programmers dismiss this as "yaml/bash engineering", and that's true, but it also challenges all of your assumptions in your stack. If you can't stand up a complete duplicate programmatically, you almost certainly have implicit assumptions that you haven't verified, which will make it much harder to recover from disaster or scale. Put another way, devops is a mix of declarative programming (YAML) and writing idempotent, imperative code in the presence of large side effects (bash). Learning to handle both is very educational.

mooreds 7y ago

I would say studying any of the AWS paas offerings. Not saying you have to use them, but they cover a large segment of the system component space. They can help you answer questions like "When would I use an in memory cache vs a rdbms vs a key value store?"

AaronFriel 7y ago

Value/message/actor/event oriented programming like you mentioned is useful for building distributed systems. I am a huge fan of going a step further and learning this model: Imperative shell, functional core. The external shell of code in a project is responsible for network connections, console IO, etc. But the internal guts of a program should be largely functional, that is, instead of mutating (changing) values, consider returning different forms of the same value. Decisions (branches, logic) are made at one level, data dependencies at another. The talk Boundaries by Gary Bernhardt describes this model in detail: https://www.destroyallsoftware.com/talks/boundaries https://www.destroyallsoftware.com/talks/boundaries

btown 7y ago
I'm also a fan of "imperative shell, functional core, imperative implementation of that core," where within a function that you promise to have no side effects or very specific side effects (say, a component's render function, or a transformation from one complex data representation to another), you should still feel free to use loops, imperative-style control flow, even network requests, etc. I find this puts folks used to "imperative everything" at ease, while still maintaining almost all of the benefits of functional-core provided that execution is scheduled properly.
- lenticular 7y ago
  It's really easy to do this with pure functional programming in impure languages like Scala. You can use arbitrary impure code within lazy IO values. Outside, the functional purity makes reasoning easy. If, as recommended, there is only a single point in your program where the IO values are actually executed, then the execution order can be reasoned about statically.

gitgud 7y ago

Design patterns were enlightening to me. The idea of expressing concepts in software design which can be reused was what got me out of the stone age of programming. You can describe Pub/Sub or the Builder Pattern and quickly implement it into a project and know it's advantages and limitations, definitely worth checking out if you haven't already. http://wiki.c2.com/?CategoryPattern http://wiki.c2.com/?CategoryPattern

codr7 7y ago

Macros, interpreters and embedded languages [0]. [0] https://github.com/codr7/g-fu/tree/master/v1 https://github.com/codr7/g-fu/tree/master/v1

greenyouse 7y ago

This might not be for a hobbyist since the setup is difficult but validating your frontend work with automation is very interesting. I didn't realize for a long time that I could use visual regression tools and Selenium at the end of development to catch bugs. I'm surprised that there aren't more tools in this space targeted for developers. Selenium Grid with WebdriverIO is incredibly helpful. If you put together a visual regression system for validating your daily work you could save some serious time.

eranation 7y ago

Not tools but more of a mindset: Application Security, OWASP top 10, Web Security. Learn what is XSS, CSRF, what is a CSP, why HSTS exists, why CORS allowing * can be dangerous. Also what is OAuth, OIDC, SAML, etc. How to store passwords securely, how to add security to your devops cycle. Having a good security mindset can be a great asset to any team.

ww520 7y ago

Learn Emacs. Stick with it to get comfortable. Text editing is one transferable skill that you can reuse again and reuse in different projects and on different languages. Org-mode in Emacs. My poor man's project management for a number of projects consists of a todo-<project>.org file listing all the planned features, the pending TODO items, the DONE items for the current working release, and release notes describing features and changes for each of the released version. In one place, I have the future features, immediate todo items, current completed item, and the history of all past releases in an org file, making things simple to access and manage. For theoretical stuff, learn about transaction.

jtolmar 7y ago

1. Profiler. There's a standard tool that tells you what part of your code is slow. Over half the time it'll find something dumb and easy to fix instead of whatever you expected. 2. SQL / relational database schemas. Persistence opens up a lot of capabilities. And databases themselves are very well-optimized; if you do any nontrivial data manipulation it's likely that whatever the query planner comes up with will be faster than your first idea of how to do it by hand. 3. Graph searches. An awful lot of problems can be solved by knowing how to turn problem into a graph search. Make sure not to fall into the trap of thinking a graph search is limited to paths through space - you can solve problems like "get through this dungeon with keys and doors" by adding duplicate nodes for the different states. 4. Sequential Bayesian Filters. Are almost as useful as graphs, but aren't in a standard CS curriculum so you'll look like a wizard. These solve the problem of "I want to know a thing and I know how it changes over time, but I only can get rough estimates for its current state." Kalman Filters are simple and give great results when applicable. Particle Filters have lower quality but are applicable to more problems and dirt simple to code.

enangel 7y ago
> Particle Filters have lower quality but are applicable to more problems and dirt simple to code. Particle Filters have lower quality than Kalman Filters?
rassibassi 7y ago
Support for 4! Yet, my understanding is that particle filters are superior but computational more demanding. For nonlinear problems, the extended Kalman Filter linearizes the task, whereas particle filters don't and work with many point estimates instead. I loved this book: https://users.aalto.fi/~ssarkka/pub/cup_book_online_20131111.pdf https://users.aalto.fi/~ssarkka/pub/cup_book_online_20131111... and also Thomas Schoen group does great work on Sequential Monte Carlo (SMC), MCMC for sequential data :) http://user.it.uu.se/~thosc112/index.html http://user.it.uu.se/~thosc112/index.html They are also building a probabilistic programming language for sequential data! https://github.com/lawmurray/Birch https://github.com/lawmurray/Birch
- 7y ago

achenatx 7y ago

There are so many specialized areas that are useful, like common design patterns, basic data structures, various algorithms, transactions, multithreading, parallel processing, etc. Too many things to list, but I agree with a poster below that suggested looking at the course offerings for a computer science degree. You might be able to find outlines of the courses to get specific topics. On a day to day basis, code coverage is something that more people should use when writing unit tests to ensure they have tests that execute all their code.

quickthrower2 7y ago

Tools that have changed how I think about things, and yes some are hyped things you have probably heard of. But the opposite effect of people ignoring it because it's too hyped might come into play: Docker - figuring out what you really need to get something running and having a reproducible way of doing it. I did a blog post on it: https://stackabuse.com/how-docker-can-make-your-life-easier-as-a-developer/ https://stackabuse.com/how-docker-can-make-your-life-easier-... Node JS - much derided, but nothing compares in terms of the speed you can quickly hack up a tool in Node JS and the ecosystem you have access to. I don't use it as production server, but just as a quick way to hack up tools. Pandoc - being using this tool to convert markdown to PDF, and it does a nice job. Uses Latex & friends so you can find a nice template somewhere and base off that. Markdown - I love using markdown formats. Especially with the tooling, things like Pandoc and Github are enough to justify learning a bit of MD. Touchtyping - Makes typing a bit nicer and a bit faster. I could type without looking at the keyboard before, but now I don't lose my hands, just feel the bumps!

jasonhansel 7y ago

Shell scripting (trust me, this is way more useful than you might think).

alphaoide 7y ago

This was recommended in a blog post I read. I haven't read it myself but the table of content looks promising https://www.educative.io/collection/5668639101419520/5649050225344512 https://www.educative.io/collection/5668639101419520/5649050... Designing a URL Shortening service like TinyURL Designing Pastebin Designing Instagram Designing Dropbox ... Key Characteristics of Distributed Systems Load Balancing Caching Sharding or Data Partitioning ...

ilaksh 7y ago

The premise that self-taught programmers necessarily have core holes in their knowledge and skills that would have been filled if they had a CS degree is entirely false. Start with the example you gave of messaging middleware. There are many BS CS curricula that do not address this at all. Also he mentioned that he had already learned about named pipes on his own. For many applications, named pipes could be a perfectly valid alternative to some external message queue system. Looking at the items submitted, the vast majority are core skills that would necessarily be picked up by people who need to work with them. The idea that someone would not know about Makefiles or debugging or profiling or SQL just because they were a hobbyist or self-taught is ludicrous. If you are serious about C programming, whether it's a job or your hobby, you are going to learn about Makefiles. Likewise anyone seriously working on a data-centric application is likely to become well-versed in some database technology, up until a few years ago that would have automatically been relational. And one other thing. Some of the most important skills in programming are in the domain of software engineering. Software engineering is very poorly addressed by many BS CS programs. So again, whether they have good SE skills is often not going to be determined by whether they have a BS degree or not. It's not even necessarily determined by whether they are working in a professional environment. It's mainly going to be a factor of their motivation to self learn and above all practical experience.

djhaskin987 7y ago
My experience has been quite different. True that some of the most technically skilled programmers I know of had no degree, but the polished ones, the ones I find easier to work with, tend to have one. Further it's pretty easy for me to tell if a person's degree was a CS degree or not just by talking to someone about the problems they have and how such problems might be solved with code. That's not to say it's required; some of the best professionals I know have non-CS degrees (one in fine art -- painting) or no degree. But if you're still young, I submit that a CS degree is totally worth your time.
- ilaksh 7y ago
  The argument that I was making was clearly stated at the beginning of my comment. It is significantly different from the supposed argument that you seem to be refuting. Notice how I did not say that "the developers that certain degree holders find to be most 'polished' and easiest to work with will on average not have a degree". Notice how I also did not say that a CS degree was not worth people's time.

jdavis703 7y ago

I’m not sure if I count as self taught or not since I first learned programming outside of formal education, only later getting a degree in IT (for which I really didn’t learn anything new except graph theory). But we never talked about queues in school, in fact I only learned about them when I started working professionally. If you want to be exposed to all kinds of interesting problems, I would suggest working at some mid-stage startup (i.e. around the B or C rounds). They’ll be starting to get in good people who are fixing up the mess the early employees made* and can help you learn why certain patterns are anti-patterns and how to fix them. At this stage they’re still small enough that if you pay attention, you can just learn by paying attention to what everyone else is working on. * Before you downvote me I’ve contracted for pre-seed startups, am currently working at a seed stage startup and have been at companies all the way from a series A to a series E. So yeah I’ve been the one making a mess (because of the whole minimum part of MVP) and cleaning up said mess.

whalesalad 7y ago

Low level unix tools. Learning how to read a file, tail a file as it grows (like a logfile), or look at just the beginning, or just the number of lines in the file, or merge a bunch of files into one, etc... How to get a file from one machine to another with scp. How to use more advanced features of SSH like agents, forwarding, your config file, SOCKS proxying, etc..., How to debug system issues, find where config files should be, find out why your app wont compile. Learn how to install code from source, using configure and make. Learn how to operate your own basic network services like HTTP servers, mail servers, local file sharing with NFS or SMB, etc...

smoe 7y ago

Monitoring From the beginning of my career in web development I worked in companies that had good to great monitoring capabilities in place to log and measure performance, resource usage, availability, errors, etc. in all parts of the system. so I took it as something self-evident that you would add that first thing, before deploying a new project. Only recently as I started mentoring people in other companies did I realize that many developers or not aware of what exists in this space. In two cases I got asked for help to deal with performance issues, and in neither case did they have any idea what was causing them, because they had pretty much no tooling to tracking it down, so they resorted to speculation. We installed a application monitoring solutions and they were able to fix the problem in no time once it was identifiable.

alphaoide 7y ago
Can you recommend books/tools? Thanks!
- smoe 7y ago
  This chapter of the Googles SRE book gives you a good overview. https://landing.google.com/sre/sre-book/chapters/monitoring-distributed-systems/ https://landing.google.com/sre/sre-book/chapters/monitoring-... But I think it is less abut what specific tool to use, but more to just get started with one and learn how to understand your production system behavior and dependencies from metrics and graphs. An easy way to get going for monitoring an application is using a hosted solution like newrelic.com Their free version should be sufficient for a very long time. If you want to run it yourself there are opensource solutions like prometheus.io or riemann.io among many others.

ajflores1604 7y ago

tldr: redis Hey man, I went down an eerily similar path as you. Seld taught and building a trading system. Wading thru sockets and pipes before discovering zmq and having my whole paradigm for programming completely shifted. Absolutely love zmq, and enjoy thinking of new projects to use with it. More towards your actual question, I don't think it's as ground shifting of a discovery as zmq, but Redis has also helped me a lot with the trading bot. I use Interactive Brokers, and how their api examples are set up, I haven't figured out how to design my system using only message requests like you see in micro-service style programs. Also when requesting a price for any given contract, I don't like the idea of having to request it once I need the price and then waiting on the network for the brokerage to receive my request and then the price to come back to me. By that point the price or whatever data you're requesting can get stale depending on the speed and timefrime you're wanting to play. So what I did was just request ahead of time streaming data for contracts I'm interested in, and I write that continuously to a localhost redis server. And then whenever any of my bots need information, they check to pull the latest value from that redis server first before going out to the brokerage platform to request the data directly. Basically cuts the full round trip time in half if it can find the data locally first. I believe this is a similar programming paradigm as using redux if you have experience with that, but I've never personally used redux so don't quote me on that.

nathanasmith 7y ago
Absolutely agreed. Redis is a godsend for small time traders like us. I've scaled up to where I'm trading on about 20 crypto exchanges in addition to dabbling in stocks and prior to redis I was using flat json files to store all my data so every time a new price came over the wire the bot would read the file, parse the json, append the price then write it all back out. That was quick and easy to put together and it worked great at first but later when I had all the exchanges going even with an 8700k and Optane drive the computer would just fall over when volume got high. Got sick of that, converted it all over to redis with rejson, added another 32 gigs of RAM and problem is utterly solved. Redis' built-in queue has been extremely useful too as a job system.

xs83 7y ago

DEBUGGING! And I don't just mean echo'ing out statements to check the values, I mean an actual attached debugger to your code. GDB for C / compiled languages XDEBUG for PHP etc The actual act of stepping through your code and looking at the values, datatypes and their transitions massively increases your productivity if something is tricky!

lixtra 7y ago

For learning what’s going on in compiled programs strace can be very helpful. https://www.tecmint.com/strace-commands-for-troubleshooting-and-debugging-linux/ https://www.tecmint.com/strace-commands-for-troubleshooting-...

croo 7y ago

Did you know there are document storage/management systems out there? Its useful when one needs to save a lot of documents(duh) or other too-large-to-save-in-rdbms type of content with metadata. Examples are FileNet or Alfresco. Did you know there is a protocol called CMIS [1] which you can use to query content from these systems? It's similar to a subset of SQL. [1] https://en.wikipedia.org/wiki/Content_Management_Interoperability_Services https://en.wikipedia.org/wiki/Content_Management_Interoperab...

contingencies 7y ago

Anything that automatically generates pictures from text: mscgen, imagemagick, graphviz, SVG, povray, gnuplot, R, etc.

nprateem 7y ago

I used to read High Scalability [1] which has write ups of real-life architectures of lots of the largest web sites. I found it great for learning about different architectures. I'd definintely recommend it as a way to find out about thing like messaging queues, asynchronous vs synchronous processing, etc. On a similar note but more theoretical, my life changed when I learned about design patterns. Martin Fowler's books/site are a good place to start [2]. [1] http://highscalability.com http://highscalability.com [2] https://www.martinfowler.com/eaaCatalog/index.html https://www.martinfowler.com/eaaCatalog/index.html

olingern 7y ago

Just a few that I've found liberating over the past few years: 1. In memory caches for small projects / one-offs. Redis and memcache are sometimes not needed. Wrapping your cache in an class whose storage is dependency injected is a nice way to keep moving along. --- 2. grep Not much to say here other than it's a great search tool. Admittedly, I'm still learning working on wielding it even better --- 3. Piping output: [CMD] > myoutput.txt Lifesaver for large output.

jondubois 7y ago

Corporations have been pushing hard on marketing their vendor-lock-in as-a-service solutions so you don't hear about simple free open source tools anymore; all developer channels are saturated with corporate marketing. So much so that you don't even realize that it's marketing.

shiloa 7y ago

An IT automation framework like Ansible (my favorite), Chef, Puppet, Salt. Highly recommended if your work includes doing repetitive remote server tasks or any sort of infrastructure / devops stuff.

fa 7y ago

Is this trading bot you’re writing available online?

geirman 7y ago

Learn to use Proxy Servers for Troubleshooting Whenever your web app, mobile app, etc depends on a network response, Charles Proxy (https://www.charlesproxy.com/ https://www.charlesproxy.com/) can be super helpful. It sits in between and captures all the HTTP request/responses and allows you to manipulate them. So, you could capture an API response (or request), manipulate it, then let it continue. It also let's you interrogate the requests easily.

anonu 7y ago

> Googling, I encountered something I hadn't heard of called a message queue. Former high frequency trader here. Messaging middleware is a god-send for distributed systems. It used to be quite commonplace in the late 1990s/early 2000s for creating trading systems. By mid-to-late 2000s - and especially post GFC (great financial crisis), volatility lowered, technology improved and middleware systems were never again put in the critical path. The important measure here is "tick-to-trade" - from the moment a market-data message comes off the wire to the moment you send an action to buy/sell/cancel back on the wire. A middleware system just slows this down considerably. As a result, you want "tick-to-trade" to be in the same process, preferably single-threaded and boost your thread affinity to minimize context switching. To answer your actual question: I would say learn about the concepts that the big cloud providers are pushing. If you open the AWS app drop down - there are dozens of concepts that have been encapsulated in managed or serverless frameworks. They are all worth learning IMHO - as they define the current and next generation of computing.

tynpeddler 7y ago
Message queues are really cool, but to paraphrase an old joke, "Some people see a distributed systems problem and think, "I know, I'll use a message queue." Now they have otw problems.

tosh 7y ago

* relational databases (e.g. sqlite, postgres) * regular expressions * lisp / scheme / clojure * emacs * mini kanren / datalog / prolog * neural networks and deep learning * state machines * caches

mntmoss 7y ago

Some simple things with good ROI: 1. Ergonomics tooling: Set up your screen at an appropriate height and your keyboard at an appropriate distance, so that you aren't in pain. If you use a laptop often, try setting it on a shoebox. Use a work timer to take breaks(I've been using Workrave on its default settings lately, which is quite harsh but has worked for my productivity). 2. Diary keeping: Note what you did, what you plan to do, and the date and time. Note as often as you feel necessary. Record notes both in source comments or in general purpose diaries. Use the date to eliminate or rewrite notes that are stale. 3. File management, process management and editing. Take a little time to learn things about your operating system and editor. You don't have to master it, or do elaborate configuration(in fact, having a complex config makes it hard to transfer to other environments). What you want are little things like knowing a few handy shortcut keys or a few built-in tools. 4. Gaining familiarity with writing simple "skeleton" or mock-up code, and waiting patiently for it to mature. When first building a system it may be tempting to apply the biggest algorithmic hammer or design pattern you know of. This is the kind of trick you are learning in learning about message queues. But the most likely outcome of trying to use a trick, no matter how well intentioned, is that this will get you a wrong result more slowly, because the full shape of any problem tends to come into view slowly, progressing from blurry and unclear with a rapidly changing design into something sharp, with well-defined boundaries. As such every new system demands a beginner's mindset, and some ability to refuse engineering in-depth until you absolutely must do so to progress. Done properly, you build a system that leverages existing tools well, has some kind of value now(even if it's limited or lacking a critical feature), and then can harvest the learned lessons into a more complete form later. Trying to get all the features into one pass creates a messy soup: iterating on a subset of them naturally leads towards development of tricks like the message queuing architecture, without any prompting. (edit) 5. Read c2 wiki when bored - it covers a lot of recurring discussions in programming: http://wiki.c2.com http://wiki.c2.com If you know the stuff in it you'll be reasonably prepared to think about any unusual ideas and compare them with existing examples.

nojvek 7y ago

As a programmer, learning how to use a user analytics tool really shift my perspective on things. Profilers and debuggers are tools that tell you where to fix the code. A user analytics tool tells you where to fix the product. Being able to hypothesize ideas, measure conversion, engagement, retention, A/B test, slice by cohorts and validate whether an experiment validates your hypothesis. It makes you tackle problems very systematically. Being able to measure both impact and effort. Always evaluating whether the impact was worth the effort and fine tuning it. The product version or tree falling in the forest joke: If you build something and no one uses it, does it really exist?