4 ms·

> Web development is a solved thing, it seems like. That doesn't interest me. I was in the exact same boat as you when I started out. After a few years of web

> Web development is a solved thing, it seems like. That doesn't interest me. I was in the exact same boat as you when I started out. After a few years of web development, I was getting closer to a sinking suspicion that I was nearing the end of the rope, so to speak... And then the rabbit hole opened up to me. Trust me when I say: the web development stack is very, very deceptive. It would take at least a decade, if not more to fully master every aspect of it. And it hasn't even been around that long (comparatively), which makes finding true experts for our "hard problems" quite difficult. The knowledge that I have to go into work knowing inside and out every day? Domain registration, sysops, provisioning a server from scratch, chef/puppet for configuration management, bash scripting, server security, firewall rules, DDOS prevention, nginx/apache hosting software usage and configurations, the dozen and a half different fucking flavors of wsgi servers and how to properly reverse proxy into them, redundant load balancing with HAProxy and heartbeat, postgres/mysql/nosql database software usage and configuration, database sharding and connection pooling, master/slave and master/master setup advantages and disadvantages, proper schema creation/indexing/querying efficiently, proper static and media file headers/caching/expiration, python/ruby, django/rails, process management ala supervisord/runit/upstart/initd, semantic HTML, flexible CSS through a separate tool like SASS/LESS, javascript and what used to be the giant clusterfuck of front-end engineering, ajax, comet/long-polling/buzzwordsgalore, getting SSL to properly work with all of the above, Solr search engine usage/configuration, caching with varnish and memcached, queuing all over the place with rabbitmq, lightweight key/value stats with redis, proper version control, continuous integration, project management, one-click builds and deploys with fabric/capistrano, and the 1001 insecurities built in to naive web development and how to properly protect against them. The list goes on and on and on. I'm sure I'm forgetting to mention a lot more. Now I know how silly I was back then to believe that I was "done" with web development after a short amount of time setting up toy websites. I still consider myself a novice, and I haven't even scratched the surface of what it takes to make a web site/service scale to massive amounts of traffic. The parent comment from a Facebook dev is somewhat spot-on though: for the hard Computer Science problems, you have to get into the low-end systems level programming. On the complete opposite end of the spectrum, I deal with problems at the macro level. Computer Science, while extremely helpful in some cases, in no way, shape, or form prepared me for the bottomless pit of knowledge required for modern web development. To each their own, but both sides are deluding themselves if they believe that they somehow have the "tougher job" when it comes to programming. The only difference is that a CS degree better prepares you for one over the other, due to the nature of the work involved.

mattdeboard 15y ago
Sorry if I gave you the impression that all I've done is set up toy websites. In your list, just in the past year, I've touched a solid 80%, if not more, of the technologies you listed, professionally. The only ones I'd leave out are database sharding, connection pooling, redis, project management... and I think that's it. That's kind of my point. This stuff isn't hard and oftentimes the things you mentioned are the big obvious solution, because a million people have done it before and blogged about it. That's not to say it isn't really cool. ad hoc multiprocessing with rabbit (or whatever) & celery is awesome. Sass is neat, etc., etc. But by and large they're solved problems. That doesn't make them unworthy of anyone's time, or stupid or dumb or whatever. I didn't say the tech stack is shallow; I said the problems you try to solve on the web are solved by and large, except at the edges where companies like Twitter, Disqus, etc., are doing new things to scale. There are people out there who collect spoons. They have an enormous wealth of knowledge about all kinds of spoons. There is practically a bottomless pit of knowledge about spoons. Their history, origins, etc. That doesn't make it interesting to everyone, necessarily.
- Pewpewarrows 15y ago
  That was partially my point. It doesn't matter if you're doing systems engineering, high frequency finance trading, web development, games development, database systems creation, mobile app development, or working with embedded systems. Most of the problems you run into are solved. But when you work with any of these fields at length, new and largely unsolved problems crop up. That's not unique to people working on cache systems. Whenever you're in a field that runs into scaling/performance issues (which is all of them), you run into hard problems. If you were to do a 180 and go into a job building database engines instead of web sites, guess what? Most of the stuff you deal with has been blogged about, answered on StackOverflow, and is generally a solved problem. Where does it break down into tough programming? Making it perform bleeding fast. Same goes for web development, and games, and so on. It might _seem_ like web development is more trivial than the rest, but that's just a common misconception.
  - mattdeboard 15y ago

mattdeboard 15y ago

Sorry if I gave you the impression that all I've done is set up toy websites. In your list, just in the past year, I've touched a solid 80%, if not more, of the technologies you listed, professionally. The only ones I'd leave out are database sharding, connection pooling, redis, project management... and I think that's it. That's kind of my point. This stuff isn't hard and oftentimes the things you mentioned are the big obvious solution, because a million people have done it before and blogged about it. That's not to say it isn't really cool. ad hoc multiprocessing with rabbit (or whatever) & celery is awesome. Sass is neat, etc., etc. But by and large they're solved problems. That doesn't make them unworthy of anyone's time, or stupid or dumb or whatever. I didn't say the tech stack is shallow; I said the problems you try to solve on the web are solved by and large, except at the edges where companies like Twitter, Disqus, etc., are doing new things to scale. There are people out there who collect spoons. They have an enormous wealth of knowledge about all kinds of spoons. There is practically a bottomless pit of knowledge about spoons. Their history, origins, etc. That doesn't make it interesting to everyone, necessarily.

Pewpewarrows 15y ago
That was partially my point. It doesn't matter if you're doing systems engineering, high frequency finance trading, web development, games development, database systems creation, mobile app development, or working with embedded systems. Most of the problems you run into are solved. But when you work with any of these fields at length, new and largely unsolved problems crop up. That's not unique to people working on cache systems. Whenever you're in a field that runs into scaling/performance issues (which is all of them), you run into hard problems. If you were to do a 180 and go into a job building database engines instead of web sites, guess what? Most of the stuff you deal with has been blogged about, answered on StackOverflow, and is generally a solved problem. Where does it break down into tough programming? Making it perform bleeding fast. Same goes for web development, and games, and so on. It might _seem_ like web development is more trivial than the rest, but that's just a common misconception.
- mattdeboard 15y ago