10 ms·
Web application from scratch, part I
- ggambetta 9y agoI love seeing projects/tutorials like this. Too many developers treat too many layers of the stack as "magic", and this is great to show it's not so. We should change the definition of "full-stack developer" to "given a computer, an opcode reference sheet, a way to enter bytes into a computer, and enough time, is capable of making an operating system and a bunch of applications" :)
- deleted 9y ago[deleted]
- reificator 9y ago> given a computer, an opcode reference sheet, a way to enter bytes into a computer, and enough time, is capable of making an operating system and a bunch of applications Yeah, as long as “enough time” is measured in years I could still be a fullstack developer under that definition. There's still a good reason to treat layers as magic though. Mental working space is limited and abstractions with minimal leakage are the only way to get things done at ever increasing speeds.
- jonbarker 9y agoThis comment reminds me of Feinman describing how he out-computed an abacus expert just extrapolating from a few facts he had memorized. I think mental working space is limited and should be optimized on (as in, don't load your mind up with too much low level stuff), sure, but it's the ability to synthesize known tool sets and their capabilities rapidly that helps more. Learning 'another way of thinking about it' is generally the value of doing these first principles style tutorials although I think one can take it too far.
- DanielBMarkham 9y agoYep. And what happens far too often is developers concentrate on one layer and framework, then spend all their time learning tiny details about the abstraction system they're using instead of the job they're doing. New version comes out, something breaks in a non-obvious way, the industry moves on? They're left with a head full of nothing.
- reificator 9y agoThat's less about layers of abstraction and more to do with intellectual laziness in general. Abstraction is a tool, and like all others it can be abused. > All problems in computer science can be solved by another level of indirection - except for the problem of too many layers of indirection.
- DanielBMarkham 9y agoI agree it is about intellectual laziness. I was a developer for a decade or two before I finally realized something critical to my career: other developers are actively selling me on things that may or may not be a net benefit to me as the years pass. Developers are a market just like any other, so people make abstraction layers and then become "evangelists" that go out and sell people on stuff. What's to sell? The idea that the framework takes away all the hard work. (I was a big sucker for frameworks with slick UIs back in the day.) We want to believe with a minimum amount of thought, something super cool that will come out that users and other developers will fawn over. Once I started teaching developers, I ran into people who called themselves professionals but were unable to think through simple activities. Yes, these people were lazy, but they were also products of an environment where they were just moving from shiny object to shiny object. They knew a little bit about various abstraction layers here or there, but nothing about how anything worked. Most of the time, once they went off "happy path", they just banged away at the compiler until things looked like they were working. And these were people being paid to program! I completely agree this is because of laziness. I don't think it's the entire picture, however.
- 9y ago
- ggambetta 9y agoI'd say there's good reasons to treat some layers as black boxes, but not as magic; you use abstractions precisely because mental space is limited, not because the underlying object is intrinsically non-understandable. As I replied to another comment below, I wrote "is capable of doing", not "does".
- avyfain 9y agoThis made me think of GEB: > ... we all know that we human beings are composed of an enormous number of cells (around wenty-five trillion), and therefore that everything we do could in principle be described in terms of cells. Or it could even be described on the level of molecules. Most of us accept this in a rather matter-of-fact way; we go to the doctor, who looks at us on lower levels than we think of ourselves. We read about DNA and "genetic engineering" and sip our coffee. We seem to have reconciled these two inconceivably different pictures of ourselves simply by disconnecting them from each other. We have almost no way to relate a microscopic description of ourselves to that which we feel ourselves to be, and hence it is possible to store separate representations of ourselves in quite separate "compartments" of our minds. Seldom do we have to flip back and forth between these two concepts of ourselves, wondering "How can these two totally different things be the same me?" -Douglas Hofstadter, "Gödel, Escher, Bach", Chapter 10
- reificator 9y agoAs I replied to another comment below: I understand the distinction you're making but in my experience the words are interchangeable. Just because I treat a system as magic one moment does not mean that I can't later dive under the hood and fix it the next. It's just a matter of context switching.
- jonnydubowsky 9y agoMany thanks to Reificator, avyfain, and ggambetta for the string of comments which synthesize so much of how i think about cyberphysical systems, but in a way that has crystalized it and given me a breakthrough in writing for my research. I'll ping you the result of this if you are open to a dm. Either way, you've given me a gift in your comments and bogdamp, awesome post! Quite pure and actionable insights.
- maxxxxx 9y agoYou shouldn't treat anything as magic. Whenever you look under the hood of things that look like magic you always find that they have been built by people like you and me and are perfectly understandable if you put in the time. For example i always thought that Windows was developed by people who are much more qualified than myself but when I saw some parts of the source code I quickly realized that it was written by human beings without any magic. Treat things as a black box but never as magic.
- reificator 9y agoI understand the distinction you're making between black boxes and magic. However, in my vernacular they're the same thing. I can describe something as magic to one group and describe the internals to another group without issue. If magic wasn't a black box it wouldn't be magic.
- tonyarkles 9y agoHeh, I jokingly refer to myself as a full-full-stack developer. I can do schematic design and board layout, baremetal or RTOS embedded software, Linux kernel hacking, etc all the way up to web apps with React (preferring Elixir on the backend but shrug).
- pavel_lishin 9y agoSurely you mean "given a stack of microchips and lengths of wire"?
- amelius 9y ago> Too many developers treat too many layers of the stack as "magic", and this is great to show it's not so. My high-school maths teacher referred to it as "just plumbing".
- blunte 9y agoI assume this is sarcasm. Those who do not build upon the achievements of others are far less likely to reach any new heights. It's good to remember what we take for granted, and it's good to understand what is happening in layers below our concern. But pragmatically speaking, it's more valuable to know how to study and understand what's beneath us when we need rather than spending time proving ourselves just for the sake of proving. Do you still hunt, kill, and butcher your own meat?
- ggambetta 9y agoIt's not sarcasm. I don't hunt/kill/butcher my food. But note that I wrote "is capable of doing", not "does".
- ivanhoe 9y agoThere's a huge difference between using the prepackaged black box solution for the sake of efficiency (as everyone should), and not caring at all to understand even the basics of how it actually works. The later usually leaves one too intimidated to change anything and vulnerable to any unexpected situation. And things break all the time, you need to debug problems with tools and servers and run into all kinds of exotic issues. I once worked with a front-end dev who would restart his machine when he needed to reset a local web server, because "it's not his job" to learn a bit about using the command prompt. For me that is plain crazy attitude and it's not that rare...
- vbezhenar 9y agoIf you're buying software, you don't need to understand how it was made. It's similar to meat. If you're producing meat, you have to butcher animals at some layer.
- mathgladiator 9y ago> Do you still hunt, kill, and butcher your own meat? Um, yes? it is not that hard. You do need land for it. I think reaching new heights requires a full understanding of how things work in general. Here is the thing with computers, it isn't that hard to get that knowledge. You just have to apply yourself and build a great deal of things over a career.
- pvg 9y agoI like this kind of tutorial as well but I have a hard time telling who this particular one is aimed at. It's definitely not any kind of beginner. The socket API is nontrivial and is hardly explained at all. A lot of implementation magic is just coded right in without a word of explanation. The shiniest Python wizardry is used throughout without much of a benefit to clarity - you've got a generator, f-literals, type annotations, typed name tuple syntax, etc. Even if you can read and like type annotations as a documentation aid, the code as written says weird things like 'an http request is a a named tuple and depends on sockets'.
- sotojuan 9y agoI'd guess it's written for someone comfortable with Python or programming in general (so they understand the syntax quickly) but has mostly worked with web frameworks and libraries.
- voltagex_ 9y agoI thought I was comfortable with Python but I've never seen `-> typing.Generator[bytes, None, bytes]` nor that arrow syntax before - guess I have some catching up to do.
- jcims 9y agoI came up through sysadmin, networking, infrastructure type work. I always assumed that programmers knew everything I knew PLUS how to write programs. First project where I spent any appreciable amount of time with programmers was eye opening. I was perplexed that not a single one of them had ever run a network sniffer or strace or had the faintest clue about HTTP or how browsers maintained state with their application. I squatted in a room with them for 4-5 weeks and acted a bit like an xray machine for any problems they were encountering. I didn't know anything about programming, so they would tell me what the code was supposed to do and I would show them what was actually happening on the wire or filesystem or whatever. It was a lot of fun and we all left the exercise having learned a lot more than expected. This was nearly 20 years ago, tooling has improved a good bit since then but it still helps to be able to get into the weeds.
- dijit 9y agoThis mirrors my experience and assumptions, I always assumed developers knew more than me- it's not so and I'm glad that we can compliment each other skills and learn from each other. I do wish that the skills of sysadmins weren't so downplayed though. (Companies hiring SWE for Ops positions and the notion of 'NoOps'[0] being examples) [0] https://go.forrester.com/blogs/11-02-07-i_dont_want_devops_i_want_noops/ https://go.forrester.com/blogs/11-02-07-i_dont_want_devops_i...
- analogic 9y agoWell ya, was never able to figure out what a netmask did so went into programming instead.
- n0tme 9y agoI have kind of the same experience being a database administrator. In so many cases programmers have no clue what is actually happening behind their queries. The worst thing is that they think they know or just don't pay attention! :D
- zamber 9y agoHad an opposite experience. Did QA for an Android app which set up a local VPN and selectively pushed traffic to a SPDY proxy doing some magic. Had to do a fair bit of work with Wireshark to troubleshoot protocol issues. It was eye-opening too, especially the amount of traffic generated in the background by Facebook. Sadly, developers rarely go any further than the network tab in a browser debug tools.
- threatofrain 9y agoIn my view, full-stack is principally a business term, meant to communicate how many people you have to hire, or whether you can skimp on hiring.
- flexie 9y agoWhy stop there? What about assembling the computer? In fact, he should be able to make the electronics if he truly knows about computers. Now that I think of it, given a mine, our superhero should be able to find ore needed, and given a field, he should be able to create electricity. Then we have a true full stack developer. One that shouldn’t feel bad about himself.
- twtw 9y ago> One that shouldn’t feel bad about himself. Nobody is suggesting that people who don't know all this stuff should feel bad about themselves, just that striving to understand it is admirable. Also, the idea that you can continue in the same thought process all the way to mining is a little bit weird. Understanding your CPU, its ISA, your OS, etc. are arguably all potentially relevant for software developers. Knowing how to mine is not.
- nottorp 9y agoActually yes, I know how to assemble a computer as well as to do sysadmin stuff (or at least enough to be able to search for the relevant docs). Knowing about the abstraction layers under your javascript helps a lot with efficiency. If more people knew that maybe I wouldn't need 16 Gb of ram just for the browser tabs... Edit: that doesn't mean you should write your own web server every time you do a new web site, but it would help if you do that once to get an idea of what's going on.
- psyc 9y agoCertain Minecraft and Factorio players have done things like this often.
- zentiggr 9y agoThis makes me think that Chuck Moore would be the prototypical full-stack dev... just give everyone a Forth image with core vocabulary and turn them loose.
- r0m4n0 9y agoCouldn’t help but think of this quote... “If you wish to make an apple pie from scratch, you must first invent the universe.” -Carl Sagan
- matte_black 9y agoNow there's an article I'd like to see.
- deleted 9y ago[deleted]
- platz 9y ago> we need to create a socket, bind it to an address and then start listening for connections. This is the most important part. Is there a tutorial on sockets, perferably not written from a C perspective and in a language agnostic way? How OS specific is it? Is there a common basis over most OSes?
- pavlov 9y agoSockets are a C language API because of their Unix origin. Other languages provide wrappers around the underlying C API. If you want to understand something like select() vs. epoll(), there's no way around C.
- deleted 9y ago[deleted]
- oldcynic 9y agoYou could do worse than start with the FreeBSD Handbook section on sockets as it covers a lot of the how and why things are glued together. It is C based however, as *nix and data structures were built in C. [0] https://www.freebsd.org/doc/en_US.ISO8859-1/books/developers-handbook/ipc.html https://www.freebsd.org/doc/en_US.ISO8859-1/books/developers...
- maxyme 9y agoSockets are mostly implemented outside of userspace. It's actually very easy to make a TCP socket in C: https://stackoverflow.com/questions/204169/how-to-create-a-tcp-socket-connect-using-c-to-a-predefined-port https://stackoverflow.com/questions/204169/how-to-create-a-t... UDP is pretty similar but you have to change the code to be connectionless.
- acomjean 9y agoSince the socket/networking code is in the OS libraries its in C. Other languages usually just wrap them. They're pretty similar across OSs (I used to do sockets in linux/hpux and solaris). There was a book I borrowed that was useful, UNIX Network Programming, Volume 1, Networking APIs: Sockets and XTI I found beej's guide useful as well. http://beej.us/guide/bgnet/html/multi/syscalls.html http://beej.us/guide/bgnet/html/multi/syscalls.html
- windlessstorm 9y agoTried similar stuff few months back. Start a simple socket listening at port 80. Use any browser to make connection. After recieving any connection on server, send a properly formated HTTP response ( precede with 200 ok and must include content-length, everything else I found was optional). Now scale this to include custom root locations. Extending this to handle all the specifications the thousands of common protocols a server should handle will now feel like headache, so here I didn't continued anymore. Next I wanted to replace the browser with my own client. Soon realized how huge of a obstacle I was heading towards. Got bogged down by the complexity of a simple browser and still havn't got started on this. Maybe someday :D
- amanzi 9y agoThis is interesting, but for an internet facing server I would rather use an existing server that would be far more secure than anything I could create in an afternoon.
- pvsukale3 9y agoCache link: http://webcache.googleusercontent.com/search?q=cache:defn.io/2018/02/25/web-app-from-scratch-01/ http://webcache.googleusercontent.com/search?q=cache:defn.io...
- bratah 9y agoI get over quota error.
- hiisukun 9y agoHere is the archive.org mirror, since the server is responding with error 503 (over capacity at the moment). https://web.archive.org/web/20180226040031/https://defn.io/2018/02/25/web-app-from-scratch-01/ https://web.archive.org/web/20180226040031/https://defn.io/2...
- Bogdanp 9y agoThanks! I woke up to realize the site was still on the free AppEngine tier and had used up all the bw. It should be back up now. :D
- mar77i 9y agoI think you should use .replace(os.linesep, "\r\n"), in case the system's newlines coincide with CRLF.
- Bogdanp 9y agoIt's been a while since I've used Windows, but does that matter since I'm using string literals here? I would think that even on platforms like that, literal string lines would be terminated by `\n`.
- gkya 9y agoIf I'm not mistaken CRLF is part of the HTTP spec as a field separator. Edit: I've read the article after this comment, now I see what the parent comment meant. I guess in this case sth like "\r\n".join(["line1", "line2", "line3"]) is best.
- gkya 9y agoI think it's useful to know what your abstractions (i.e. the web framework and the web server here) abstract away, so this is really nice material. But for anything you're actually going to publish, using a well-founded framework or library is the way to go, because so many things you can get wrong is fixed by many smart people in such projects (i.e. Django, Rails, Nginx, Apache, &c).
- plopz 9y agoNot to mention the difficulty that would be trying to roll your own https.
- jventura 9y agoA blog post similar to this one (same content) that I did last year. I use it as basis for a small project on how to implement an http server from scratch in my distributed computing classes: http://joaoventura.net/blog/2017/python-webserver/ http://joaoventura.net/blog/2017/python-webserver/
- kazinator 9y agoHere is a not quite from scratch (uses Apache for CGI dispatch) application: http://www.kylheku.com/cgit/tamarind/tree/ http://www.kylheku.com/cgit/tamarind/tree/ Tamarind allows users to manage throw-away e-mail aliases. I've been using this almost daily for a few years. It's self-contained; no framework or external libraries are required other than what is in that directory. It just requires an installation of the TXR language in which it is written. Though it doesn't include its own HTTPD server, it's in a language that I made myself. The arbitrary boundary defining "from scratch" could easily be moved to encompass "own TCP/IP stack and ethernet driver" or "own filesystem", and so on down to the hardware.