4 ms·

the speed you gain by being able to whip up a feature in 10 minutes is worth the pain of an occasional dead link This is everything that is wrong with the soft

the speed you gain by being able to whip up a feature in 10 minutes is worth the pain of an occasional dead link This is everything that is wrong with the software industry, summarized in one sentence. Speed gain enjoyed by developers is paid for by the users in pain. It used to be that developers would go through tremendous amounts of pain just to squeeze out a few instructions from a UI drawing routine in order to make it just a little bit smoother for the users. Now those developers are derided as "greybeards." I'm sorry, I probably could have found a less harsh and cynical way of writing all that, but I feel like the Internet is getting worse everyday and there's not enough urgency among tech people.

sillysaurusx 7y ago
Not at all. I share your frustration. Closures on a server are a powerful way of representing data flows. But they come at a cost: the links expire after some time. How do you strike a balance? The simplest way is to put in the extra time to make everything into a persistent link. But, that's equivalent to removing all the benefits of lexical scoping. If you've ever created an inner function before, you know how powerful that technique can be. You can encode the entire state of the program into closures -- no need to store things in databases. Want a password reset link? Spin up a closure, email the link, done. Literally identical to storing a reset token in a database, except there's no database. Another solution is to fix the root problem. Does the closure really need a random ID every time the page refreshes? The closure links die because they have to be GC'd periodically, to keep memory usage down. Even if you cache the page for 90 seconds, that's still 960 refreshes per day for logged-out users. Then if you have a few hundred regular users, that's at least another factor of two. And certain pages might create hundreds of closure links each refresh, so it quickly gets out of hand. Ironically, the solution was emacs -- in emacs, they store closures in a printable way. A closure is literally a list of variable values plus the function's source code. That got me thinking -- why not use that as the key, instead of making a random key each time the page refreshes? After all, if the function's lexical variables are identical, then it should produce identical results each time it's run. No need to create another one. That's what I did. It took a week or so, which is a week I'll never get back for building new features. But at least users won't have to deal with dead links anymore. Clever readers will note a theoretical security flaw: an attacker might be able to guess your function IDs if they knew the entire state of the closure + the closure's source code (which is the default case for an open source project). That might give them access to e.g. your admin links. But that's not an indictment of the technique; it's easily solved by concatenating the key with a random ID generated at startup, and hashing that. I'm just making a note of it here in case some reader wants to try implementing this idea in their own framework. The closure technique has nontrivial productivity speedups (that I think someone will rediscover some years from now). I hope the idea becomes more popular over time.
- kazinator 7y ago
  > How do you strike a balance? How about a keepalive from the client side: little bit of JavaScript that somehow tells the server that the session is still alive, so don't blow away the closure or continuation.
- sillysaurusx 7y ago
  Since this is getting a surprising amount of interest, let me sum up the technique here. It's really not hard to implement it in Javascript using Express. 1. inside of your express endpoint, create a closure that captures some state. For example, the user's IP address. (This is a dumb example, but the point is that you can capture whatever state you want. Some cases are mentioned here: http://paulgraham.com/road.html http://paulgraham.com/road.html and here: http://ep.yimg.com/ty/cdn/paulgraham/bbnexcerpts.txt http://ep.yimg.com/ty/cdn/paulgraham/bbnexcerpts.txt) EDIT: I updated this to capture the date + time the original page was loaded, which is slightly more compelling than a mere IP address. app.get('/', function (req, res) { let ip = req.headers['x-forwarded-for'] || req.connection.remoteAddress; let now = new Date(); let date = now.getFullYear()+'-'+(now.getMonth()+1)+'-'+now.getDate(); let time = now.getHours() + ":" + now.getMinutes() + ":" + now.getSeconds(); let fn = (req, res) => { res.send(`hello ${req.query.name}. On ${date} at {time}, your IP address was ${ip}`) } ... see below ... }) 2. insert that closure into a global hash table, keyed by a random ID. g_fnids = {}; app.get('/', function (req, res) { let fn = ... let id = <generate random ID> g_fnids[id] = fn; res.send(`<a href="/x?fnid=${id}&name=bob">Say hello</a>`); }) 3. create an endpoint called /x which works like `/x?fnid=<function id>&foo=1&bar=2`. Use <function id> to look up the closure. Call the closure, passing the request to it: app.get('/x', function (req, res) { let id = req.query.fnid; let fn = g_fnids[id]; fn(req, res) } Done. Congratulations, your closure is now an express endpoint. Except you didn't have to name it. You can link users to it like `<a href="/x?fnid=<function id>&name=bob">Say hello</a>`. The reason this is a powerful technique is that you can use it with forms. The form target can be /x, and the query params are whatever the user types into the form fields. I bet you already see a few interesting use cases. And you might notice that this makes scaling the server a little more difficult, since incoming requests have to be routed to the server containing the actual closure. But in the meantime, you now have "inner functions" in your web framework. It makes implementing password reset functionality completely trivial, and no database required. If it seems slightly annoying to use – "I thought you said this was a productivity boost. But it's annoying to type all of that!" – lisp macros hide all of this boilerplate code, so there's zero extra typing. You can get lisp macros for Javascript using Lumen lisp: https://github.com/sctb/lumen https://github.com/sctb/lumen Even without macros, though, I bet this technique is shorter. Suppose you had to store the date + time + IP address somewhere. Where would you put it? I assume some sort of nosql database like firebase. But wouldn’t that code be much longer and more annoying to write? So this technique has tremendous value, and I’m amazed no one is using it circa 2020.
specialist 7y ago
Yup. I somehow became a graybeard. I really didn't fit in at my last three gigs doing "backend" work. I always play to win, so try to understand why & how I failed. My current theory: I had good successes doing product development. Shipping software that had to be pretty close to correct. Today's "product development" is really IT, data processing. Way more forgiving of stuff that's not quite right. Often not even close to right. (Guessing that about 1/3rd of the stuff I supported didn't actually do what the original author thought it did, and no one was the wiser, until something didn't seem quite right.) One insightful coworker said it best: "I learned to do everything to 80% completion." My observation is that most team mates created more bugs than they closed. Maybe incentivized by the "agile" methods notions of "velocity". And they were praised for their poor results. Whereas my tortoise strategies nominally took longer. So I had fewer, larger commits. Way fewer "points" on the kanban board. Created much fewer lines of code. (When fixing [rewriting] other people's code, mine was often 50% to 80% smaller. Mostly by removing dead code and deduplication.) I was able to bang out new stuff and beat deadlines when I was working solo. I think the difference between solo and team play is mostly due to style mismatches. It's very hard for me to collaborate with teammates who are committing smaller, more frequent, often broken, code changes. Any way. That's my current best guess at what's happening to this graybeard. More optimistically... I'm very interested in the "Test Into Prod" strategies advocated by the CTO from Gilt (?). It's the first QA/Test strategy (for an "agile" world") that makes any kind of sense to me. So I think I could adapt to that work style. (I served as SQA Manager for a while. It's hard to let go of those expectations. It's been maybe 20 years since I've seen anyone doing actual QA/Test. I feel bad for today's business analysts (BAs) who get stuck doing requirements and monkey style button pushing. Like how most orgs functioned in the 80s.)
- throwanem 7y ago
  I see more gray in my beard every morning. And the thing about "...to 80% completion" is that the first 80% of value is captured in the first 80% of effort, and the last 20% of value in the other 80% of effort. It's important to know when to follow that ROI graph past the knee, for sure. But it's just as important to know when not to. (I mind me of a time a few years back when I was surprised to learn that the right method of exception handling, for a specific case in some work I was doing on a distributed system, was none - just letting it crash and try again when the orchestrator stood up a fresh container. It felt wrong at first; ever before I'd have instead gone to a lot of painstaking effort to reset the relevant state by hand, and my first instinct was to do the same here. But crashing turned out to be the right thing to do, because it worked just as well and took no time at all to implement.)

sillysaurusx 7y ago

Not at all. I share your frustration. Closures on a server are a powerful way of representing data flows. But they come at a cost: the links expire after some time. How do you strike a balance? The simplest way is to put in the extra time to make everything into a persistent link. But, that's equivalent to removing all the benefits of lexical scoping. If you've ever created an inner function before, you know how powerful that technique can be. You can encode the entire state of the program into closures -- no need to store things in databases. Want a password reset link? Spin up a closure, email the link, done. Literally identical to storing a reset token in a database, except there's no database. Another solution is to fix the root problem. Does the closure really need a random ID every time the page refreshes? The closure links die because they have to be GC'd periodically, to keep memory usage down. Even if you cache the page for 90 seconds, that's still 960 refreshes per day for logged-out users. Then if you have a few hundred regular users, that's at least another factor of two. And certain pages might create hundreds of closure links each refresh, so it quickly gets out of hand. Ironically, the solution was emacs -- in emacs, they store closures in a printable way. A closure is literally a list of variable values plus the function's source code. That got me thinking -- why not use that as the key, instead of making a random key each time the page refreshes? After all, if the function's lexical variables are identical, then it should produce identical results each time it's run. No need to create another one. That's what I did. It took a week or so, which is a week I'll never get back for building new features. But at least users won't have to deal with dead links anymore. Clever readers will note a theoretical security flaw: an attacker might be able to guess your function IDs if they knew the entire state of the closure + the closure's source code (which is the default case for an open source project). That might give them access to e.g. your admin links. But that's not an indictment of the technique; it's easily solved by concatenating the key with a random ID generated at startup, and hashing that. I'm just making a note of it here in case some reader wants to try implementing this idea in their own framework. The closure technique has nontrivial productivity speedups (that I think someone will rediscover some years from now). I hope the idea becomes more popular over time.

kazinator 7y ago
> How do you strike a balance? How about a keepalive from the client side: little bit of JavaScript that somehow tells the server that the session is still alive, so don't blow away the closure or continuation.
sillysaurusx 7y ago
Since this is getting a surprising amount of interest, let me sum up the technique here. It's really not hard to implement it in Javascript using Express. 1. inside of your express endpoint, create a closure that captures some state. For example, the user's IP address. (This is a dumb example, but the point is that you can capture whatever state you want. Some cases are mentioned here: http://paulgraham.com/road.html http://paulgraham.com/road.html and here: http://ep.yimg.com/ty/cdn/paulgraham/bbnexcerpts.txt http://ep.yimg.com/ty/cdn/paulgraham/bbnexcerpts.txt) EDIT: I updated this to capture the date + time the original page was loaded, which is slightly more compelling than a mere IP address. app.get('/', function (req, res) { let ip = req.headers['x-forwarded-for'] || req.connection.remoteAddress; let now = new Date(); let date = now.getFullYear()+'-'+(now.getMonth()+1)+'-'+now.getDate(); let time = now.getHours() + ":" + now.getMinutes() + ":" + now.getSeconds(); let fn = (req, res) => { res.send(`hello ${req.query.name}. On ${date} at {time}, your IP address was ${ip}`) } ... see below ... }) 2. insert that closure into a global hash table, keyed by a random ID. g_fnids = {}; app.get('/', function (req, res) { let fn = ... let id = <generate random ID> g_fnids[id] = fn; res.send(`<a href="/x?fnid=${id}&name=bob">Say hello</a>`); }) 3. create an endpoint called /x which works like `/x?fnid=<function id>&foo=1&bar=2`. Use <function id> to look up the closure. Call the closure, passing the request to it: app.get('/x', function (req, res) { let id = req.query.fnid; let fn = g_fnids[id]; fn(req, res) } Done. Congratulations, your closure is now an express endpoint. Except you didn't have to name it. You can link users to it like `<a href="/x?fnid=<function id>&name=bob">Say hello</a>`. The reason this is a powerful technique is that you can use it with forms. The form target can be /x, and the query params are whatever the user types into the form fields. I bet you already see a few interesting use cases. And you might notice that this makes scaling the server a little more difficult, since incoming requests have to be routed to the server containing the actual closure. But in the meantime, you now have "inner functions" in your web framework. It makes implementing password reset functionality completely trivial, and no database required. If it seems slightly annoying to use – "I thought you said this was a productivity boost. But it's annoying to type all of that!" – lisp macros hide all of this boilerplate code, so there's zero extra typing. You can get lisp macros for Javascript using Lumen lisp: https://github.com/sctb/lumen https://github.com/sctb/lumen Even without macros, though, I bet this technique is shorter. Suppose you had to store the date + time + IP address somewhere. Where would you put it? I assume some sort of nosql database like firebase. But wouldn’t that code be much longer and more annoying to write? So this technique has tremendous value, and I’m amazed no one is using it circa 2020.

specialist 7y ago

Yup. I somehow became a graybeard. I really didn't fit in at my last three gigs doing "backend" work. I always play to win, so try to understand why & how I failed. My current theory: I had good successes doing product development. Shipping software that had to be pretty close to correct. Today's "product development" is really IT, data processing. Way more forgiving of stuff that's not quite right. Often not even close to right. (Guessing that about 1/3rd of the stuff I supported didn't actually do what the original author thought it did, and no one was the wiser, until something didn't seem quite right.) One insightful coworker said it best: "I learned to do everything to 80% completion." My observation is that most team mates created more bugs than they closed. Maybe incentivized by the "agile" methods notions of "velocity". And they were praised for their poor results. Whereas my tortoise strategies nominally took longer. So I had fewer, larger commits. Way fewer "points" on the kanban board. Created much fewer lines of code. (When fixing [rewriting] other people's code, mine was often 50% to 80% smaller. Mostly by removing dead code and deduplication.) I was able to bang out new stuff and beat deadlines when I was working solo. I think the difference between solo and team play is mostly due to style mismatches. It's very hard for me to collaborate with teammates who are committing smaller, more frequent, often broken, code changes. Any way. That's my current best guess at what's happening to this graybeard. More optimistically... I'm very interested in the "Test Into Prod" strategies advocated by the CTO from Gilt (?). It's the first QA/Test strategy (for an "agile" world") that makes any kind of sense to me. So I think I could adapt to that work style. (I served as SQA Manager for a while. It's hard to let go of those expectations. It's been maybe 20 years since I've seen anyone doing actual QA/Test. I feel bad for today's business analysts (BAs) who get stuck doing requirements and monkey style button pushing. Like how most orgs functioned in the 80s.)

throwanem 7y ago
I see more gray in my beard every morning. And the thing about "...to 80% completion" is that the first 80% of value is captured in the first 80% of effort, and the last 20% of value in the other 80% of effort. It's important to know when to follow that ROI graph past the knee, for sure. But it's just as important to know when not to. (I mind me of a time a few years back when I was surprised to learn that the right method of exception handling, for a specific case in some work I was doing on a distributed system, was none - just letting it crash and try again when the orchestrator stood up a fresh container. It felt wrong at first; ever before I'd have instead gone to a lot of painstaking effort to reset the relevant state by hand, and my first instinct was to do the same here. But crashing turned out to be the right thing to do, because it worked just as well and took no time at all to implement.)