5 ms·

You might have a workable compromise if you cheat and only serialize the portion of the closure's environment that refers to some simplified set of types. A wa

by pc 18y ago

You might have a workable compromise if you cheat and only serialize the portion of the closure's environment that refers to some simplified set of types. A way of serializing closures such that they kept bindings to symbols, numbers, strings and conses in the local environment would probably be pretty useful. (On the web, I think you usually want to preserve fairly simple values - usernames or whatever.) map-closure would make it easy to do at the language level, too.

pc 18y ago
After looking through the News.YC source, it seems that almost every single closure that's created on the fly refers to one of the above types. (I think you could exclude conses, too, to make it even simpler.)
pg 18y ago
I suppose it wouldn't be too gross to have an operator where you explicitly specified a list of variables you were going to refer to. (IIRC one had to do that in Franz Lisp.) It would only add one expr to the code I'd had before. Maybe I'll try it.
- pc 18y ago
  Not sure if you're intentionally proposing something different, or if I wasn't clear, but I didn't mean you should have to list the vars you intended to later refer to -- I think the serialize operator should automatically serialize all values in the local environment that are of the allowed set of types.
  - pg 18y ago
    I was thinking of something that looked like let. Since you wouldn't be able to do it for all types, it seems like it would be better to make the programmer see the vars he was going to close over. Otherwise you'd run into the kinds of problems Lisp hackers ran into in the days of dynamic scope, where your code works (till it doesn't) by accident.
    - almkglor 18y ago
      Well, currently a few people in the Arc community are building implementations of Arc. It may actually be possible to include as an axiom a few functions that allow a closured function to be destructured. For example, it may be possible to define a set of functions which can be applied on a function to determine if it encloses any variables or not, what number of enclosed variables, and the variable values (if not the names). It would also be possible to add as an axiom a function to extract the "code part" of a closure, at least as an opaque (but uniquely-comparable) object. Speaking as an implementeer hacking on the low-level side, generally the implementation of a closured function is simply two pointers: a pointer to the enclosed environment, and a pointer to the code (of course, in arc2c it's just an array where the first entry is a reference to the code and the succeeding array entries are the enclosed variables). Only the pointer to the enclosed environment needs to actually be considered: the code is constant (except for code updates, but one would expect updates to be rarer than actual invocations of the code). If we can extract the data of each closured variable, we can determine if it's trivially serializable (e.g. strings, numbers, proper lists of strings and numbers), and if so, it is potentially possible to encrypt the values into the URL. We can then also add some functions to reconstruct a closure, given only an opaque code reference and a bunch of data. Since my Arc implementation, SNAP, will require the ability to serialize and deserialize closured functions (including code and data) anyway, I'll have to extend Arc that way. The advantage of this is that you don't need to explicitly state what you want to close over: you just close over variables, and the base system will destructure your closure, and just encode the trivially serializable variables into the generated URL link.
      - jules 18y ago
        Note that although serializing simple values (like numbers and symbols) to urls works, serializing big data doesn't work because urls have a length limit. What I was trying to explain is a system that stores the data on the server and passes pointers to this data to the client. Web applications that use databases already work this way. The id/primary key column of a database table is the pointer. To serialize a closure you generate ids for the values of the free variables. Then you save the values in a hash table and send the ids to the client (instead of the actual values). To deserialize a closure you fetch the free variables from the hash table. This system doesn't work yet because this hash table keeps growing. Every time you serialize a closure you put the free variables in the hash table. This is wasteful because some of the values might already be in the hash table. This is simple to fix; just look if the value is in the hash table, and if it is you return the existing id (this can be done efficiently by having two hash tables: ids->values and values->ids).
- jules 18y ago
  It wouldn't be too hard to figure that out automatically. Compilers have to do that anyway. But it's a good starting point.

by pc 18y ago

pc 18y ago

After looking through the News.YC source, it seems that almost every single closure that's created on the fly refers to one of the above types. (I think you could exclude conses, too, to make it even simpler.)

pg 18y ago

I suppose it wouldn't be too gross to have an operator where you explicitly specified a list of variables you were going to refer to. (IIRC one had to do that in Franz Lisp.) It would only add one expr to the code I'd had before. Maybe I'll try it.

pc 18y ago
Not sure if you're intentionally proposing something different, or if I wasn't clear, but I didn't mean you should have to list the vars you intended to later refer to -- I think the serialize operator should automatically serialize all values in the local environment that are of the allowed set of types.
- pg 18y ago
  I was thinking of something that looked like let. Since you wouldn't be able to do it for all types, it seems like it would be better to make the programmer see the vars he was going to close over. Otherwise you'd run into the kinds of problems Lisp hackers ran into in the days of dynamic scope, where your code works (till it doesn't) by accident.
  - almkglor 18y ago
    Well, currently a few people in the Arc community are building implementations of Arc. It may actually be possible to include as an axiom a few functions that allow a closured function to be destructured. For example, it may be possible to define a set of functions which can be applied on a function to determine if it encloses any variables or not, what number of enclosed variables, and the variable values (if not the names). It would also be possible to add as an axiom a function to extract the "code part" of a closure, at least as an opaque (but uniquely-comparable) object. Speaking as an implementeer hacking on the low-level side, generally the implementation of a closured function is simply two pointers: a pointer to the enclosed environment, and a pointer to the code (of course, in arc2c it's just an array where the first entry is a reference to the code and the succeeding array entries are the enclosed variables). Only the pointer to the enclosed environment needs to actually be considered: the code is constant (except for code updates, but one would expect updates to be rarer than actual invocations of the code). If we can extract the data of each closured variable, we can determine if it's trivially serializable (e.g. strings, numbers, proper lists of strings and numbers), and if so, it is potentially possible to encrypt the values into the URL. We can then also add some functions to reconstruct a closure, given only an opaque code reference and a bunch of data. Since my Arc implementation, SNAP, will require the ability to serialize and deserialize closured functions (including code and data) anyway, I'll have to extend Arc that way. The advantage of this is that you don't need to explicitly state what you want to close over: you just close over variables, and the base system will destructure your closure, and just encode the trivially serializable variables into the generated URL link.
    - jules 18y ago
      Note that although serializing simple values (like numbers and symbols) to urls works, serializing big data doesn't work because urls have a length limit. What I was trying to explain is a system that stores the data on the server and passes pointers to this data to the client. Web applications that use databases already work this way. The id/primary key column of a database table is the pointer. To serialize a closure you generate ids for the values of the free variables. Then you save the values in a hash table and send the ids to the client (instead of the actual values). To deserialize a closure you fetch the free variables from the hash table. This system doesn't work yet because this hash table keeps growing. Every time you serialize a closure you put the free variables in the hash table. This is wasteful because some of the values might already be in the hash table. This is simple to fix; just look if the value is in the hash table, and if it is you return the existing id (this can be done efficiently by having two hash tables: ids->values and values->ids).
jules 18y ago
It wouldn't be too hard to figure that out automatically. Compilers have to do that anyway. But it's a good starting point.