5 ms·

You’re really on to something with this query language for structured data idea! I’d find it super-useful as a dev to be able to think of a backing service not

by riwsky 6y ago

You’re really on to something with this query language for structured data idea! I’d find it super-useful as a dev to be able to think of a backing service not as a set of disconnected endpoints, but a base of operations for dealing with data. If only such structured query language data bases existed, I bet they would get a lot of use. (No but really: how is what you’re describing meaningfully different from SQL, especially with the restrictions you’d want to put on the code in practice?)

nawgz 6y ago
I agree with your comment highly. I also find the idea of arbitrary code execution on a remote host to build a derivative service to be a quite naive point-of-view. Even if you can validate that the code isn't going to pwn you - a mighty stretch - how can one justify being a recipient of free compute cycles on someone else's infra? It's not quite realistic to me
- anthuswilliams 6y ago
  I don't think OP necessarily implied that the compute cycles had to be free.
  - nawgz 6y ago
    Right, so then he's proposing that this streaming service is enabling derivative services by not only managing the API for the streaming service but also by running their own public cloud which issues micro-charges for compute cycles. That seems even more pie-in-the-sky.
    - anthuswilliams 6y ago
      It's definitely pie-in-the-sky. I wouldn't argue otherwise. But I do think this is a problem our industry will have to solve eventually. The volume of data we have to move between servers in order to ask questions is exploding. We move petabytes of image data between filesystems in order to train machine learning models. If the problems in this thread can be solved (admittedly a big if), moving the code to the data would be fundamentally better.
    - miki123211 6y ago
      Mind you, most of this should be handled by some kind of runtime. Any complex enough web service runs HTTP Servers, databases, sometimes messaging queues, caches or even things like Kubernetes. There's no reason why it couldn't be running yet another program to manage all this. Such a thing could become the de-facto API, if Rest or GraphQL were needed, they could even be implemented on top of it.
      - nawgz 6y ago
        >There's no reason why it couldn't be running yet another program to manage all this Yes there is - that program would need to be extremely hardened (even against DDOS style attacks - run a ton of memory-consuming cycle-hogs on a service's platform), integrated with tracking & billing, and open-source (or itself charging). I just don't see the incentive for someone building this, and then I doubly don't see the incentive for a corp running this. Why take on this huge extra burden of infra to help derivative services? It's not going to make you money.
- miki123211 6y ago
  > I also find the idea of arbitrary code execution on a remote host to build a derivative service to be a quite naive point-of-view. Even if you can validate that the code isn't going to pwn you - a mighty stretch Be aware that's exactly what you are doing when you go to any modern site. You let a remote server execute arbitrary code on your computer, trusting it's not going to pwn you. Sure, there were tons of loopholes in the first few years of JS, but now browsers and operating systems are so secure that it's mostly a non-issue. Web assembly is a sandboxing-first technology, so I don't think it's an impossible or even a particularly hard issue to solve. >how can one justify being a recipient of free compute cycles on someone else's infra? It's not quite realistic to me Well, there are API call limits, or something equivalent to them. Maybe computing credits in this instance. Calculating how much computing resources you're using is something that i.e. AWS Lambda does already, no reason why an open-source runtime couldn't do so either. The fact it wouldn't be native code, executed directly by the CPU, but Wasm executed by a VM, makes things even simpler.
  - nawgz 6y ago
    >but now browsers and operating systems are so secure that it's mostly a non-issue Right, which is the point - it took an absolutely behemoth effort backed by megacorps to get us to this point. These applications are public-facing because that's the key way for the corps to make money - content delivery. Why would anyone put this same effort into building more complicated internal services enabling derivative services to extract value from their platform? > Calculating how much computing resources you're using is something that i.e. AWS Lambda does already Yes, exactly - another megacorp built a service that makes them huge amounts of money, and now they sell access to it. This code is certainly not publicly accessible So, again, help me bridge this gap - what possible reason would Spotify (to further GP's example) build such a platform for? This means that Spotify needs to maintain massively more infrastructure in order to help others extract value from their platform. It just doesn't check out. Is it objectively a more efficient method of computation? Yes. Is it also completely misaligned with corporate incentives in the modern day? In my opinion, yes.
miki123211 6y ago
You're onto something with SQL here. It definitely should be a big part of this, especially for the querying data aspect. Of course, we wouldn't query over raw, database data, but over sanitized, secure views. SQL isn't everything, though. Be aware that such APIs, for security reasons, wouldn't allow you to modify some of the data in arbitrary ways, but require you to use special functions instead. Think of being able to query the number of upvotes, call an upvote function, without being able to just change the upvote count. Maybe a mix of wasm (as an imperative platform) combined with a structured query API? Maybe a programming language with something like Elixir's Ecto or C#'s LINQ? Not plain SQL, but an imperative language with the power of relational queries?
- riwsky 6y ago
  Your characterization of sanitized secure views as something other than data in a database strikes me as vitalistic (and dubious). The “communicate with code, not messages” in your first comment strikes me the same way: code is data, and data is code, and Wasm does nothing to change that. Nothing about making an interface more expressive guarantees one is making it better, and overspecification of user intent can make it much harder for the system to optimize execution. Even if we really do want imperativity: your upvote example, even including the security aspects, is easily handled using stored procedure support that’s been available in SQL dbs for decades. It is not unfair to describe that support, and its variants (eg postgres lets you use python) as “not plain SQL, but an imperative language with the power of relational queries”. Can this approach handle everything? No; it would be madness to try and build e.g. huge neural network models in SQL + some stored procedures. But it handles a lot, including your two examples, and handles them more compactly than a general purpose language would, and exists. Mind you, I’m not saying your general concept isn’t great. It is great, it is a great step. It’s just that distributed programming already took it, back in the 70s.

by riwsky 6y ago

nawgz 6y ago

I agree with your comment highly. I also find the idea of arbitrary code execution on a remote host to build a derivative service to be a quite naive point-of-view. Even if you can validate that the code isn't going to pwn you - a mighty stretch - how can one justify being a recipient of free compute cycles on someone else's infra? It's not quite realistic to me

anthuswilliams 6y ago
I don't think OP necessarily implied that the compute cycles had to be free.
- nawgz 6y ago
  Right, so then he's proposing that this streaming service is enabling derivative services by not only managing the API for the streaming service but also by running their own public cloud which issues micro-charges for compute cycles. That seems even more pie-in-the-sky.
  - anthuswilliams 6y ago
    It's definitely pie-in-the-sky. I wouldn't argue otherwise. But I do think this is a problem our industry will have to solve eventually. The volume of data we have to move between servers in order to ask questions is exploding. We move petabytes of image data between filesystems in order to train machine learning models. If the problems in this thread can be solved (admittedly a big if), moving the code to the data would be fundamentally better.
  - miki123211 6y ago
    Mind you, most of this should be handled by some kind of runtime. Any complex enough web service runs HTTP Servers, databases, sometimes messaging queues, caches or even things like Kubernetes. There's no reason why it couldn't be running yet another program to manage all this. Such a thing could become the de-facto API, if Rest or GraphQL were needed, they could even be implemented on top of it.
    - nawgz 6y ago
      >There's no reason why it couldn't be running yet another program to manage all this Yes there is - that program would need to be extremely hardened (even against DDOS style attacks - run a ton of memory-consuming cycle-hogs on a service's platform), integrated with tracking & billing, and open-source (or itself charging). I just don't see the incentive for someone building this, and then I doubly don't see the incentive for a corp running this. Why take on this huge extra burden of infra to help derivative services? It's not going to make you money.
miki123211 6y ago
> I also find the idea of arbitrary code execution on a remote host to build a derivative service to be a quite naive point-of-view. Even if you can validate that the code isn't going to pwn you - a mighty stretch Be aware that's exactly what you are doing when you go to any modern site. You let a remote server execute arbitrary code on your computer, trusting it's not going to pwn you. Sure, there were tons of loopholes in the first few years of JS, but now browsers and operating systems are so secure that it's mostly a non-issue. Web assembly is a sandboxing-first technology, so I don't think it's an impossible or even a particularly hard issue to solve. >how can one justify being a recipient of free compute cycles on someone else's infra? It's not quite realistic to me Well, there are API call limits, or something equivalent to them. Maybe computing credits in this instance. Calculating how much computing resources you're using is something that i.e. AWS Lambda does already, no reason why an open-source runtime couldn't do so either. The fact it wouldn't be native code, executed directly by the CPU, but Wasm executed by a VM, makes things even simpler.
- nawgz 6y ago
  >but now browsers and operating systems are so secure that it's mostly a non-issue Right, which is the point - it took an absolutely behemoth effort backed by megacorps to get us to this point. These applications are public-facing because that's the key way for the corps to make money - content delivery. Why would anyone put this same effort into building more complicated internal services enabling derivative services to extract value from their platform? > Calculating how much computing resources you're using is something that i.e. AWS Lambda does already Yes, exactly - another megacorp built a service that makes them huge amounts of money, and now they sell access to it. This code is certainly not publicly accessible So, again, help me bridge this gap - what possible reason would Spotify (to further GP's example) build such a platform for? This means that Spotify needs to maintain massively more infrastructure in order to help others extract value from their platform. It just doesn't check out. Is it objectively a more efficient method of computation? Yes. Is it also completely misaligned with corporate incentives in the modern day? In my opinion, yes.

miki123211 6y ago

You're onto something with SQL here. It definitely should be a big part of this, especially for the querying data aspect. Of course, we wouldn't query over raw, database data, but over sanitized, secure views. SQL isn't everything, though. Be aware that such APIs, for security reasons, wouldn't allow you to modify some of the data in arbitrary ways, but require you to use special functions instead. Think of being able to query the number of upvotes, call an upvote function, without being able to just change the upvote count. Maybe a mix of wasm (as an imperative platform) combined with a structured query API? Maybe a programming language with something like Elixir's Ecto or C#'s LINQ? Not plain SQL, but an imperative language with the power of relational queries?

riwsky 6y ago
Your characterization of sanitized secure views as something other than data in a database strikes me as vitalistic (and dubious). The “communicate with code, not messages” in your first comment strikes me the same way: code is data, and data is code, and Wasm does nothing to change that. Nothing about making an interface more expressive guarantees one is making it better, and overspecification of user intent can make it much harder for the system to optimize execution. Even if we really do want imperativity: your upvote example, even including the security aspects, is easily handled using stored procedure support that’s been available in SQL dbs for decades. It is not unfair to describe that support, and its variants (eg postgres lets you use python) as “not plain SQL, but an imperative language with the power of relational queries”. Can this approach handle everything? No; it would be madness to try and build e.g. huge neural network models in SQL + some stored procedures. But it handles a lot, including your two examples, and handles them more compactly than a general purpose language would, and exists. Mind you, I’m not saying your general concept isn’t great. It is great, it is a great step. It’s just that distributed programming already took it, back in the 70s.