5 ms·

Not to be a downer, but while I think in theory this sounds interesting, and treating frontend and backend differences as a DAG that is compiled for the differe

by yladiz 4y ago

Not to be a downer, but while I think in theory this sounds interesting, and treating frontend and backend differences as a DAG that is compiled for the different environments seems cool, I'm skeptical that it would actually be productive in a large application. One thing I'd be particularly concerned about is debugging. One of the things that's on the roadmap is "developer experience", and I don't see any particular debug docs or helpers, so it tells me that if there is a bug somewhere debugging it is going to be a pain, even more so because I won't easily be able to see whether the problem is my client or server code as they live together and are compiled into separate things. As a critique the fact that developer experience is on the roadmap rather than thought about from the beginning is a little off-putting, but maybe that's not what was meant here. Another thing that gives me hesitation is that I can't see how I would extend the setup to support offline capable applications. This is increasingly becoming a big thing for web+desktop apps, does this easily support that? Lastly, the docs seem to point to use a homegrown DOM library, rather than one of the major players. How easily would this work with a library like Reagent, Helix, or UIx?

scary-size 4y ago
Agreeing on the debugging concerns. Adding another layer of abstraction on top of everything else isn't going to help anyone. Creating an abstraction that fits so well that you don't have to worry about the layer underneath is extremely hard. That might not show during initial development, but will come up in the maintenance phase for sure. A bit unsure where the observation of more offline capabilities comes from. I have the impression that things are moving in the opposite direction: Everything needs to be online.
- ldh0011 4y ago
  I think they mean apps that don't just stop working when you're offline, not apps that are always offline.
  - yladiz 4y ago
    Yes, that's what I meant. I didn't mean an app that's never online (or connected to a server), since that doesn't really make sense in the context of this kind of framework, I meant something that is resilient to being offline (aka "offline capable").
    - yenda 4y ago
      based on the todomvc example, I can say that it is resilient to being offline, in the sense that whatever you do while temporarily offline is resumed when you are back online. there's no optimistic updates so it looks like the items you add while offline are disappearing, but they end up in the list as soon as you are back online.
- ElectricalUnion 4y ago
  Everything needs to be online, but "online" those days probably means cutting off constantly between 5GHz and 2.4GHz wifi, with bad ISP DNS and IPv6 routing in the way.
- michaelteter 4y ago
  > Adding another layer of abstraction on top of everything else isn't going to help anyone Ignoring the point that the frameworks we already use are major abstractions on top of abstractions, if this new "abstraction" truly eliminates the complexities of client-server-client it will greatly benefit developers and users alike. A good many (most?) of the bugs and performance problems (and security lapses) in today's complex webapps is due to the complexities of managing this client-server divide. Even when the developer does things right according to docs and examples, it's possible to end up with a system which does not perform well without solid understanding of what's happening in at the endpoints and in the middle and clever optimizations to solve the special needs.
  - scary-size 4y ago
    You're right, I was a bit harsh by disregarding the current use of frameworks. There're huge upsides to them. They still hide stuff, some do it well, some not so. Electric gives me the (unproven) vibe of hiding the lower layers a bit too well. Regarding the causes of many bugs, your guess is as good as mine. You'll continue to have bugs and you'll have to continue looking under the covers to understand them.
    - deleted 4y ago
      [deleted]
- dustingetz 4y ago
  You're right, over-abstracting is a primary risk and has been top of mind for us since project conception in ~2012. ORM, for example, is a nightmare of over-abstraction (remember Spring Framework's AbstractSingletonProxyFactoryBean?). In fact, studying the O/R impedance mismatch from first principles back in ~2010 was what led me to functional programming in 2011. Electric is an attempt to find exactly the right level of abstraction. The goal is to remove and flatten layers, not add them, thus decreasing abstraction weight in the end if we succeed. Maybe we fail, but first let me share some details about how we think about this: 1. I've personally failed to build this project several times, Electric Clojure is something like the 7th attempt. 2. strong composition model as a starting point, based on category theory generalization of "function" -> "async function" -> "reactive function" -> "stream function" -> "distributed function". ORM does not have this. (This rigor is in response to the past failures.) 3. Functional effect system (monad stuff) at the bottom, which provides strong semantics guarantees about glitch-free reactive propagation, process supervision (like Erlang) (transparent propagation of cancellation and failure), strong resource cleanup guarantees (DOM nodes can never be left hanging, event handlers can never fail to be detached and disposed). Already this results in tighter operational semantics than we have ever achieved with manual resource management (and, again, we tried, see past failures). Our functional effects library: https://github.com/leonoel/missionary https://github.com/leonoel/missionary 4. Electric affords the programmer trapdoors to the underlying FRP/concurrency primitives. Electric is essentially a Clojure-to-FRP compiler, so if you code raw concurrency and effect management, that actually typechecks with what Electric generates, allowing seamless transition in and out of the abstraction. 5. 3k LOC + 3k test LOC is the size of Electric today (includes a rewrite of the Clojure analyzer). Spring Framework is, let me go check, 59k just for `spring-core/src/main/java`, and there are like 20 other modules I excluded. Indeed it is not a fair comparison but certainly we have complexity budget to spare.
den1k 4y ago
I've been building with this for 2+ months. Debugging is fine (not great but when is it great? Especially in Clojure). It prints stacktraces in each env. Haven't found debugging harder than traditional client/server dev. But complexity has decreased thanks to it so less bugs.
yenda 4y ago
Regarding offline I tested the photon-starter-app repo a bit and it looks like it already supports disconnections but without optimistic updates: Here's what I did - killed the network - added a bunch of items to the todo list, which disappeared from the input field but didn't appear in the list - switched network back on, items appeared in the list I'm curious to see how adding optimistic updates to the mix would work
- dustingetz 4y ago
  (founder here) Should be straightforward; as you discovered, everything is properly backpressured by the compiler (automatically!) so no state is lost.
dustingetz 4y ago
(Founder here; I'll address the debugging concern in a separate comment) Here's a demo of React.js interop via Reagent, the wrapper is 25 LOC: https://gist.github.com/dustingetz/9854d23037b55bfab3845539f3e66e02 https://gist.github.com/dustingetz/9854d23037b55bfab3845539f... Our DOM module is only 300 LOC - it's bare metal DOM point writes + Electric (reactive language) + macros for JSX-y syntax. When the programming language itself is reactive, DOM rendering falls out for free. Mechanically, Electric is comparable to Solid.js except the reactive engine is general purpose, not coupled to DOM rendering, which is a special case of incremental view maintenance. Electric's reactive engine is https://github.com/leonoel/missionary https://github.com/leonoel/missionary ; Electric can be seen as a Clojure to Missionary compiler (if you ignore the distribution stuff - which btw is optional). Missionary is cross-platform functional effect and streaming system that today is tested on JVM, Node and Browser platforms. Missionary was created by Léo Noel, who is also the architect of Electric.
dustingetz 4y ago
Re. abstraction weight - You're right, it's true that the program is less inspectable than before, and that it adds more distance between programmer and metal. We're aware of the issue and have already begun – we landed async stack traces a while ago (which works about the same as in React.js). And yet. It's also true that HLL (high level languages) like C, Java, Python all faced the same issue. The benefit is a huge gain in terms of ability to reason about ever larger programs. Java gave us garbage collection. Not everyone liked that; C++ developers want to manage their own memory. But at what cost? Our thinking is: first get the semantics right. Then polish the experience (today we are here). Then add observability tools. Then make it fast. And if we've screwed up the semantics, then the project will fail for the reasons you describe. The saving grace is we're seeing 10x LOC reduction (18k to 2k) in rebuilding Electric's sister project, Hyperfiddle (a spreadsheet like tool for robust UI development), as well as massive gains in performance. Re. offline-first - we think the model can be extended to that but this is a blog post for another day
- dustingetz 4y ago
  One quick sketch of offline-first (there are many other ways): imagine a database like DataScript (already OSS and built in Clojure), move key components like the query engine into Electric Clojure, now the "DAG slice" can cut right through the query engine and it's internal data structures, such that the indexes are partially on the client, and the final stages of query filtering are on the client. A network-transparent database! The bulk of the work, and sensitive data, stays on the server; but the last mile user data querying happens on the client. Electric can transparently move portions of the underlying database indexes to the client for local offline query. 0 LOC cost to userland! No loss of relational query when offline! No performance cost of putting the entire query engine in the browser (a low compute environment)!
mst 4y ago
> As a critique the fact that developer experience is on the roadmap rather than thought about from the beginning is a little off-putting Implementing it is on the roadmap but that's no evidence that it hasn't been thought about extensively and for something like this you really -have- to think things out up front (it's more like a math proof than a CRUD app in a lot of ways) or you're going to find yourself with holes in both feet and a codebase you have to throw away and restart from scratch. From what I can see, they've been laser focused on getting the core to be correct by design, which will have required repeatedly gutting parts of it as conceptual mistakes are shaken out, so if you -wrote- the DX code as you were going rather than simply keeping in mind that the codebase needs to be amenable to it -being- written (and periodically thinking through how you'd do so as a sort of mental dry run) I'd expect the DX parts to have to be thrown away entirely, repeatedly, which is a motivation sink and in the best case scenario would've substantially slowed down getting the core right. But yeah, the key thing is "not yet implemented" and "hasn't been thought about at all" are very different things and I'd strongly suspect that my read of what was meant is more likely to be the right one given the nature of the project. (please take the above as "an explanation of why I suspect their actions are fine and their chosen trade-offs make sense" rather than "let me tell you why you're wrong" because you do have a point that if you don't keep DX in mind at all in the early stages the end result tend to be unhappy-making, and, well, let's just say I speak from repeated experience of doing that that to myself ;)

by yladiz 4y ago

scary-size 4y ago

Agreeing on the debugging concerns. Adding another layer of abstraction on top of everything else isn't going to help anyone. Creating an abstraction that fits so well that you don't have to worry about the layer underneath is extremely hard. That might not show during initial development, but will come up in the maintenance phase for sure. A bit unsure where the observation of more offline capabilities comes from. I have the impression that things are moving in the opposite direction: Everything needs to be online.

ldh0011 4y ago
I think they mean apps that don't just stop working when you're offline, not apps that are always offline.
- yladiz 4y ago
  Yes, that's what I meant. I didn't mean an app that's never online (or connected to a server), since that doesn't really make sense in the context of this kind of framework, I meant something that is resilient to being offline (aka "offline capable").
  - yenda 4y ago
    based on the todomvc example, I can say that it is resilient to being offline, in the sense that whatever you do while temporarily offline is resumed when you are back online. there's no optimistic updates so it looks like the items you add while offline are disappearing, but they end up in the list as soon as you are back online.
ElectricalUnion 4y ago
Everything needs to be online, but "online" those days probably means cutting off constantly between 5GHz and 2.4GHz wifi, with bad ISP DNS and IPv6 routing in the way.
michaelteter 4y ago
> Adding another layer of abstraction on top of everything else isn't going to help anyone Ignoring the point that the frameworks we already use are major abstractions on top of abstractions, if this new "abstraction" truly eliminates the complexities of client-server-client it will greatly benefit developers and users alike. A good many (most?) of the bugs and performance problems (and security lapses) in today's complex webapps is due to the complexities of managing this client-server divide. Even when the developer does things right according to docs and examples, it's possible to end up with a system which does not perform well without solid understanding of what's happening in at the endpoints and in the middle and clever optimizations to solve the special needs.
- scary-size 4y ago
  You're right, I was a bit harsh by disregarding the current use of frameworks. There're huge upsides to them. They still hide stuff, some do it well, some not so. Electric gives me the (unproven) vibe of hiding the lower layers a bit too well. Regarding the causes of many bugs, your guess is as good as mine. You'll continue to have bugs and you'll have to continue looking under the covers to understand them.
  - deleted 4y ago
    [deleted]
dustingetz 4y ago
You're right, over-abstracting is a primary risk and has been top of mind for us since project conception in ~2012. ORM, for example, is a nightmare of over-abstraction (remember Spring Framework's AbstractSingletonProxyFactoryBean?). In fact, studying the O/R impedance mismatch from first principles back in ~2010 was what led me to functional programming in 2011. Electric is an attempt to find exactly the right level of abstraction. The goal is to remove and flatten layers, not add them, thus decreasing abstraction weight in the end if we succeed. Maybe we fail, but first let me share some details about how we think about this: 1. I've personally failed to build this project several times, Electric Clojure is something like the 7th attempt. 2. strong composition model as a starting point, based on category theory generalization of "function" -> "async function" -> "reactive function" -> "stream function" -> "distributed function". ORM does not have this. (This rigor is in response to the past failures.) 3. Functional effect system (monad stuff) at the bottom, which provides strong semantics guarantees about glitch-free reactive propagation, process supervision (like Erlang) (transparent propagation of cancellation and failure), strong resource cleanup guarantees (DOM nodes can never be left hanging, event handlers can never fail to be detached and disposed). Already this results in tighter operational semantics than we have ever achieved with manual resource management (and, again, we tried, see past failures). Our functional effects library: https://github.com/leonoel/missionary https://github.com/leonoel/missionary 4. Electric affords the programmer trapdoors to the underlying FRP/concurrency primitives. Electric is essentially a Clojure-to-FRP compiler, so if you code raw concurrency and effect management, that actually typechecks with what Electric generates, allowing seamless transition in and out of the abstraction. 5. 3k LOC + 3k test LOC is the size of Electric today (includes a rewrite of the Clojure analyzer). Spring Framework is, let me go check, 59k just for `spring-core/src/main/java`, and there are like 20 other modules I excluded. Indeed it is not a fair comparison but certainly we have complexity budget to spare.

den1k 4y ago

I've been building with this for 2+ months. Debugging is fine (not great but when is it great? Especially in Clojure). It prints stacktraces in each env. Haven't found debugging harder than traditional client/server dev. But complexity has decreased thanks to it so less bugs.

yenda 4y ago

Regarding offline I tested the photon-starter-app repo a bit and it looks like it already supports disconnections but without optimistic updates: Here's what I did - killed the network - added a bunch of items to the todo list, which disappeared from the input field but didn't appear in the list - switched network back on, items appeared in the list I'm curious to see how adding optimistic updates to the mix would work

dustingetz 4y ago
(founder here) Should be straightforward; as you discovered, everything is properly backpressured by the compiler (automatically!) so no state is lost.

dustingetz 4y ago

(Founder here; I'll address the debugging concern in a separate comment) Here's a demo of React.js interop via Reagent, the wrapper is 25 LOC: https://gist.github.com/dustingetz/9854d23037b55bfab3845539f3e66e02 https://gist.github.com/dustingetz/9854d23037b55bfab3845539f... Our DOM module is only 300 LOC - it's bare metal DOM point writes + Electric (reactive language) + macros for JSX-y syntax. When the programming language itself is reactive, DOM rendering falls out for free. Mechanically, Electric is comparable to Solid.js except the reactive engine is general purpose, not coupled to DOM rendering, which is a special case of incremental view maintenance. Electric's reactive engine is https://github.com/leonoel/missionary https://github.com/leonoel/missionary ; Electric can be seen as a Clojure to Missionary compiler (if you ignore the distribution stuff - which btw is optional). Missionary is cross-platform functional effect and streaming system that today is tested on JVM, Node and Browser platforms. Missionary was created by Léo Noel, who is also the architect of Electric.

dustingetz 4y ago

Re. abstraction weight - You're right, it's true that the program is less inspectable than before, and that it adds more distance between programmer and metal. We're aware of the issue and have already begun – we landed async stack traces a while ago (which works about the same as in React.js). And yet. It's also true that HLL (high level languages) like C, Java, Python all faced the same issue. The benefit is a huge gain in terms of ability to reason about ever larger programs. Java gave us garbage collection. Not everyone liked that; C++ developers want to manage their own memory. But at what cost? Our thinking is: first get the semantics right. Then polish the experience (today we are here). Then add observability tools. Then make it fast. And if we've screwed up the semantics, then the project will fail for the reasons you describe. The saving grace is we're seeing 10x LOC reduction (18k to 2k) in rebuilding Electric's sister project, Hyperfiddle (a spreadsheet like tool for robust UI development), as well as massive gains in performance. Re. offline-first - we think the model can be extended to that but this is a blog post for another day

dustingetz 4y ago
One quick sketch of offline-first (there are many other ways): imagine a database like DataScript (already OSS and built in Clojure), move key components like the query engine into Electric Clojure, now the "DAG slice" can cut right through the query engine and it's internal data structures, such that the indexes are partially on the client, and the final stages of query filtering are on the client. A network-transparent database! The bulk of the work, and sensitive data, stays on the server; but the last mile user data querying happens on the client. Electric can transparently move portions of the underlying database indexes to the client for local offline query. 0 LOC cost to userland! No loss of relational query when offline! No performance cost of putting the entire query engine in the browser (a low compute environment)!

mst 4y ago

> As a critique the fact that developer experience is on the roadmap rather than thought about from the beginning is a little off-putting Implementing it is on the roadmap but that's no evidence that it hasn't been thought about extensively and for something like this you really -have- to think things out up front (it's more like a math proof than a CRUD app in a lot of ways) or you're going to find yourself with holes in both feet and a codebase you have to throw away and restart from scratch. From what I can see, they've been laser focused on getting the core to be correct by design, which will have required repeatedly gutting parts of it as conceptual mistakes are shaken out, so if you -wrote- the DX code as you were going rather than simply keeping in mind that the codebase needs to be amenable to it -being- written (and periodically thinking through how you'd do so as a sort of mental dry run) I'd expect the DX parts to have to be thrown away entirely, repeatedly, which is a motivation sink and in the best case scenario would've substantially slowed down getting the core right. But yeah, the key thing is "not yet implemented" and "hasn't been thought about at all" are very different things and I'd strongly suspect that my read of what was meant is more likely to be the right one given the nature of the project. (please take the above as "an explanation of why I suspect their actions are fine and their chosen trade-offs make sense" rather than "let me tell you why you're wrong" because you do have a point that if you don't keep DX in mind at all in the early stages the end result tend to be unhappy-making, and, well, let's just say I speak from repeated experience of doing that that to myself ;)