10 ms·
Aquameta: Web development platform built in PostgreSQL
- mdemare 7y agoSQL on Rails, 2006 https://m.youtube.com/watch?v=Es9-l1up3r8 https://m.youtube.com/watch?v=Es9-l1up3r8
- MrRadar 7y agoUnfortunately that upload is only in 240p so most of the jokes are hard to read. I have a copy of the original version which I've uploaded here: https://2by2.info/sql_on_rails_screencast2_lq.mov https://2by2.info/sql_on_rails_screencast2_lq.mov (VLC can play it).
- degenerate 7y agoI uploaded your file to Streamable: https://streamable.com/o5g3f https://streamable.com/o5g3f
- gwbas1c 7y agoCool concept! Has anyone built anything in Aquameta? Would like to see some success stories.
- erichanson 7y agoNo. :) It is still in early stages. I use it for my projects but I end up dropping to the PostgreSQL prompt occasionally and doing things nobody but me would know how to do. Every time I do though I try to fix the source of the problem. Really need to start putting up some example projects though. I'd say it's ready for very early adopters.
- micburks 7y agoI worked on Aquameta for a few years and I'll say that having a build running locally is a joy I've never encountered elsewhere in my career. Every time I need a little app or something, instead of reaching for some SAAS product, I would just build a quick prototype for myself. I could make a prototype in a couple hours and over the course of a week or two I would polish it up when I had a few extra minutes. Rather than using something built for the masses, I could tweak the interface to my own taste and I owned all the data.
- Terretta 7y agoAquameta has been the life project of Eric Hanson for close to 20 years off-and-on. Functional prototypes have been developed in XML, RDF and MySQL, but PostgreSQL is the first database discovered that has the functionality necessary to achieve something close to practical, and huge advances in web technology like WebRTC, ES6 modules, and more have shown some light at the end of the tunnel. Aquameta is an experimental project, still in early stages of development. It is not suitable for production development and should not be used in an untrusted or mission-critical environment. Not really the basis for ’reference success stories’ approach to evaluation. More, check out the repo and hack at Postgres, see where the hypothesis can be made pragmatic.
- zubairq 7y agoIs this comparable to Oracle Apex, which is a web development tool written entirely in oracle PL/SQL?
- Farbklex 7y agoI had to think about Oracle Apex as well. Back when I started developing software, it blew my mind, that I don't have any code files I can see anywhere. I still don't know why anybody would use Oracle Apex for slightly bigger projects. It just seems so abstract.
- dahdum 7y agoUsed Apex when it first came out (as HTMLDB). I’ve never found anything that compared to its speed in rolling out web based forms for internal use. At the time, it was the best BI tool for ad hoc queries around. Nowadays I use metabase, but I’ve often missed Apex speed and wished PG had it.
- smt88 7y agoI guess this isn't that dissimilar from Git, except that when you snapshot your environment, you get everything (code, IDE settings, issue tracking, and persistent data) all in the same dump. There's definitely something to this. I'm not sure that building the IDE on top of it is practical (partly because web IDEs are still pretty limited and partly because the scope is just enormous). But if we had a way to version code that combines source, data, and issues, I'm super interested. It just needs to pipe into existing tools rather than recreating existing tools.
- 52-6F-62 7y agoWell, just glancing at that source code: I do not know Postgres like I thought.
- t0mbstone 7y agoWell, to be fair, you can do all sorts of things with Postgres if you build custom extensions for it
- pgt 7y agoGreenspun's tenth rule may need to be updated for IDEs: "Any sufficiently complicated C or Fortran program contains an ad-hoc, informally-specified, bug-ridden, slow implementation of half of a JavaScript IDE." (previously Common Lisp)
- stagas 7y agoTangential, from the introduction: "Centralized systems are BORING. The early days of the web were honestly more exciting, more raw, more wild-wild-west. We need to get back to that vibe." - I adhere to that philosophy, anyone know of more projects towards that direction?
- MuffinFlavored 7y agoStoring HTML in a database is wild-wild-west? I thought it is just silly because you can't stream it from disk over a socket.
- metamet 7y agoWait until you hear about my storage array built entirely on floppy disks.
- smacktoward 7y agoPunched cards or GTFO
- erichanson 7y agoThe long-term idea with Aquameta is that users run their own local database and make connections directly to their friends/peers to basically "git pull" new content as it appears via pub/sub. Then users would do the "browsing" on localhost. I still have some work to do figuring out all that NAT piercing p2p stuff, but some combination of WebRTC and headless chrome is showing some promise.
- ficklepickle 7y agoNeat! I was wondering how webRTC fit in. Thanks for sharing, I find outside-the-box stuff like this inspiring.
- rancor 7y agoMy favorite NAT-busting mechanism is Tor hidden services, The onion network provides the basic P2P overlay as well, so it keeps life simple. STOMP over WebSockets is cool, and I believe can be implemented without using any Chrome code these days.
- endlessvoid94 7y agoI think this is an awesome idea. I really like the Smalltalk approach of not using files and instead representing the structure of a program purely in memory. I also love the idea of drawing inspiration from spreadsheets and databases instead of representing programs purely in lines of code. I applaud this effort and can't wait to try it out!
- jagged-chisel 7y agoAny long-lived executable sits in memory (notwithstanding paging, which the database server is also subject to), and such an executable is free to maintain any files it accesses in memory as well.
- gnode 7y agoI think what was being referred to is the technique of defining a program, not as source code files which are then compiled/run in an interpreter, but as an in-memory VM environment (think REPLs). The memory-based environment can then be saved to a file, similar to making a core dump. This changes the structure of programs from a file/disk based structure, to the potentially richer structure of the programming environment (e.g. graphs of objects). Although it comes with disadvantages like losing the tool inter-operability of files.
- bernawil 7y agoThat sounds like doing for code what we used to do for deployments before docker/kubernetes. Like going back from a declarative and reproducible aproach to a more procedural one. There's good reasons infrastructure as code took hold.
- breck 7y agoI have a hard time seeing advantages of ditching files. I like this concept, but you can have it both ways (come up with a graphical notation that you can then manipulate with graph like tools), but at the end of the day you need a source of truth, and the file metaphor (a named region of code) is hard to beat (impossible to beat?). Even databases store things as files.
- auspex 7y agoMake sure you back up that database...
- adipginting 7y agoLol you have a great advice over there
- jagged-chisel 7y agoExperienced something like this early in my career when a co-worker discovered PL/SQL - he campaigned for putting the entire web app into an Oracle database with all the logic in stored procedures. Experienced a slightly different take a few years later where Some Genius wrote an interpreter in stored procedures in an off-brand RDBMS. Web pages were a combination of HTML and his custom language, and these pages were stored in the database. I have strong negative opinions about putting a web development platform entirely inside an RDBMS.
- gnode 7y ago> I have strong negative opinions about putting a web development platform entirely inside an RDBMS. What about your experiences led to your negative opinions?
- ehnto 7y agoOff the top of my head: * version control * environment mismatches (think managing dev, staging, prod with multiple devs) * deployment management * discoverability of where things live so you can maintain them, and autocomplete in IDEs The last is the big one. If a client asks me to change the header, and me going to my IDE and quick searching "header" doesn't automatically show me all header related files/code then your platform has already lost the developer experience battle. Something like this would need tooling to match what is already possible in that regard.
- jagged-chisel 7y agoIndeed these were the major downsides. Further, in my Oracle instance, relying on the late-1990s Oracle database server to also run our business logic came with performance and scalability concerns, especially around licensing. In my other case, the custom language was terrible, but that's not a complaint about the use of the db engine. The interpreter was implemented in stored procedures, the pages were stored in text fields ... every bit of the stack depended entirely on this off-brand database that had its own issues (like corrupt indexes that needed repairing about once per month.) I think ultimately, in both cases it just felt so much like putting all your eggs in one basket and then hoping you never needed to augment the basket with additional baskets, nor replace the basket with one of a different shape or made of other materials.
- joewrong 7y agoreminds me of couchdb apps where you'd use document attachments to store html/public files and serve them directly from couch. https://docs.couchdb.org/en/2.0.0/couchapp/ https://docs.couchdb.org/en/2.0.0/couchapp/
- sheeshkebab 7y agoCall me old fashioned, but I prefer my web app spread across a dozen servers, file systems, hundreds of virtualized processes, and few thousand folders, and employing a small army of system enginneers (sres) and developers to maintain.
- erichanson 7y agoHa :)
- fooblitzky 7y ago"Under the hood, Aquameta is a "datafied" web stack, built entirely in PostgreSQL. The structure of a typical web framework is represented in Aquameta as big database schema with 6 postgreSQL schemas containing ~60 tables, ~50 views and ~90 stored procedures. Apps developed in Aquameta are represented entirely as relational data, and all development, at an atomic level, is some form of data manipulation." This is triggering painful flashbacks to SharePoint development.
- slifin 7y agoData driven applications are cool, I think I'd use Datomic over Postgres but it didn't exist 20 years ago
- e12e 7y ago> Technical goals of the project include: > Allow complete management of the database using only INSERT, UPDATE and DELETE commands (expose the DDL as DML) There might be other reasons - but note that postgres actually supports transactions and rollback of DDL - Oracle with full enterprise lisence has something similar (there's undo/a "trashcan" style cache for schéma modifications). But in general (as far i understand; I have not used this in anger) pg lets you simply BEGIN drop table (...) ALTER table (...) (who's, something wrong) ROLLBACK. It's very neat, and not generally supported.
- doctor_eval 7y agoYes almost all DDL is transactional in PG, and it works as you say. You can create tables, drop indexes, replace functions, update views - and then roll it all back. There are a very small number of DDL statements that you can’t do in a transaction, but the devs are fixing them (adding a value to an enum is the only one that comes to mind) We use it to great effect when upgrading schemas.
- pritambaral 7y ago> adding a value to an enum is the only one that comes to mind Adding enum values in a transaction is now supported in PG 12 as long as you don't try to use the new value in the same transaction.
- cryptonector 7y agoYes, yes, but, it's nice to be able to treat schema as relational data itself. Indeed, the information_schema and pg_catalog already let you do that, but the information_schema is incomplete, and the pg_catalog is hard to use and unstable (that is, each release can change the pg_catalog backwards-incompatibly). Besides letting you read schema via normal SELECTs on a well-designed meta-schema, something like Aquameta can let you run DMLs as DDls too. That means you can now generate and execute DMLs dynamically but without needing EXECUTE -- you can just have a bunch of normal INSERT/UPDATE/DELETE statements (including via CTEs) and generate schema from other data the same way you'd generate data from data. I think that's a big plus. But then I've worked with a database before that did this schema as data-in-a-meta-schema thing, and I find it very comfortable. For example, without a metaschema you have to use CREATE THING IF NOT EXISTS then ALTER THING ... in order to apply schema changes. Whereas with something like Aquameta you just INSERT INTO ... WHERE NOT EXISTS ... (or with ON CONFLICT DO ...) and that's that. You can have one set of DDLs that create schema, and the same set of DDLs also updates schema. That's wonderful.
- dmix 7y agoTheir argument from the introduction that software can slow down a business by making changing/evolving the systems difficult and requiring programmers at every step, is an interesting one: http://blog.aquameta.com/introducing-aquameta/ http://blog.aquameta.com/introducing-aquameta/ Obviously this is has been the panacea goal in business software forever and there's a long trail of failed companies or projects who tried to do this. IMO it's always going to require specialized knowledge of a certain level of abstraction above the machine, it will never be as simple as pushing buttons in a GUI. The question is making the languages/frameworks simpler and lowering the bar. In practice these sorts of things work great for simple scenarios but are very brittle one you start to. Just like Excel spreadsheets used like databases it will quickly turn into a hacky maze of things forced into places where it shouldn't be. I'd also never think 'PLpgSQL' when it comes to simplifying things. I'm genuinely curious to see if they can pull it off... eliminating the file system part is also an interesting idea.
- pm90 7y agoThe MO of SV startups seems to be to slap together something that has product market fit while ignoring performance and scaling concerns, at first. Once you have enough customers and market, you then raise funding and use that to hire more experienced system folks who can make your system performant, efficient and scalable.
- dmix 7y agoI wasn’t talking about scaling performance wise. I mean scaling the problem set beyond the initial easy ones.
- cryptonector 7y agoIt's not that using PlPgSQL simplifies things, but that putting business logic in the RDBMS does simplify things. It does so by letting you use an ecosystem of tools you otherwise could not use. There are several tools that let you serve a DB via any number of protocols, including RESTful APIs, for example. You could even give out direct SQL access to power users and not have to teach them how to maintain referential integrity. PlPgSQL is not a great language, and that's obviously not a reason to use it. But it's not obviously a reason to not use it, and more than that, it's no reason not to put business logic in the RDBMS. Once you decide to put business login in the RDBMS, PlPgSQL is just one of several languages you could use, and actually the most accessible one.
- msvan 7y agoThis was announced on the Future of Coding Slack [1] not long ago. I encourage people who are into things like structured/projectional editors, visual programming and other wild re-imaginations of programming to join. [1]: https://futureofcoding.org/community https://futureofcoding.org/community
- jwhiz22 7y agoThis greatly reminds me of my experience with AEM (Adobe Experience Manager). Web-based IDE, version control via OSGI bundles, content repository via Jackrabbit. I did not enjoy my time working with it but it had some neat ideas and did do some things well.
- erichanson 7y agoI'm going to Twitch stream the Aquameta install and give a little demo, and answer questions folks might have. 5pm CDT. https://www.twitch.tv/events/j5MGrQ91TwSzRpo28wqGdw https://www.twitch.tv/events/j5MGrQ91TwSzRpo28wqGdw
- breck 7y agoVery cool. Does that record? Would like to watch later.
- anthony_doan 7y agoTwitch does not record broadcast unless streamer enable the option. VOD, video on demands, for twitch stay up for 16 to 60 days (depending on sub, turbo, etc). If it's a highlight video it stays indefinately.
- thepaulstella 7y agoVOD is available - starts around 6min into the video. https://www.twitch.tv/videos/495999132 https://www.twitch.tv/videos/495999132
- erichanson 7y agoI cut out the five minutes of me trying to figure out Twitch so here's a better link: https://www.twitch.tv/videos/496396478 https://www.twitch.tv/videos/496396478
- cryptonector 7y agoI like their meta, and semantics layers. In the past I've built a scheme similar to their semantics layer, where a bunch of additional metadata is associated with schema metadata via PostgreSQL's COMMENT statement (which lets you associate a free-form text comment with all sorts of schema elements). In that scheme we had JSON COMMENTs and a set of views that ultimately generates a nice JSON representation of a database's schema, including the JSON COMMENTs in the right places, and _that_ gave us UI control that we could then use to generate Admin-on-REST UIs from. And PostgREST can be used to get an HTTP JSON API for free. For event pub/sub, I've written an used an alternative (all-PlPgSQL-coded) view materialization system that supports live-updating of materialized views as well as recording deltas, which then can be combined with NOTIFYs. Unlike Aquameta, the fact that NOTIFY requires no authorization, and its payload is free-form text, I feel queasy about sending out too much information in NOTIFYs -- instead I use them to drive queries for new deltas as recorded by the delta recorder mentioned earlier in this paragraph. The component that LISTENs for NOTIFYs then writes deltas to a file which is served with a special HTTP server that supports hanging GETs of files -- "tailfhttpd" -- and any HTTP client can then be used to tail these files. Anyways, the Aquameta scheme is pretty good. EDIT: I've been tempted to write an authorization-for-NOTIFY patch to PG... I really don't like the idea that if I give someone direct access to the DB they can NOTIFY anything they like on any channel.
- erichanson 7y agoDang some cool ideas in here. Yeah trying to figure out how to annotate the schema was a big motivator for inventing the meta identifier system. Thought about using COMMENT for documentation, but once you have meta-ids, I thought it was cleaner to just put schema annotations in a different table. Yeah I think the LISTEN/NOTIFY system in PostgreSQL is a bit primitive. Still working on that part of the project. We got NOTIFYs to propagate up through nginx over a WebSocket and get them into web-world that way, but that section of the project is still fairly immature.
- cryptonector 7y agoThanks. You can find some of that code here: https://github.com/twosigma/postgresql-contrib https://github.com/twosigma/postgresql-contrib I need to finish the open sourcing of tailfhttpd though. It's... an open-coded HTTP server, written in C, specifically tailored for tailing files over HTTP. It uses epoll and inotify, and is C10K, and blazingly fast. You have to front it with nginx or envoy to get TLS support, naturally. Because it's so simple, open-coding HTTP/1.1 seemed reasonable at the time, though nowadays writing this in Rust with some reasonable framework would be better. The key to tailing files over HTTP is to have a server that supports: - weak ETags (st_dev, st_ino, generation number) - If-Match and If-None-Match conditional request support - Range requests - when the right end of the requested range is left unspecified, use chunked transfer-encoding and don't send the terminator chunk until a) EOF is reached, and b) the file is renamed out of the way or unlinked (which the server can detect using inotify or similar) With this clients can tail a file. Heartbeats can only be supported by the application itself -- HTTP/1.1 does not have the ability to send empty non-terminating chunks (HTTP/2.0 doesn't have that problem). If client loses its tail, it can simply resume by using If-Match and Range starting at the next byte offset after the last byte received before. So recovery is trivial. And presto, super cheap, simple, and reliable pub/sub.
- xet7 7y agoInterviews of Aquameta: https://twit.tv/shows/floss-weekly/episodes/527 https://twit.tv/shows/floss-weekly/episodes/527 https://twit.tv/shows/floss-weekly/episodes/449 https://twit.tv/shows/floss-weekly/episodes/449
- miffy900 7y agoThis is actually really similar to on-premise SharePoint development; list data, content types, workflow definitions, front-end HTML, CSS, JS code are just stored in the back end database.