5 ms·
Interesting ideas in Observable Framework
- simonw 3y agoIn a way, Observable Framework is the Avengers: Endgame of the Mike Bostock Cinematic Universe. It brings together d3, Observable, Observable Plot, HTL and layers on a bunch of new ideas as well.
- codetrotter 3y agoThanks for your writeup on Observable Framework. I had bookmarked both Observable, and Observable Framework before, but not started looking into the details of it. Today I was beginning to look at how to host a static Jupyter Notebook, or hosting it interactive with WASM. But actually I think that for most of my purposes Observable Framework will be a better fit.
- drewda 3y agoFor what it's worth, Polymaps is probably still my favorite of his creations...
- NelsonMinar 3y agoNot to forget: Polly-B-Gone https://cs.stanford.edu/people/mbostock/polly/ https://cs.stanford.edu/people/mbostock/polly/
- ak39 3y agoThanks for the easily understandable TLDR!
- luke-stanley 3y agoI have a feeling that we're making things for "human augmented" AI developer agents! Observable have existing AI integration, to me it seems like this could just be a more easy to compose wrapper for it to make use of! Your strategy assessment didn't sit right without AI. Thanks for the nice write up.
- floodle 3y agoI just feel like they are limiting their user base by only supporting Javascript. It's of course the de-facto language for interactive display in browsers. The use case for dashboards and data visualisation is clear. But it's an awful language for data science and data analysis, compared to Python or R.
- no_wizard 3y ago>It's of course the de-facto language for interactive display in browsers This is it, more or less. It is far, far easier to build an app like this where you want a plethora of users as a web application than a native one, for instance. For anything JavaScript as a runtime / language is missing, WASM can boost as well. For math and data science, WASM is a natural choice for any missing pieces
- FarhadG 3y agoCan you elaborate further on what you have in mind here?
- jamra 3y agoI used Pyodide (https://pyodide.org/en/stable/ https://pyodide.org/en/stable/) as a python execution environment in browser. That was pretty successful. There are also python libraries that let you generate a config which is later made responsive via a JS library. Pyodide runs in wasm.
- simonw 3y agoOne of the neat new features of Observable Framework is you can drop in a build script to gather the data that's written in any language you like. https://observablehq.com/framework/loaders https://observablehq.com/framework/loaders So you absolutely can do the data processing step in R or Python and have that output JSON or CSV which is then visualized at the end using JavaScript. Not a small feature, but I bet it would be possible to use WebAssembly to add support for Markdown blocks that get executed in other languages as well, using Pyodide for Python for example.
- jarpineh 3y ago
- willmeyers 3y agoThe author’s spot on about framework. I tried out Observable Framework and built a little interactive plot (https://github.com/willmeyers/observable-ssta https://github.com/willmeyers/observable-ssta). It was incredibly easy to setup and get data plotted. My only gripe is that I wish you could configure Python data loaders to use virtualenvs.
- simonw 3y agoCan you put a shebang line in a .sh data loader that points to the full path to bin/python within the virtual environment directory?
- timmattison 3y agoYou can. But then the only time it realizes that the code has been updated is when you update the script or touch it. It’s a minor annoyance but it adds up when making lots of changes. Periodically deleting the cache works too but also annoying.
- learned 3y agoI have a setup with poetry that runs the python data loaders in the poetry-managed virtualenv. I just created a python project and then instead of `yarn run dev` to start the dev server, just run `poetry run yarn run dev` so the python is executed within the virtualenv. This setup also lets you use a custom python package to define reusable and unit-testable code for the dataloaders that you can import into the *.json.py files to keep those really simple.
- eacapeisfutuile 3y agoWhy do you need to bundle these, is it to simplify iterating on frontend and data loaders simultaneously? Why not run them separately?
- learned 3y agoIt’s totally possible to decouple these if your python outputs plain JSON/csv into the data/ directory that you commit into the repo or generate just before build time. Then you can import that raw json data into an Observable .md file. But if you want dynamically generated data at build time and want to make use of Observable’s dataloader automatic execution of data/*.json.py, for instance, while still maintaining a custom virtualenv for the project rather than your system python, you’ll need some way to specify that virtualenv’s interpreter while observable executes the build for the dev server or the full dist/ output. So for both options it’s largely a matter of taste. I personally like using the poetry virtualenv because it’s simple to manage dependencies and the venvs in one tool, while letting me use observable’s dataloaders with third-party or custom python packages. It sounded like the parent comment wanted to use this type of approach so I focused it to that scenario specifically. I like the simplicity of the single command to generate the data and build the site.
- llimllib 3y agoFramework is also super easy to publish to a github site, I wrote up a note with the steps and a sample github action: https://notes.billmill.org/programming/observable_framework/github_workflow_for_publishing_an_observable_framework.html https://notes.billmill.org/programming/observable_framework/...
- qwertox 3y agoMy issue with Observable is that it appears to be the examples-resource for d3 [0], but you can't just copy-paste the code because it is designed to run in that framework. And it's not like d3 is easy to use so that you can use it without examples, specially considering that changes between versions are often incompatible. But apart from this, there's a lot of incredible graphics to find on the site. [0] https://observablehq.com/@d3/gallery https://observablehq.com/@d3/gallery
- btbuildem 3y agoYup, I find that incredibly frustrating -- it's a platform lock-in that any corp would be proud of. I've had this gripe more with ObservableHQ notebooks -- great examples and a pointless resource all at the same time. This framework effort seems to be a bit more open though (at least you can self-host), so I'm keeping an eye on it.
- kragen 3y agowith alex garcia's dataflow it has been possible for some time to self-host observablehq notebooks without proprietary software: https://github.com/asg017/dataflow https://github.com/asg017/dataflow i, embarrassingly, haven't tried it
- deleted 3y ago[deleted]
- dleeftink 3y agoSome community members have made available resources for converting Observable flavoured JS to vanilla (it mostly involves rewriting top-level cell definitions): [0]: https://observablehq.com/@bumbeishvili/convert-observable-code-to-normal-javascript https://observablehq.com/@bumbeishvili/convert-observable-co...
- iansinnott 3y ago100%. Was never able to get past the fact that it's slightly just not quite actual JS. Close enough to the base language that it seems they could have easily used JS, probably with some additional APIs for showing graphics.
- zX41ZdbW 3y agoObservable integrates really well with ClickHouse using its REST API, like in this example: https://observablehq.com/@stas-sl/github-issues-survival-analysis https://observablehq.com/@stas-sl/github-issues-survival-ana... But I didn't try the new Observable Framework - interesting to see similar examples where it queries a database live. I hope that preloading and caching all the data is not the only option because these types of apps should be interactive. Ideally, it should expose SQL for live editing.
- simonw 3y agoFetching data live still works - the static data loader piece is optional. My demo here uses fetch() to load data at runtime: https://simonw.github.io/observable-framework-experiments/package-downloads https://simonw.github.io/observable-framework-experiments/pa...
- orf 3y agoThis looks fantastic! I’ve been waiting for something like this. My only gripe is that data loaders don’t seem to support Parquet files, which is really annoying. There’s an interesting possibility here where you can have large datasets in Parquet, exposed via HTTP, whilst being generated at build time with all the benefits that gives you (being able to read only specific columns, filtering via row group statistics etc). Not dissimilar to the “SQLite over http” WASM demo I guess. Because right now I need to take my large, nicely compressed dataset and export it as either a CSV or a zip file? And then the browser needs the entire thing, even if I’m just viewing a subset of the data? Which is much bigger and much slower than it needs to be.
- simonw 3y agoThere's a parquet example here: https://github.com/observablehq/framework/blob/main/examples/api/docs/index.md?plain=1 https://github.com/observablehq/framework/blob/main/examples... Using data from here: https://github.com/observablehq/framework/tree/main/examples/api/docs/data https://github.com/observablehq/framework/tree/main/examples... Rendered version here: https://observablehq.com/framework/examples/api/ https://observablehq.com/framework/examples/api/ If you want to run a Data loader that outputs to parquet there are plenty of ways to do that - I would suggest a Bash or Python script that wraps DuckDB.
- mbostock 3y agoWe do support it. (And use it!) Please see: https://observablehq.com/framework/lib/arrow#apache-parquet https://observablehq.com/framework/lib/arrow#apache-parquet
- orf 3y agoAhh, amazing! That’s not entirely clear from the data loading docs[1], which when I read it seemed only focused around CSV and JSON. 1. https://observablehq.com/framework/loaders https://observablehq.com/framework/loaders
- nojito 3y agoObservable still pales in comparison to Quarto. https://quarto.org/ https://quarto.org/
- eacapeisfutuile 3y agoIn what way? Isn’t a big part of observable the community provided content, which does not seem to be what this tool provides?
- kragen 3y agohow?
- spinningslate 3y agothat's an unhelpful articulation that runs the risk of having the opposite effect from the one you want. That would be a pity, because Quarto is really good. I haven't tried Observable yet, but in outline they have some similarities: 1. Documents written in Markdown 2. Ability to embed code blocks in the Markdown, with code executed when the document is rendered. 3. Ability to embed output of the code blocks in the rendered result (e.g. tables, charts). 4. Ability to render to multiple formats (pdf, static site, ...). Quarto supports Python and R as languages in the code blocks (maybe more, not sure). I personally prefer it to Jupyter notebooks because the source is plain text so (1) there's a choice of editor and (2) moving between text and code blocks is seamless. I can't say Quarto is better than Observable but it is good. It has depth from its history in RMarkdown (like rendering mathematical equations, naming & cross-referencing). It's certainly worth consideration for anyone looking for a "code notebook" solution.
- tel 3y agoI recently finished my first "in anger" project with an Observable notebook. This involved learning Observable Plot, Arquero, relearning bits of Javascript, and integrating it with a Rust based simulator that's my data generation process. It's honestly been really wonderful. Learning all of those tools has taken some significant energy, and I'm missing some functionality I'd love around parameterizing my data generator, but the final notebook is beautiful and functional. Using markdown and reactivity makes notebooks like this actually feel usable. Jupyter's custom format made version control a giant pain and without reactivity your iteratively designed notebook easily becomes a write-only, stateful mess. I've also tried making this work using Quarto and their Observable integration and it was hacky and piecemeal. Genuinely, this was the first time I've been pleasantly surprised and excited to write a notebook and share it with others. I'm sure there will be more sharp edges, but it's become my first choice notebook tool after this project.
- dleeftink 3y agoFor those looking for an alternative to Quarto, check out the recently released Living Papers for authoring reactive/static documents from a single source: [0]: https://living-papers.vercel.app/ https://living-papers.vercel.app/
- dleeftink 3y agoIf you want to quickly try and tinker with Framework in your browser, I've set up some Codespace devcontainers that automatically configure Node and Python environments here: [0]: https://github.com/dleeftink/observable-codespace https://github.com/dleeftink/observable-codespace
- LuciBb 3y ago[dead]
- klysm 3y agoI don't know much about observable, but it seems like they might be hijacking d3 a bit too much for my taste - it makes me a bit nervous about the future of d3.
- simonw 3y agoObservable is from the same creator as D3 (Mike Bostock) and D3 has been a core component of the Observable platform since they first launched their notebook product back in 2018. I don't see Framework changing things there - if anything the ISC license should make it a better partner for D3.
- theK 3y agoParaphrased from the article: > Everything in a code block with the js content hint will be executed in the users browser immediately. If you want to show the code you have to hint 'js echo' Am I the only one thinking that it would have been better for backwards compatibility if it where the other way around? I.e: having an opt-in code hint like 'js exec' that runs code in a user's browser and leaving the widely used 'js' hint alone? The way this currently is set up, you cannot integrate that renderer in an existing app without having to manage where it is allowed to run.
- mbostock 3y agoWe’re planning to allow changing the default options for blocks (either per-page in front matter or across an entire project using the project config); you could then make `js run=false` the default and `js run` to opt-in to live code as you wish. But we chose to make live code the default since that’s our primary use case.
- crabmusket 3y agoThis is great to hear, though your decision does make sense! I'm really keen to play with Framework.
- crabmusket 3y agoYes, it has the same issue as e.g. automatically rendering Mermaid diagrams on GitHub: now you can't just show a block of Mermaid code, without dropping the language annotation.
- wodenokoto 3y agoShould I move from jupyter notebooks to Observable? Or is that the wrong dichotomy?
- tomgp 3y agoI think the question is are you likely to be more productive with python and its attendant ecosystem or js and the packages it has available (most saliently D3 etc.)
- wodenokoto 3y agoI was hoping for some more opinionated responses, but I guess the truth is always “it depends” Asking between Python and R tends to get people to throw around opinions.
- tomgp 3y ago:D I mean personally I prefer Observable becasue my interest is primarily in data presentation. I imagine if you're more on the analysis side of things (tidyverse etc) Jupyter would still be the way to go.
- lakomen 3y agoSo, instead of HTML you have to use Markdown and special tags if you want to use JS. I don't see the advantage
- simonw 3y agoTake a look at my dashboard example, it should help show why this is useful. You can get a lot done with very little combined Markdown and JavaScript - building the same thing in HTML plus JavaScript would have taken a bunch more code. Demo: https://simonw.github.io/observable-framework-experiments/package-downloads https://simonw.github.io/observable-framework-experiments/pa... Source code: https://github.com/simonw/observable-framework-experiments/blob/main/docs/package-downloads.md?plain=1 https://github.com/simonw/observable-framework-experiments/b...
- johnnunn 3y agoJust wondering, can one embed the observable framework page into another site ? Or should it have to be a separate static site as demoed in the website.
- skadamat 3y agoSimon -- have you played with Evidence.dev much?
- jsnelgro 3y agoI spent a night going down a rabbit hole with observable framework and it was terrific! It more or less just got out of the way and I was able to visualize and explore my google maps history in detail. Some of the data loader environment stuff wasn't especially clear but running in a poetry env did the trick for python. I love kotlin and tried creating a data loader for a kotlin script but that had some rough edges. Kotlin expects script files to be named foo.main.kts but observable expects executable shebang loaders to have a foo.exe extension. So I created a proxy exe script to call the kotlin script, but it then doesn't trigger auto reloads of the data. A bit of friction compared to marimo or jupyter is using variables between data loaders and the notebook. For example, I want to use the date picker view component to change the range of data fetched by my loader. It's not clear how to do that, so exploratory analysis is slowed down a little. I'm aware this goes against the paradigm but just wanted to point it out. It ends up with you potentially moving a lot of the data munging to the notebook as you explore, which isn't ideal from a performance perspective. One last thing is I wish you could define dataloaders inline. I'm a big fan of single files, so being able to just add a python code block and let Framework extract that as a file would be a nice little QoL improvement. Still the early days, but Framework seems promising! I'd love to have my all my markdown notes running through it to get a sort of org-mode type situation without going full emacs.
- mbostock 3y agoThanks for the feedback. We have a PR open to make it easier to register new interpreters (without needing to fallback to .sh or .exe); it’ll let you specify the interpreter associated with a given file extension (e.g., .kts for Kotlin). https://github.com/observablehq/framework/pull/935 https://github.com/observablehq/framework/pull/935 As for inputs-driving-data-loaders, that does go against the grain a bit since Framework favors static data snapshots so that the built site is self-contained and performant. But a technique that works well is to generate Parquet files in data loaders representing the superset of data that you want to interact with, and then using DuckDB/SQL in the client to extract the subset you want to visualize. This tends to perform well, though obviously it’s dependent on the size of the superset you want to interact with.