10 ms·
How to create a dashboard in Python with Jupyter Notebook
- oneoff786 4y agoMy initial reaction is that there’s a ton of lag on the initial gif. Makes me think this is going to be a cumbersome thing to work with. Is it representative of a typical task, whatever that means?
- pplonski86 4y agoThe demo is hosted on free Heroku dyno with 0.5 GB RAM. This is version 1.0 and still many things can be improved in the Mercury framework.
- lmeyerov 4y agoAwesome to see! We've been encouraging Graphistry users to look at Streamlit for the low-code side (and getting bundled into our next release for public+private pydata GPU dashboarding!) or Databricks dashboards. Then at the expense of significantly more code / complexity without going full Flask/Django, we generally recommend switching to Dash/Voila/etc for getting way more control. Mercury seems quite close to the Streamlit/Databricks side, which in our experience is right for many pydata dashboarding teams yet most tools don't really nail. It may even be on the path to one-upping streamlit on the simplicity side. Good job & will be tracking!
- pplonski86 4y agoThanks! Mercury's goal is to make notebook sharing as easy as possible for creators (developers) and viewers (non technical users).
- criddell 4y agoEver since I read about the Panic status board project [1] I've been looking for similar projects. This one looks pretty cool. [1]: https://panic.com/blog/the-panic-status-board/ https://panic.com/blog/the-panic-status-board/
- westurner 4y agoSurprised I hadn't heard of mplfinance. Similar code for econometric charts with bands and an annotation table with dated events would be useful and probably already exists somewhere. Maybe Predictive Forecasting libraries for time series already have something that could be factored out into a maintained package for such common charts? E.g. seaborn has real nice confidence intervals with matplotlib, too; though the matplotlib native color schemes are colorblind-friendly: "Perceptually Uniform Sequential colormaps" https://matplotlib.org/3.5.0/tutorials/colors/colormaps.html https://matplotlib.org/3.5.0/tutorials/colors/colormaps.html awesome-jupyter > Rendering/Publishing/Conversion https://github.com/markusschanta/awesome-jupyter#renderingpublishingconversion https://github.com/markusschanta/awesome-jupyter#renderingpu... : > ContainDS Dashboards - JupyterHub extension to host authenticated scripts or notebooks in any framework (Voilà, Streamlit, Plotly Dash etc) IIRC, ContainDS and Voila spawn per-dashboard and/or per-user Jupyter kernels with JupyterHub Spawners and Authenticators; like Binderhub ( https://mybinder.org/ https://mybinder.org/ ) but with required login and without repo2docker? Streamlit lists Bokeh, Jupyter Voila, Panel, and Plotly Dash as Alternative dashboard approaches: https://github.com/MarcSkovMadsen/awesome-streamlit#alternative https://github.com/MarcSkovMadsen/awesome-streamlit#alternat... Says here that mljar/mercury is dual-licensed AGPL: https://github.com/mljar/mercury https://github.com/mljar/mercury
- westurner 4y agoFrom the readme I can't tell what powers the cited REST API functionality. From setup.py > mercury/requirements.txt it's DRF: Django REST Framework. https://github.com/mljar/mercury/blob/main/mercury/requirements.txt https://github.com/mljar/mercury/blob/main/mercury/requireme... E.g. BentoML is built on FastAPI which is async (sanic) and built by the DRF people, but FastAPI doesn't yet have the plethora of packages with tests/ supported by the Django community and DSF Django Software Foundation.
- pplonski86 4y agoThe REST API functionality allows to execute the notebook by POST request. It is powered by DRF. You can read more in the docs https://mercury-docs.readthedocs.io/en/latest/notebook-as-rest-api/ https://mercury-docs.readthedocs.io/en/latest/notebook-as-re...
- kaa2102 4y agoMercury is a great find for me as it solves UI/UX design & usability concerns for my Python automation projects for non-techninal users.
- anigbrowl 4y agoThis is great! I have a lot of tools built in notebooks that I'd like to be able to share with non-technical collaborators, and have been procrastinating building a UI in Flask or Django as I'm really not interested in front-end stuff. This is a fun and simple way to deploy that won't be a distraction from collection and analysis I actually enjoy. I look forward to using it!
- pplonski86 4y agoThe Mercury framework was designed to make notebooks sharing easy. You don't need to build UI. Just define the widgets in the YAML header. Widgets are directly connected to variables in Python code. You can share notebook with non-technical users and hide the code (show-code: False). The user can tweak widgets and execute the notebook. Additional features for sharing: - user can download notebook as PDF or HTML, - you can schedule notebook execution, and send executed notebook as PDF attachment in email notification, - adding login view to notebooks is simple, just add `share: private` to YAML
- crisbal_ 4y agoHow does this compare to Voila from the Jupyter team? ( https://voila.readthedocs.io/en/stable/index.html https://voila.readthedocs.io/en/stable/index.html )
- pplonski86 4y agoMercury has a different architecture. The Voila keeps the live kernel and connection to UI (using Tornado framework). To use Voila you need to add widgets (with ipywidgets) to the notebook (mix UI code with analytics code). The Mercury generates UI based on the YAML header (very simple, no need to mix UI with analytics code). When user tweaks widgets values, the whole notebook is executed with new parameters and converted with nbconvert (using Django + Celery). Mercury can serve multiple notebooks to multiple users on one server. It has option to export notebook to PDF or HTML. You can schedule the notebook execution with crontab string (for example `schedule: '30 8 * * 1-5'`) and add email notifications. You can easily add authentication to notebooks. It was designed to make notebook sharing fast and easy. What is more, I'm thinking about including Voila into Mercury. So you will be able to serve Voila apps with Mercury. The end-goal is to make notebooks sharing easy. Mercury github repository https://github.com/mljar/mercury https://github.com/mljar/mercury
- sdoering 4y agoThis is great. Just sent the link to colleagues looking for an easy solution to building and sharing dashboards.
- urthor 4y agoYMMV, has any technologist found a dashboard useful? I've had a very bad history with dashboards. All I've seen get ordered up by nontechnical leadership, not looked at, and involved endless fiddling with painful corporate applications. The non-technical stakeholders I've had a good relationship with, I vend a SQL script/Jupyter Notebook and educate. They're happy to up-skill. I've only had dashboards requested by users with a dysfunctional relationship with engineering, in companies where the data is irrelevant. In environments where data matters, fast moving consumer goods, web, etc, stakeholders speak SQL/Jupyter. I've seen remarkably non-technological people learn SQL and query read replicas because they need sales aggregates. In environments where data doesn't matter, dashboards are requested as a corporate power play. A director who doesn't care about your project asks for a dashboard to assert their authority, and remind you your budget can be cancelled.
- pid-1 4y agoYes I've seen many good uses of dashboards. That said, I totally get where are you coming from, been there as well.
- cirgue 4y agoDashboards are really useful when you’re trying to distribute a set of metrics that large numbers of people actually need to know and agree on in order to do their jobs. Most metrics aren’t like this, so there are a lot of bad dashboards, but they definitely have their place. Edit: A good example of this is something like incoming shipments for a warehouse. Management all the way down to warehouse workers need to know what is slated for delivery, what’s already been delivered in the last few days, and what’s late.
- javier_e06 4y agoLook at my latest gizmo...on a drone! Look at my latest data crunch app...on the web! Yep, smells like management all over.
- quinoablast 4y agoYou’re definitely right that metrics are often used as post-hoc justification of decisions already made. An example of actual dashboard culture is AWS’s operational metrics meetings. The construction of the dashboards themselves is scrutinized just as much as any latency or availability metric (may God have mercy on your soul if use a pie chart or display the median of metrics without justification). And it’s actually useful for surfacing issues and knowledge sharing. The frequent cadence can be a big cause if stress though. https://aws.amazon.com/blogs/opensource/the-wheel/ https://aws.amazon.com/blogs/opensource/the-wheel/
- curiousgal 4y agoNothing beats R's Shiny for making dashboards from my experience.
- discordance 4y agoR/Shiny is sweet. Since stepping into the python DS world I’ve used Streamlit to roughly the same effect.
- pplonski86 4y agoR Shiny is fantastic solution. To be honest, the Mercury is heavily inspired by R Markdown.
- swyx 4y agothere’s nothing inherent about why we cant port Shiny’s features into the Python ecosystem. what do you like most about Shiny that is missing? (from Mercury or otherwise)
- nicolaskruchten 4y agoWe've got a number of solutions in Python that rival Shiny IMO: Dash, Voilà, Panel, Streamlit, new stuff like Mercury etc :) There's a neat overview of the landscape here https://www.youtube.com/watch?v=4a-Db1zhTEw https://www.youtube.com/watch?v=4a-Db1zhTEw
- db65edfc7996 4y agoEhhhhh, I'm a Python guy, and I find all of the Python Shiny alternatives lacking. Shiny delivers an incredible amount of value per line of code. My experiments with Dash and Streamlit required a lot more boilerplate to get something running vs the R equivalent.
- pplonski86 4y agoI love your attitude!
- sireat 4y agoMercury looks rather neat for quick dashboards. For dynamic data I build my dashboards with Flask and Plotly. However that is certainly a more involved process to migrate Jupyter notebook Python plots to JS plots. For static data one can just export html(with embedded JS) from Jupyter notebook that uses plotly. The main difference with quick solutions is the level of customizability.
- pplonski86 4y agoMercury works very well with Plotly (here is a demo https://mercury.mljar.com/app/7 https://mercury.mljar.com/app/7). If you need to choose between Flask+Plotly and Mercury I would strongly recommend Mercury because of the speed of development. What do you mean by dynamic data? Do you need automatically fetch new data in time intervals? (Mercury has scheduling and auto-update built-in with 1 minute lowest time interval).
- pea 4y agoRe your HTML point, we've been hacking on a framework which lets you programmatically create and share data reports in Python which you might find helpful (https://github.com/datapane/datapane https://github.com/datapane/datapane). It supports Plotly, Pandas, Altair, Folium, MPL, etc., and provides some neat layout components like pages, selects, columns, and dropdowns. If you need anything or want some help, feel free to make an issue or ping me on leo [-at-] datapane.com
- blagie 4y agoI looked over the web site. A few questions: - Why won't the OSS version provide features like history of execution? It seems like those are pretty basic to adoption. I'm working on an open source project, and it'd be a nice tool to potentially integrate. The commercial license is of zero interest since it's, well, all open-source. - Why don't you sell the OSS version as well (e.g. for people who want to support you, or who want support)? - Do you provide consulting services? Are they in the $50/hour range (just extrapolating from $200/year for up to 4 hours support) - Why Django? I like Django, but it seems like using the same tech stack as Jupyter would make for simpler integration, administration, and lower TCO (plus the universe is going async) - For presentation, the speed is an issue. Regeneration takes a while (I'm playing with the demo on Hugging Face). It's not quite adequate for interactive presentation. I do really like the concept of making presentations in Jupyter. - And on that topic, the biggest gap I have is with interactive front-ends. I'd like to be able to somehow connect notebooks (both for dashboards and otherwise) for data munging to data visualization frameworks like D3. Jupyter is nice. Observable would be nice if it were open-source. There's a mile-high wall between the two.
- pplonski86 4y ago- What is your project? is it from data science space? - Selling OSS might be a great idea. Didn't consider it. I thought the Personal Pro version ($199/y) will be good for persons that would like to support the project or just need a support/guidance. - Yes, I do consulting services. I don't treat support hours in similar way as consulting. The support is for product that I love working on and I love supporting people using it. - Django - I'm pretty good at Django. What is more, I wanted to use Celery for processing background tasks and scheduling. This gives an architecture that can handle large loads. The frontend is in React with TypeScript. - Speed can be improved. There are many ways to better handle that (it is version 1.0). You can add Plotly charts to make presentation more interactive. The advantage of Mercury is that you can recompute all slides if needed.
- blagie 4y agoI don't disclose my identity online, and I think if I posted the project (or a description), it would do that. If you're curious, I'm glad to reach out by email. It is a data science project, and potentially, high-profile if it goes well. On a mile-high level, Mercury seems like a perfect fit for what I'm doing. Conceptually, it's exactly what I was looking for. Diving into the code, the fit seems less great, for reasons which are incidental (and wouldn't be worth changing for one integration). I'm trying to constrain the number of technologies in the overall project, to make it maintainable and deployable, and you seem to be on a completely orthogonal stack. So I'll keep diving in and trying to understand it better.
- maliker 4y agoWhen I need to expose jupyter notebooks to non-technical users I put them on colab.research.google.com. It's a free, interactive, hosted Jupyter environment. It's not as slick as a dashboard, but it just requires an upload instead of maintaining a server.
- pplonski86 4y agoWith Mercury you can: - host multiple notebooks, - you can easily hide code, - you can schedule notebooks, - you can download notebooks as PDF, - you can easily add authentication to the notebooks. It all depends on your needs and project requirements.
- eoinmurray92 4y agoyou might want to give Kyso a try (Im a cofounder): https://about.kyso.io/ https://about.kyso.io/ We render jupyter notebooks, markown and html - we dont host running jupyter environments -> but we focus on making the experience of sharing with non-techies on your team easy. So can show/hide the code, get comments etc.
- analog31 4y agoAsk HN: Any idea if jupyter lite is a contender for sharing notebooks?
- pplonski86 4y agoIt might be a solution but not all python packages work with WebAssembly. The next problem might be connection to external data sources. You cant keep secrets in frontend code.
- analog31 4y agoGood point about secrets. Fortunately everything I write is for "internal use only" at my workplace, or if it's a home project, then I share everything. But I can see having to warn a non-techie that you're giving something which could reveal secrets if they pass it on to a third party. Especially since the secret stuff will be invisible to them unless they ask.