11 ms·
Bento: Jupyter Notebooks at Meta
- 01222685480 2y agoFady11857@gmail.com 01222685480 F
- web3aj 2y agoThe internal tools at Meta are incredible tbh. There’s an ecosystem of well-designed internal tools that talk to each other. That was my favorite part of working there.
- Random_BSD_Geek 2y agoPolar opposite of my experience. To achieve the technical equivalent of changing a lightbulb, spend the entire day wrangling a dozen tools which are broken in different ways, maintained by teams that no longer exist or have completely rolled over, only to arrive at the finish line and discover we don't use those lightbulbs anymore. Move things and break fast.
- extr 2y agoYeah 100%. I found it immensely frustrating to be using tools with no community (except internally), so-so documentation, and features that were clearly broken in a way that would be unacceptable for a regular consumer product. If you have a question or error not covered by an internal search or documentation, good luck, you'll need it. Literally part of the reason I left the company.
- landedgentry 2y agoWell, you're supposed to read the code and figure it out. And if you can't, you're not good enough an engineer. According to people at Meta.
- extr 2y agoPeople probably think you’re exaggerating but it’s true. Sometimes when I would get blocked the suggestion was to “read the source code” or “submit a fix” on some far flung internal project. Huge fucking waste of time and effort, completely unserious.
- hnav 2y agoDoesn't sound like your type of company tbh, the flipside is that a "serious" company will often have broken bs too except now nobody is going to look at your contribution/fix.
- KaiserPro 2y agoPfft. "your type of company" sod off. Meta is only like this because its got a massive advertising revenue stream. the sheer amount of engineering time wasted because we don't document stuff is astounding. For example, how many message queue systems do we have? how many half arsed message queues have been created because they didn't know about FOQS?
- extr 2y agoYes lmao, the number of times I would start off on some nominally useful task only to find out 3 weeks later that there is actually already a solution to that created by team XYZ that nobody in my reporting chain has ever heard of…(3 weeks was optimistic case, I remember my team member getting like 2 months in to some new data pipeline before finding out some tables already existed that did what he needed…)
- lclarkmichalek 2y agoI think a fair few of them were created because they knew a bit too much about FOQS
- tru3_power 2y agoNo matter what, tools will be broken. Having access to the source and being able to land a diff to fix the issue is awesome imo.
- zer0zzz 2y agoAgreed. I often get my work done using open source build instructions and tools and then when everything works I port it to internal infra. Other people are the opposite though, which for open source based code bases has a nasty side effect of the work having no upstream able tests!
- loeg 2y agoIMO there's a mix of a few really good, widely used, well-supported tools as well as a long tail of random tiny tools where the original team is gone that are cruftier.
- uuddlrlrbaba 2y agoMmm breakfast
- grantsucceeded 2y agohaha the reason I stayed as long as i did
- deleted 2y ago[deleted]
- bozhark 2y agoMove Smooth and Fix Things (tm) is our nonprofit corporation’s version of this atrocious motto.
- ElonChrist 2y ago[dead]
- ec109685 2y agoLarge checkouts is a solved problem now https://github.com/facebook/sapling/blob/main/eden/fs/docs/Overview.md https://github.com/facebook/sapling/blob/main/eden/fs/docs/O...
- aprilthird2021 2y agoBut you're both talking about different things. The tools are both often left in disuse, lacking documentation, etc. But they also have a really tight integration with each other that allows for unparalleled visibility and ability over enormous systems with many moving parts.
- deleted 2y ago[deleted]
- crabbone 2y agoA friend of mine is doing his PHD while being an intern at Meta. He does not share your excitement... at all. To summarize his complaints: a framework written a long while ago with design flaws that were cast in stone, that requires exorbitant effort to accomplish simple things (under the pretense of global integration that usually isn't needed, but even if was needed, would still not work).
- slt2021 2y agohow else can you build empire as Engineering Manager and get promo? fork open source, then demand resources to maintian this monster. easiest promotion + job security. its even called "Platform Engineering" these days
- almostgotcaught 2y ago> A friend of mine is doing his PHD while being an intern at Meta I interned thrice as phd student at FB. your friend isn't entirely wrong but also just doesn't have enough experience to judge. all enormous companies are like this. FB is far and away better than almost all such companies (probably only with the exception of Google/Netflix).
- jonathanyc 2y agoAgreed. I'm reading some complaints in the thread about being told to "just read the source code" for internal tools at Meta. When I worked at Apple we didn't even get the source code!
- crabbone 2y agoI don't see why saying that Facebook's tools are bad should be invalidated by saying that Google's or others' tools are bad too. Google being bad doesn't vindicate or improve Facebook tools. There's no need for perspective: if it doesn't work well for what's it designed to do, then that's all there is to it.
- almostgotcaught 2y ago
- jchonphoenix 2y agoMeta tools are best in class when the requirement is scale. Or that the external tools haven't matured yet
- JohnMakin 2y agoOne of the crazier things a L4 meta colleague of mine told me, that I still don’t believe entirely, is that meta pretty much has their own fork of everything, even tools like git. is this true?
- sdenton4 2y agoIt wouldn't be terribly surprising. Forking everything provides a liiiitle bit of protection against things like the 'left pad' incident.
- deleted 2y ago[deleted]
- 3eb7988a1663 2y agoLeft pad was from the creator pulling the code from the public source forge, not from a destructive code change. I assume all of the big tech companies host internal mirrors of every single code dependency + tooling. Otherwise they could not guarantee that they can build all of their code.
- tqi 2y agoFacebook actually doesn't use git, they use mercurial (https://graphite.dev/blog/why-facebook-doesnt-use-git https://graphite.dev/blog/why-facebook-doesnt-use-git). That decision is also illustrative of why they end up forking most things - Facebook's usage patterns at the far extreme end for almost any tool, and things thats are non-issues with fewer engineers or a smaller codebase become complete blockers.
- kridsdale3 2y agoYes when I used to talk about this to interviewees, I described that every tool people commonly use is somewhere on the Big-O curves for scaling. Most of the time we don't really care if a tool is O(n) or O(10 n) or whatever. At Meta, N tends to be hundreds of billions to hundreds of trillions. So your algorithm REALLY matters. And git has a Big-O that is worse than Mercurial, so we had to switch.
- Qshdg 2y agoLooking at some of the bureaucracy in their open source projects, I'd say that they need less tooling and more thinking. These tools help to keep spaghetti code bases from imploding totally.
- moandcompany 2y agoMy opinion: Many Meta tools and processes seem like they were created by former Googlers that sought to recreate something they previously had at Google, during the Google->FB Exodus, but also changed aspects of the tool that were annoying or diverged from their needs. This is not a bad thing. Since Bento doesn't appear to be usable by the public, aparallel version of this that people can get a feel for cross-tool integration would be Google's Colaboratory / Colab notebooks (https://colab.research.google.com/ https://colab.research.google.com/) that have many baked-in integrations driven by actual internal use (i.e. dogfooding).
- kridsdale3 2y agoAs someone from both, I confirm/support your opinion 100%.
- mark_l_watson 2y agoI agree, the paid for Pro version of Colab just seems to have the features I need. I often use it because it simply saves me time and hassles.
- baggiponte 2y agoUuuh can you tell a bit more about wasabi, the Python LSP? Saw a post years ago and been eager to see whether it’d be open sourced (or why it wouldn’t).
- KaiserPro 2y agoYou and I must be working in different areas. For any kind of general Python/C++ work, its a _massive_ pain. The integrated debugger rarely works, and its a 30 minute recompile to figure that out. The documentation for actually being efficient in build/run/test is basically "ask the old guy in the corner". You'd best hope they know and are willing to share. The code search is great! The downside is that nobody bothers to document stuff, so thats all you've got. (comments/docstrings are for weaklings apparently) You want to use a common third party library? You'd best hope its already ingested, otherwise you're going to be spending the next few days trying to get that into the codebase. (yes there are auto tools, no they don't always work.) Also, you're now on the hook to do security upgrades.
- deleted 2y ago[deleted]
- fauria 2y agoCan this be downloaded somewhere? Couldn't find any link in the open source site: https://opensource.fb.com/ https://opensource.fb.com/ nor the ELI5: https://developers.facebook.com/blog/post/2021/09/20/eli5-bento-interactive-notebook-empowers-development-collaboration-best-practices https://developers.facebook.com/blog/post/2021/09/20/eli5-be...
- michaelmior 2y agoI don't believe Bento has been open-sourced.
- make3 2y agointeresting that they make external articles about it
- rovr138 2y ago"Oh that's cool.", "It'd be interesting to work on problems like that.", "That's a neat solution" If anyone's on the fence about applying, that could be enough to nudge them in the direction. If anyone's worked in similar areas, could be worth applying and looking at the team, etc.
- michaelmior 2y agoTotally agree, although odd that the post was tagged as "open source."
- tqi 2y agoI think thats because it's based on an open source project
- tqi 2y agoTBH the value of bento over other notebook offerings was almost entirely how well it plays with the rest of the data and infra stack within facebook. It was super easy to go from raw data (entire DE and DI orgs responsible for ETL and cluster maintenance) to a cleaned up table (usually built by DEs) to an ad hoc table to support a specific use-case that could then be accessed via bento, analyzed, and then published / shared to anyone in the company.
- sk11001 2y agoI kind of love Meta for all the seemingly unnecessary internal stuff they do. They have so many projects that are absolutely not critical for them, maybe not even net positive, but they spend who knows how many hours building and maintaining them.
- apwell23 2y ago> Meta for all the seemingly unnecessary internal stuff they do. Netflix would like to have a word.
- Narhem 2y agoNetflix’s situation is caused by their business model.
- fwip 2y agoIs it? It seems like 90% of what Netflix is (from a technical PoV), is a CDN + video playback. There's a lot more value in the content library they've negotiated and the business agreements with ISPs than there is in the software stack. Apologies if this response is delayed, 6 posts today is "too fast."
- rNULLED 2y agoNetflix now builds many of the video production tools they need to produce their own content. This now includes games as well.
- apwell23 2y agosure but i was alluding to stuff like this https://netflixtechblog.com/maestro-netflixs-workflow-orchestrator-ee13a06f9c78 https://netflixtechblog.com/maestro-netflixs-workflow-orches...
- scottyah 2y agoI'm not sure anyone has access to the real data, but I've had a suspicion that Netflix is able to remain a lot more profitable due to their superior tech. Cloud hosting and streaming (not to mention labor) can get very expensive, and I think while it's easier to set up nowadays (in comparison to when they started) a lot of the other companies are burning cash to try to keep up. HBO Max (just Max now?) has always had poor streaming quality compared to netflix and I imagine they're paying a lot more for it.
- big-chungus4 2y agocan I, a mere mortal, use it?
- kyrrewk 2y agothis is cool! wish there was a commerical product that did this. marimo does something similar, but you have to do the deployment yourself
- mscolnick 2y agomarimo has a playground to run notebooks via WebAssembly - similar to Bento - without having to deploy yourself: https://marimo.app/ https://marimo.app/
- deleted 2y ago[deleted]
- Fraterkes 2y agoA bit off-topic, but my problem with any notebook type of tool (ie you create a document that mixes code, the output of that code, and text/media) is that they always feel like they're meant to be these quick, off the cuff ways to present data. But when I try to use them they just feel awkward and slow. (I tried doing a jupyter notebook with the vscode plugin, and while everything was very polished, it feld like I was ponderously coding in Word or something. The same was true for R-notebooks in rstudio. Maybe it's a better experience if you have a decently fast laptop)
- Fraterkes 2y agoAlso I always think it's a littly sad that Jupyter was one of the best shots for Julia to get more mainstream attention, and instead the notebooks people write are basically exclusively python
- paddy_m 2y agoAlso the Julia people wrote their own notebook system called Pluto. Which is so on brand for them. It might be technically better, but they miss out on the whole jupyter ecosystem, further isolating the language.
- lamename 2y agoIME notebooks in VS Code are even worse (but improving). Jupyter lab is faster...but that depends on how fast you prefer ;)
- wenc 2y agoI have the exact opposite experience — VS Code notebooks are much snappier and are possibly the best Jupyter implementations I’ve ever used (better and more responsive than vanilla Jupyter or Jupyter labs). VS code notebooks also support LSPs with refactoring, typing etc. Black is supported. Step by step debugging is supported. Venv is built in. There are so many conveniennces in VS Code that whenever I have to use Jupyter Lab I feel a lot of stuff is missing.
- talles 2y agoTanya Rai - Introducing Bento: Jupyter Notebooks @ Facebook | JupyterCon 2020 : https://www.youtube.com/watch?v=f3UfVX4_PD4 https://www.youtube.com/watch?v=f3UfVX4_PD4
- talles 2y agoI wish more people used marimo, so much better than jupyter
- tantalor 2y agoGlad to see people using the term "serverless" to mean "actually without a server" instead of what other places are doing.
- mhh__ 2y agoI've been using Marimo along these lines recently. I'm fan. So so glad to not use Jupyter.
- deleted 2y ago[deleted]
- bsimpson 2y agoI love that notebooks started as a student hacking together a Python fork and now they're core infrastructure for all these places trying to make sense of GenAI.
- quantadev 2y agoThe original "Block Editor" (that Jupyter modeled itself after) is the one that's now called "Quanta", and has been around for decades in various forms and incantations: https://github.com/Clay-Ferguson/quantizr https://github.com/Clay-Ferguson/quantizr I'm thinking that Jupyter might still not be "Tree Based" but that would be a heck of a leap in capability if they "fix" that.
- chthonicdaemon 2y agoI always thought Jupyter was based on other notebook-style interfaces like Mathematica or Maple.
- mkl 2y agoYes, but it's closer to Sage (browser and Python based). I don't know what quantadev is thinking of, but Quanta seems totally different and not a programming notebook at all. Its README also claims "Quanta is a new kind of platform with a new kind of architecture", while quantadev claims it "has been around for decades".
- quantadev 2y agoI meant the "block editor" aspect, like how individual chunks of text and images can be independently selected and moved around or even shared with their own URL. I've long believed some system like that could and should some day replace even HTML and the web, and that it'll only happen if the Semantic Web ever takes off in a big way where chunks of stuff are "typed" (like a Type-Safe Web). Even Tim Berners Lee has been dreaming of this for decades, but the world is still stuck in HTML-land for the foreseeable future.
- ryannz 2y ago[dead]
- woadwarrior01 2y agoHaving used both Bento and later Colaboratory, a few years ago, I think I liked the latter a lot more. Google's internal tools are usually much more polished and better-designed, perhaps because they've been around for much longer.
- deleted 2y ago[deleted]