11 ms·
Lightpanda migrate DOM implementation to Zig
- barishnamazov 9mo agoThis reminds me of the Servo project's journey. Always impressed to see another implementation of the WHATWG specs. It's interesting to see Zig being chosen here over Rust for a browser engine component. Rust has kind of become the default answer for "safe browser components" (e.g., Servo, Firefox's oxidation), primarily because the borrow checker maps so well to the ownership model of a DOM tree in theory. But in practice, DOM nodes often need shared mutable state (parent pointers, child pointers, event listeners), which forces you into Rc<RefCell<T>> hell in Rust. Zig's manual memory management might actually be more ergonomic for a DOM implementation specifically because you can model the graph relationships more directly without fighting the compiler, provided you have a robust strategy for the arena allocation. Excited to learn from Lightpanda's implementation when it's out.
- fbouvier 9mo agoHi, I am Francis, founder of Lightpanda. We wrote a full article explaining why we choose Zig over Rust or C++, if you are interested: https://lightpanda.io/blog/posts/why-we-built-lightpanda-in-zig https://lightpanda.io/blog/posts/why-we-built-lightpanda-in-... Our goal is to build a headless browser, rather than a general purpose browser like Servo or Chrome. It's already available if you would like to try it: https://lightpanda.io/docs/open-source/installation https://lightpanda.io/docs/open-source/installation
- barishnamazov 9mo agoThanks Francis, appreciate the nice & honest write-up with the thought process (while keeping it brief).
- parhamn 9mo agoOff topic note: I read the website and a few pages of the docs and it's unclear to me for what I can use LightPanda safely. Like say I wanted to swap my it as my engine on playwright, what are the tradeoffs? What things are implemented, what isnt?
- h33t-l4x0r 9mo agoI think it's really more of an alternative to JSDom than it is an alternative to Chromium. It's not going to fool any websites that care about bots into thinking it's a real browser in other words.
- fbouvier 9mo agoThanks for the feedback, we will try to make this clearer on the website. Lightpanda works with Playwright, and we have some docs[1] and examples[2] available. Web APIs and CDP specifications are huge, so this is still a work in progess. Many websites and scripts already work, while others do not, it really depends on the case. For example, on the CDP side, we are currently working on adding an Accessibility tree implentation. [1] https://lightpanda.io/docs/quickstart/build-your-first-extraction-script https://lightpanda.io/docs/quickstart/build-your-first-extra... [2] https://github.com/lightpanda-io/demo/tree/main/playwright https://github.com/lightpanda-io/demo/tree/main/playwright
- epolanski 9mo agoI was actually interested into using lightpanda for E2Es to be honest, because halving the feedback cycle would be very valuable to me.
- infogulch 9mo agoMaybe you should recommend a recipe for configuring playwright with both chromium and lightpanda backends so a given project can compare and evaluate whether lightpanda could work given their existing test cases.
- nicoburns 9mo agoI see you're using html5ever for HTML parsing, and like it's trait/callback based API (me too). It looks like style/layout is not in scope at the moment, but if you're ever looking at adding style/layout capabilities to lightpanda, then you may find it useful to know that Stylo [0] (CSS / style system) and Taffy [1] (box-level layout) are both avaiable with a similar style of API (also Parley [2] which has a slightly different API style but can be combined with Taffy to implement inline/text layout). [0]: https://github.com/servo/stylo https://github.com/servo/stylo [1]: https://github.com/DioxusLabs/taffy https://github.com/DioxusLabs/taffy [2]: https://github.com/linebender/parley https://github.com/linebender/parley --- Also, if you're interested in contributing C bindings for html5ever upstream then let me know / maybe open a github issue.
- quotemstr 9mo agoChoosing something like Zig over C++ on simplicity grounds is going to be a false economy. C++ features exist for a reason. The complexity is in the domain. You can't make a project simpler by using a simplistic language: the complexity asserts itself somehow, somewhere, and if a language can't express the concept you want, you'll end up with circumlocution "patterns" instead. Build system complexity disappears when you set it up too. Meson and such can be as terse as your Curl example. I mean, it's your project, so whatever. Do what you want. But choosing Zig for the stated reasons is like choosing a car for the shape of the cupholders.
- hnlmorg 9mo agoThat’s not fully true though. There’s different types of complexity: - project requirements - requirements forced upon you due to how the business is structured - libraries available for a particular language ecosystem - paradigms / abstractions that a language is optimised for - team experiences Your argument is more akin to saying “all general purpose languages are equal” which I’m sure you’d agree is false. And likewise, complexity can and will manifest itself differently depending on language, problems being solved, and developer preferences for different styles of software development. So yes, C++ complexity exists for a reason (though I’d personally argue that “reason” was due to “design by committee”). But that doesn’t mean that reason is directly applicable to the problems the LightPanda team are concerned about solving.
- vegabook 9mo agoC++ features for complexity management are not ergonomic though, with multiple conflicting ideas from different eras competing with each other. Sometimes demolition and rebuild from foundations is paradoxically simpler.
- Philpax 9mo agoYour Swiss Army Knife with a myriad of 97 oddly-shaped tools may be able to do any job anyone could ask of it, but my Swiss Army Knife of 10 well-designed tools that are optimal for my set of tasks will get my job done with much less frustration.
- meheleventyone 9mo ago
- Jweb_Guru 9mo agoRespectfully, for browser-based work, simplicity is absolutely not a good enough reason to use a memory-unsafe language. Your claim that Zig is in some way safer than Rust for something like this is flat out untrue.
- hello_moto 9mo agoIn that blog post, the author said safer than C not Rust.
- dnautics 9mo agoWhat is your attack model here? Each request lives in its own arena allocator, so there is no way for any potentially malicious JavaScript to escape and read memory owned by any other request, even if there is a miscode. otherwise, VM safety is delegated to the V8 core.
- Jweb_Guru 9mo agoBelieve it or not, using arenas does not provide free memory safety. You need to statically bound allocations to make sure they don't escape the arena (which is exactly how arenas work in Rust, but not Zig). There are also quite a lot of ways of generating memory unsafe code that aren't just use after free or array-out-of-bounds in a language like Zig, especially in the context of stuff like DOM nodes where one frequently needs to swap out pointers between elements of one type and a different type.
- nwienert 9mo agoWould be helpful to compare Lightpanda to Webkit, Playwright has a driver for example and its far faster and less resource hungry than Chrome. When I read your site copy it struck me as either naive to that, or a somewhat misleading comparison, my feedback would be just to address it directly alongside Chrome.
- aatd86 9mo agoWould be great if it could be used as a wasm library... Just saying... Is it? I would actually need and use this.
- pjmlp 9mo agoAnd use-after-free, when that arena's memory goes away.
- pron 9mo agoBut arenas have substantial benefits. They may be one of the few remaining reasons to use a low-level (or "systems programming") language in the first place. Most things are tradeoffs, and the question isn't what you're giving up, but whether you're getting the most for what you're paying.
- pjmlp 9mo agoArenas are also available in languages with automatic memory management, e.g. D, C# and Swift, to use only modern languages as example. Thus I don't consider that a reason good enough for using Zig, while throwing away the safety from modern languages.
- pron 9mo agoFirst, Zig is more modern than any of the languages you mention. Second, I'm not aware that any of those languages offer arenas similar in their power and utility to Zig's while offering UAF-freedom at the same time. Note that "type-safe" arenas are neither as powerful as general purpose arenas nor fully offer UAF-freedom. I could be wrong (and if I am, I'd really love to see an arena that's both general and safe), but I believe that in all these languages you must compromise on either safety or the power of the arena (or both).
- metaltyphoon 9mo ago> First, Zig is more modern than any of the languages you mention How so? This feels like an empty statement at best.
- pron 9mo ago"modern: relating to the present or recent times as opposed to the remote past". I agree it's not a useful concept here but I didn't bring it up. Specifically, I don't think there's any consideration that had gone into the design of D, C#, or Rust that escaped Zig's designer. He just consciously made different choices based on the data available and his own judgment.
- IshKebab 9mo agoI don't think it's really that bad in Rust. If you're happy with an arena in Zig you can do exactly the same thing in Rust. There are a ton of options listed here: https://donsz.nl/blog/arenas/ https://donsz.nl/blog/arenas/ Some of them even prevent use after free (the "ABA mitigation" column).
- mijoharas 9mo agoI'm not super experienced with zig, but I always think that in the same way that rust forces you to think about ownership (by having the borrow checker - note: I think of this as a good thing personally) zig makes you think upfront about your allocation (by making everything that can allocate take an allocator argument.). It makes everything very explicit, and you can always _see_ where your allocations are happening in a way that you can't (as easily, or as obviously - imo) in rust. It seems like something I quite like. I'm looking forward to rust getting an effects system/allocator api to help a little more with that side of things.
- silon42 9mo agoThe problem is deallocation... unless you tie the allocated object to an arena allocator with a lifetime somehow (Rust can model that).
- mijoharas 9mo agoYep, rust forces you to think about lifetimes. Zig only suggests it (because you're forced to think about allocation, which makes you naturally think about the lifetime usually) but does not help you with it/ensure correctness. It's still nice sometimes to ensure that you have to think about allocation everywhere, and can change the allocation strategy for something that works for your usecase. (hence why I'm looking forward to the allocator api in rust to get the best of both worlds).
- IshKebab 9mo agoThat's true and I liked the idea of it until I started writing some Zig where I needed to work with strings. Very painful. I'm sure you typically get a bit faster string manipulation code than what you'd get with Rust but I don't think it's worth the cost (Rust is pretty fast already).
- galangalalgol 9mo agoToo late now, but is the requirement for shared mutable state inherent in the problem space? Or is it just because we still thought OOP was cool when we started on the DOM design?
- pornel 9mo agoYes. It is required for W3C's DOM APIs, which give access to parent nodes and allow all kinds of mutations whenever you want. Event handlers + closures also create potentially complex situations you can't control, and you'll need a cycle-breaking GC to avoid leaking like IE6 did. You can make a more restricted tree if you design your own APIs with immutability/ownership/locking, but that won't work for existing JS codebases.
- 7bit 9mo ago> without fighting the compiler It's unfortunate that "writing safe code" is constantly being phrased in this way. The borrow checker is a deterministic safety net. Claiming Zig is easier ignores that its lack of safety checks is what makes it feel easier; if Zig had Rust’s guarantees, the complexity would be the same. Comparing them like this is apples vs. oranges.
- senko 9mo agoThe fact that Zig doesn't have Rust's guarantees doesn't mean Zig does not have safety checks. The safety checks that Zig does have are different, and are different in a way that's uniquely useful for this particular project. Zig's check absolutely don't go to the extent that Rust's do, which is kind of the point here. If you do need to go beyond safe code in Rust, Zig is safer than unsafe code in Rust. Saying Zig lacks safety checks is unfortunate, although I wouldn't presume you meant it literally and just wanted to highlight the difference.
- pjmlp 9mo agoThing is, those safety checks are also available in C and C++, provided that one uses the right tools like PVS and PurifyPlus (just to quote two examples), and now ongoing AI based tooling efforts for verification, thus the question is why a language like Zig in the 21st century, other than "I don't like either C++ or Rust".
- 7bit 9mo agoI never said Zig has no safety features. What I said is true, though. If it would have Rusts guarantees (as in: The same) it would be more complex.
- senko 9mo agoI mean if we're going to nitpick: >>> its lack of safety checks >> Saying Zig lacks safety checks is unfortunate, > I never said Zig has no safety features. You did. Or, alternatively, if you don't equate "checks" with "features", then I never said you said that so what are you complaining about? > If it would have Rusts guarantees (as in: The same) it would be more complex. Which is true (if tautological), and is basically what the GP said: > Zig's manual memory management might actually be more ergonomic for a DOM implementation specifically because you can model the graph relationships more directly without fighting the compiler, provided you have a robust strategy for the arena allocation Both you and the GP agree that Rust is more complex. You objected to this with: > It's unfortunate that "writing safe code" is constantly being phrased in this way. Upon which I commented that Zig does have safety features, even if they're not covering you as well as Rust's ones. Which is, again, inline with "provided you have a robust strategy for the arena allocation." Now, if you think I'm going overboard with this, I agree with you -- and this is the exact feeling I have when I look at Rust :)
- pron 9mo agoI don't think that a language that was meant to compete with C++ and in 10+ years hasn't captured 10% of C++'s (already diminished) market share could be said to have become "kind of the default" for anything (and certainly not when that requires generalising from n≅1).
- anal_reactor 9mo agoThe problem is that the number of browser engines is n=2.
- drnick1 9mo agoInterestingly, Ladybird, which aims at being the n = 3, is also written in C++.
- drannex 9mo agoLadybird is in the process of switching over to Swift, and has been for a little over a year now. Not linking to the pedophilic nazi-site, and as Nitter is dead-ish, here is the full-text announcement archived on tildes: https://tildes.net/~comp/1j7m/ladybird_chooses_swift_as_its_successor_language_to_c https://tildes.net/~comp/1j7m/ladybird_chooses_swift_as_its_...
- pjmlp 9mo agoIt has for Amazon, Adobe, Microsoft, Google and the Linux kernel. It remains to be seen which big name will make Zig unavoidable.
- nicoburns 9mo agoThis table is informative as to exactly what lightpanda is: https://lightpanda.io/blog/posts/what-is-a-true-headless-browser#what-does-true-headless-mean https://lightpanda.io/blog/posts/what-is-a-true-headless-bro... TL;DR: It does the following: - Fetch HTML over the network - Parse HTML into a DOM tree - Fetch and execute JavaScript that manipulates the DOM But not the following: - Fetch and parse CSS to apply styling rules - Calculate layout - Fetch images and fonts for display - Paint pixels to render the visual result - Composite layers for smooth scrolling and animations So it's effectively a net+DOM+script-only browser with no style/layout/paint. --- Definitely fun for me to watch as someone who is making a lightweight browser engine with a different set of trade-offs (net+DOM+style/layout/paint-only with no script)
- karel-3d 9mo agoWhen I was working before on something that used headless browser agents, the ability to do a screenshot (or even a recording) was really great for debugging... so I am not sure about the "no paint". But hey everything in life is a trade-off.
- pzo 9mo agoyeah I feel the same, I think even having a screenshot of part of rendered page or full page can be useful even for machines considering how heavy those HTML can be to parse and expensive for LLM context. Sometimes (sub)screenshot is just a better kind of compression
- fbouvier 9mo agoYes HTML is too heavy and too expensive for LLM. We are working on a text-based format more suitable for AI.
- httpteapot 9mo agoWhat do you think of the DeepSeek OCR approach where they say that vision tokens might better compress a document than its pure text representation? https://news.ycombinator.com/item?id=45640594 https://news.ycombinator.com/item?id=45640594 I've spent some time feeding llm with scrapped web pages and I've found that retaining some style information (text size, visibility, decoration image content) is non trivial.
- everlier 9mo agoWow. Lightpanda is absolutely bonkers of a project. I'd pay dearly for such an option a few years back.
- kristopolous 9mo agoI've been using it for months now ever since I saw their presentation at GitHub This is a common flow for me lightpanda url | markitdown (microsoft) | sd (day50 streamdown) I even have it as a shell alias, wv(). It's way better than the crusty old lynx and links on sites that need JS. It's solid. Definitely worth a check
- Philpax 9mo agoOh, huh, being able to convert arbitrary websites that may use JS for rendering to Markdown could be very handy indeed. Thanks for the tip!
- daddykotex 9mo agoThanks for the tip, that's very cool. I did not know about `markitdown` and `streamdown`.
- lewdwig 9mo agoA language which is not 1.0, and has repeatedly changed its IO implementation in a non-backwards-compatible way is certainly a courageous choice for production code.
- steeve 9mo agothe upside is absolutely worth it
- Philpax 9mo agoIt's certainly not a choice I would have made, but there's sufficient precedent for it now (TigerBeetle, Ghostty, etc) that I can understand it.
- hu3 9mo agoalso Bun
- blue_pants 9mo agoalso Roc
- Iridescent_ 9mo agoThis one is far from prod-ready however
- ivanjermakov 9mo agoIn my experience, migrating small-scale projects takes from minutes to single digit hours. Standard library is changing. The core language semantics - not so much. You can update from std.ArrayListUnmanaged to std.array_list.Aligned with to greps.
- bluecalm 9mo agoRight? People must really like the design choices in Zig to do that instead of choosing another language. It's very interesting just because of that.
- portly 9mo agoLove to see Zig winning!
- steeve 9mo agoThis looks incredible, congratulations!
- fbouvier 9mo agoThanks Steeve!
- tonyhart7 9mo agofinally, rewrite in zig movement is coming
- deepriverfish 9mo agoit's so tiring that every time there's a post about something being implemented in Zig or C or C++, the Rust brigade shows up trying to pick up a fight.
- hobofan 9mo agoMaybe just a reflex by people that had to hear a decade of "why not C++" whenever it was mentioned that Rust is being used?
- esafak 9mo agoI don't know, man. At this point I'm liable to ask "Why are you using C++?" if you start a new project. Let them defend their language!
- pjmlp 9mo agoAs part of the "all software should be liable brigade", it is a matter of misplaced goals after the cybersecurity agencies started looking into the matter.
- Klonoar 9mo agoIt’s a site where programming nerds congregate to waste time arguing with each other. Where do you think you are? This same pattern used to play out with Ruby, Lisp, and other languages in different eras of this site. It will probably never stop and calling it out seems to just fan the flames more than anything else.
- neoden 9mo agoI hate to say it, but time is quickly running out for Zig(( AI might never pick it up properly and without that it will never go out of its niche
- shepherdjerred 9mo agoAre you implying that programming languages are now going to be “frozen” because of AI? I can understand the source of concern but I wouldn’t expect innovation to stop. The world isn’t going to pause because of a knowledge cutoff date.
- neoden 9mo agoInnovation doesn't go for the sake of innovation itself. Innovation should serve a purpose. And the purpose of having programming languages is to overcome the limitations of human mind, of our attention span, of our ability to manipulate concepts expressed in abstractions and syntax. We don't know how long we'll need this. I really like Zig, I wish it appeared several years earlier. But rewriting everything in Zig might just not have practical sense soon.
- shepherdjerred 9mo agoI agree that programming languages will no longer need to be as accessible to humans. However there is still a strong argument to be made for protections/safety that languages can provide. e.g. would you expect a model (assuming it had the same expertise in each language) to make more mistakes in ASM, C, Zig, or Rust? I imagine most would agree that ASM/C would be likely to have the most mistakes simply because fewer constraints are enforced as you go closer to the metal. So, while we might not care about how easy it is for a human to read/write, there will still be a purpose for innovation in programming languages. But those innovations, IMO, will be more focused on how to make languages easier for AI.
- neoden 9mo ago> would you expect a model (assuming it had the same expertise in each language) to make more mistakes in ASM, C, Zig, or Rust? "assuming it had the same expertise in each language" is the most important part here, because the expertise of AI with these languages is very different. And, honestly, I bet on C here because its code base is the largest, the language itself is the easiest to reason about and we have a lot of excellent tooling that helps mitigate where it falls short. > I imagine most would agree that ASM/C would be likely to have the most mistakes simply because fewer constraints are enforced as you go closer to the metal. We need these constraints because we can't reliably track all the necessary details. But AI might be much more capable (read — scalable) in that, so all the complexity that we need to accumulate in a programming language it might just know out of the way it's built.
- MrBuddyCasino 9mo agoBecause We're Not Smart Enough for C++ or Rust Very refreshing. Most engineers would rather saw their leg off.
- Copenjin 9mo agoAny older project similar to this? Headless browser with js support I mean, I want to check various implementations of this idea.
- pulkas 9mo agozigdom all-diy