4 ms·

Wow. Good luck there. For the last year or so I've had the distinct pleasure of patching and building WebKit for a project. Some observations about the WebKit c

by sigil 13y ago

Wow. Good luck there. For the last year or so I've had the distinct pleasure of patching and building WebKit for a project. Some observations about the WebKit codebase... First off, man is it huge. There's a link stage that uses ~4GB of memory. The command line for that link is so long I needed a patched version of make [1]. Why the WebKit devs haven't knuckled under the pain and broken out multiple smaller libraries is beyond me. They must be tough. Second off, if you dive into the code -- say you're looking to change something about the layout algorithm -- you quickly discover that there is no layout algorithm. Or at least, no one place that articulates the algorithm, which is instead spread like pixie dust over dozens of objects. Each one contributes a little bit of the logic. (You're in a twisty maze of classes, all alike...) Perhaps some of this is a reasonable separation of concerns that helps the whole thing scale. But it made me step back and wonder if we're approaching these kinds of problems right, architecturally. Does WebKit really need so much code? If we could go back and write a browser in a functional language, what would that look like? Would performance necessarily doom it from the start? [1] http://blogs.gnome.org/diegoe/2012/09/23/webkitgtk-failing-to-build-argument-list-too-long/ http://blogs.gnome.org/diegoe/2012/09/23/webkitgtk-failing-t...

taeric 13y ago
This actually surprises me some. With how widespread the use of webkit is, I would have thought it was among the more cleanly designed pieces of software out there. Anyone have any good pointers on an analysis of the code base? Any easy way to just "dive in" to the code other than the obvious checkout and look around?
- azakai 13y ago
  > With how widespread the use of webkit is, I would have thought it was among the more cleanly designed pieces of software out there. Sadly being widespread is no guarantee of clean design ;) WebKit, like other browser engines, was originally designed in the 90's, and evolved over a long period of time. Again like the others, it's a large and complex C++ codebase, with all the good and bad that comes with that. WebKit does have some advantages over other browser engines though, specifically it is the most embeddable, and I would say that is the main reason for its being widespread today.
  - taeric 13y ago
    This further hits my curiosity on whether or not there are any examples of "cleanly designed" systems that have "won" in the long run. Am I just growing cynical? I still harbour feelings that something in the discipline of webkit offers lessons. I just don't know if I can see it myself. Outside of the lessons of brute force being a good tool. (Where I'm assuming there is a lot of force behind webkit.) (I do understand that they have a lot of tests that are fairly complex. Mayhap that is the true lesson. Get a good handle on how to run complicated test cases.)
    - reubenmorais 13y ago
      I would point out Qt as a very well and cleanly designed project that "won" as in it's one of the if not the top GUI frameworks out there. http://qt-project.org/ http://qt-project.org/
      - nostrademons 13y ago
        Is it actually all that widespread if you consider the full set of all apps on all platforms? The impression I get is that the majority of apps don't use any cross-platform GUI framework and instead use the native API of the platform - Win32 for pre-XP Windows, .NET for Vista+ Windows, Cocoa for Macintosh, GTK or KDE for Linux, HTML/CSS/JS for the Web, native Android for Android, native iOS for iOS, etc.
        Elv13 13y ago
        KDE is the community, not the toolkit. Most KDE specific widgets have been merged/replaced in Qt over time. KF5 mostly is a bunch of small libraries using Qt. http://dot.kde.org/sites/dot.kde.org/files/kf5_big_0.png http://dot.kde.org/sites/dot.kde.org/files/kf5_big_0.png Qt is also used by a few "big" software and as the default toolkit for BB10, Jolla and Mer (totaling a nice 1% market share ;) ). It is also seen as the successor to Motif for large scientific applications.
    - nostrademons 13y ago
- tflinton 13y ago
  There's two next steps: A good build system/toolchain with GYP that generates Xcode, VS C++, and ninja project files. That would be mega awesome. QtWebkit already has some gyp files, probably could repurpose those. Also, you can read up on embind on emscripten's website, you can use c++filterjs and the derived object WebCore.exp on the build to create javascript API's from the C++ export symbol table. I just need to actually have a few days free to finish that and hopefully can have SOME functional demos from the API (that and hopefully I can throw away 90% of the symbols being exposed in the current version and cut the code size down :/)
tflinton 13y ago
The link stage is pretty huge. It currently won't compile with emscripten with debug symbols, in-fact the entire obj-pre-link stage is 8.9GB of "bytecode". There really isn't a layout algorithm but it does seem like overtime they began to contain it as to make it not platform dependent (thank god). I think that the major achievement of webkit wasn't in the code they took from KHTML ;) but in the unit tests that were created and the robust amounts of work that went in to focus on making things pixel perfect. Prior to webkit a margin meant whatever the hell the damn implementers felt like. If it worked at all. I should say much of the code in WebKit is no-mans-land, or essentially greatest common denominator work to make webkit portable to multiple architectures. That's why the linking takes so long and you can go from >1/2 GB of source code, 4GB of linker memory down to a 20MB object file.
- bdash 13y ago
  >1/2GB of source code? WebKit's entire Source directory is 247MB, and more than 100MB of that is ChangeLog files.
sehugg 13y ago
There's Servo from the Mozilla team, written in Rust -- not a pure functional language but an interesting step. (Not that there are strong guarantees it'll be less complex overall, though it may have other properties like memory safety)
erikj54 13y ago
I'm a bit late to the party here, but figured I'd chime in with my own experiences. I spent a many days staring into WebKit and can honestly say that finding your way around does get easier. When you are searching for things, it helps to have a good base grasp of what IS, and what is NOT WebKit. Webkit is many, many things, but people make some common mistakes about what it is. Unfortunately, to get anything done with WebKit, it requires a huge amount of work, and platform specific implementations and support, which add the the general confusion.
tolmasky 13y ago
These concerns resonate a bit with Google's explanation for Blink. Really excited to see what comes out of that.

by sigil 13y ago

taeric 13y ago

This actually surprises me some. With how widespread the use of webkit is, I would have thought it was among the more cleanly designed pieces of software out there. Anyone have any good pointers on an analysis of the code base? Any easy way to just "dive in" to the code other than the obvious checkout and look around?

azakai 13y ago
> With how widespread the use of webkit is, I would have thought it was among the more cleanly designed pieces of software out there. Sadly being widespread is no guarantee of clean design ;) WebKit, like other browser engines, was originally designed in the 90's, and evolved over a long period of time. Again like the others, it's a large and complex C++ codebase, with all the good and bad that comes with that. WebKit does have some advantages over other browser engines though, specifically it is the most embeddable, and I would say that is the main reason for its being widespread today.
- taeric 13y ago
  This further hits my curiosity on whether or not there are any examples of "cleanly designed" systems that have "won" in the long run. Am I just growing cynical? I still harbour feelings that something in the discipline of webkit offers lessons. I just don't know if I can see it myself. Outside of the lessons of brute force being a good tool. (Where I'm assuming there is a lot of force behind webkit.) (I do understand that they have a lot of tests that are fairly complex. Mayhap that is the true lesson. Get a good handle on how to run complicated test cases.)
  - reubenmorais 13y ago
    I would point out Qt as a very well and cleanly designed project that "won" as in it's one of the if not the top GUI frameworks out there. http://qt-project.org/ http://qt-project.org/
    - nostrademons 13y ago
      Is it actually all that widespread if you consider the full set of all apps on all platforms? The impression I get is that the majority of apps don't use any cross-platform GUI framework and instead use the native API of the platform - Win32 for pre-XP Windows, .NET for Vista+ Windows, Cocoa for Macintosh, GTK or KDE for Linux, HTML/CSS/JS for the Web, native Android for Android, native iOS for iOS, etc.
      - Elv13 13y ago
        KDE is the community, not the toolkit. Most KDE specific widgets have been merged/replaced in Qt over time. KF5 mostly is a bunch of small libraries using Qt. http://dot.kde.org/sites/dot.kde.org/files/kf5_big_0.png http://dot.kde.org/sites/dot.kde.org/files/kf5_big_0.png Qt is also used by a few "big" software and as the default toolkit for BB10, Jolla and Mer (totaling a nice 1% market share ;) ). It is also seen as the successor to Motif for large scientific applications.
  - nostrademons 13y ago
tflinton 13y ago
There's two next steps: A good build system/toolchain with GYP that generates Xcode, VS C++, and ninja project files. That would be mega awesome. QtWebkit already has some gyp files, probably could repurpose those. Also, you can read up on embind on emscripten's website, you can use c++filterjs and the derived object WebCore.exp on the build to create javascript API's from the C++ export symbol table. I just need to actually have a few days free to finish that and hopefully can have SOME functional demos from the API (that and hopefully I can throw away 90% of the symbols being exposed in the current version and cut the code size down :/)

tflinton 13y ago

The link stage is pretty huge. It currently won't compile with emscripten with debug symbols, in-fact the entire obj-pre-link stage is 8.9GB of "bytecode". There really isn't a layout algorithm but it does seem like overtime they began to contain it as to make it not platform dependent (thank god). I think that the major achievement of webkit wasn't in the code they took from KHTML ;) but in the unit tests that were created and the robust amounts of work that went in to focus on making things pixel perfect. Prior to webkit a margin meant whatever the hell the damn implementers felt like. If it worked at all. I should say much of the code in WebKit is no-mans-land, or essentially greatest common denominator work to make webkit portable to multiple architectures. That's why the linking takes so long and you can go from >1/2 GB of source code, 4GB of linker memory down to a 20MB object file.

bdash 13y ago
>1/2GB of source code? WebKit's entire Source directory is 247MB, and more than 100MB of that is ChangeLog files.

sehugg 13y ago

There's Servo from the Mozilla team, written in Rust -- not a pure functional language but an interesting step. (Not that there are strong guarantees it'll be less complex overall, though it may have other properties like memory safety)

erikj54 13y ago

I'm a bit late to the party here, but figured I'd chime in with my own experiences. I spent a many days staring into WebKit and can honestly say that finding your way around does get easier. When you are searching for things, it helps to have a good base grasp of what IS, and what is NOT WebKit. Webkit is many, many things, but people make some common mistakes about what it is. Unfortunately, to get anything done with WebKit, it requires a huge amount of work, and platform specific implementations and support, which add the the general confusion.

tolmasky 13y ago

These concerns resonate a bit with Google's explanation for Blink. Really excited to see what comes out of that.