5 ms·

Edbrowse – A Command Line Editor Browser

57 points by gjvc 4y ago

ttgurney 4y ago
Funny seeing this here as I've been thinking a lot about text-based browsers lately. Just a couple days ago I tried to build this one from source, but I put it aside due to the dependencies on PCRE and a JavaScript engine. (I am running a hand-rolled Linux "distro" so I can't just install ready-made binary packages.) I do really appreciate that this one uses libcurl on the backend. Surprisingly few browsers do this--Lynx, Links, and w3m all have their own networking code. They have bespoke HTML parsing and rendering as well. I'm lately thinking I want to see a text-mode browser that just glues together libcurl, curses, simple HTML rendering, and maybe an existing HTML parsing library. No text-based HTML rendering library exists that I'm aware of. Also these classic text browsers have their own implementations of FTP, NNTP, and some other legacy cruft. I'm thinking most of this could easily be provided by libcurl (if at all).
- augusto-moura 4y ago
  It depends on quickjs for the JavaScript implementation, which should be fairly simpler to compile on a hand rolled Linux. I'm not so sure about PCRE though
  - ttgurney 4y ago
    Oh I'm sure the actual work to compile those packages is not much. It's more to do with keeping the number of packages on my system to a minimum. Actually I would not be surprised if the JavaScript engine can be omitted with just a little bit of patching work... assuming there's not actually a build configuration that leaves it out. I've found that with some software projects and their dependencies, "required" does not always mean required.
- shiomiru 4y ago
  > I'm lately thinking I want to see a text-mode browser that just glues together libcurl, curses, simple HTML rendering, and maybe an existing HTML parsing library. I had a similar idea a while ago, except mine was to glue together components from the nim stdlib. So I wrote something like that, then I thought "hey, why not implement some CSS too?" and that sent me down the rabbit hole of writing an actual CSS-based layout engine... I eventually also realized that the stdlib html parser is woefully inadequate for my purposes. In the end, I wrote my own mini browser engine with an HTML5 parser and whatnot. Right now I'm trying to bring it to a presentable state (i.e. integrate libcurl instead of using the curl binary, etc.) so I can publish it. Anyways, if there's a moral to this story it's that writing a browser engine is surprisingly fun, so go for it :)
  - ttgurney 4y ago
    I look forward to seeing your 1st release of this program! > Anyways, if there's a moral to this story it's that writing a browser engine is surprisingly fun, so go for it :) Good to know. I'd been fairly intimidated by the idea.
- smaudet 4y ago
  Call it Unixy or something - unix philosophy of having each program do something separate. Makes more sense, that's what this guy does anyways with the js engine? > Surprisingly few browsers do this--Lynx, Links, and w3m all have their own networking code I think people are suspicious of curl because it is a common utility, and they think it can't possibly have got it right - plus there's something mildly fun about figuring out how to monitor a socket and send/receive IP packets for the first time. I have played around a bit with the Curl code a bit, in part I also suspect other programs do it to get "closer" i.e. being able to manage/dispatch events from a thread directly instead of some signal from a curl thread, probably something about security and thread safety too...
  - shiomiru 4y ago
    The main reason for the aforementioned browsers not using libcurl is mostly historical, as it simply didn't exist back when they were created. (The newest of them is links, first released in 1999 - and according to the curl website, the first libcurl release with a proper interface was in 2000.) w3m even uses its own regex engine for search, because there was no free regex engine with Japanese support the author could've used back then.
- 1vuio0pswjnm7 4y ago
  https://github.com/google/oss-fuzz-vulns/tree/main/vulns/curl https://github.com/google/oss-fuzz-vulns/tree/main/vulns/cur... https://github.com/curl/curl/commit/68ffe6c17d6e44b459d60805813f646d244a186b https://github.com/curl/curl/commit/68ffe6c17d6e44b459d60805... https://www.cvedetails.com/product/25084/Haxx-Curl.html?vendor_id=12682 https://www.cvedetails.com/product/25084/Haxx-Curl.html?vend... Instead of only "thinking a lot about text-based browsers", I have been actively using them on a daily basis for the past 26 years. Links already uses ncurses. I am glad that it does not use libcurl and that it has its own "bespoke" HTML rendering. In over 25 years time, I still have yet to see any other program produce better rendering of HTML tables as text. I have had few if any problems with Links versions over the years. I am quite good at "breaking" software and for me Links has been quite robust. The source code is readable for me and I have been able to change or "fix" things I do not like, then quickly recompile. I can remove features. Recently I fixed a version of the program so that a certain semantic link would not be shown in Wikipedia pages. No "browser extension" required. Links' rendering has managed to keep up with the evolution of HTML and web design sufficiently for me. Despite the enormous variation in HTML acrosse the www, there are very few cases where the rendering is unsatisfactory.^1 I cannot say the same for other attempts at text-only clients. W3C's libwww-based line-mode browser still compiles and works,^2 although I would not be satisifed with its rendering. Nor would I be satisfied with edbrowse, or something simpler such as mynx.^3 I use Links primarily for reading and printing HTML. I use a variety of TCP clients for making HTTP requests, including djb's tcpclient which I am quite sure beats libcurl any day of the week in terms quality, e.g., the programming skill level of the author and the care with which it was written. This non-libcurl networking code is relatively small and does not need oss-fuzz. I do not intentionally use libcurl. It is too large and complex for my tastes. For TLS, I mainly use stunnel and haproxy. 1. One rare example I can recall is https://archive.is https://archive.is 2. https://github.com/w3c/libwww https://github.com/w3c/libwww 3. https://github.com/SirWumpus/ioccc-mynx https://github.com/SirWumpus/ioccc-mynx
  - ttgurney 4y ago
    Hey thanks for your perspective and a couple of mentions of software I'd not heard of (like tcpclient). I agree that curl is pretty big and bloated. I would not call it a deficiency that Links et al. don't depend on it. I mostly just was thinking that since I already have curl on my system, it'd be nice to have a browser that reuses that code. Especially since curl has upstream support for the much smaller BearSSL rather than depending on OpenSSL/LibreSSL.
jhallenworld 4y ago
This is very cool, but dedicated newsreaders (like trn) were easier to use to browse USENET than edbrowse is to browse Hacker News.. If you were forced to surf the web using a teletype, this is your tool. Edit: actually it's growing on me. My one complaint is that I wish it would retain the printing mode when you follow a link. For example, type "g2" to follow the 2nd link on the line. Right now I should be able to hit Enter or z to display lines from this link, but annoyingly you have to enter something like "1" to start it going.
- marttt 4y ago
  I think the main "selling point" of edbrowse for sighted people might be its scripting capabilities. You can automate a lot of browsing with it if you take the time. See the user guide: http://edbrowse.org/usersguide.html#script http://edbrowse.org/usersguide.html#script I've been curious about edbrowse for a long time, finding it a fascinating and inspiring project. It's a somewhat hidden gem for sure. Then again, I failed to build it on my (tiny) distro, and I occasionally think that maybe the internal scripting language, mail client, etc create "too many" temptations for a casual user. A simple line-mode website pager that encourages Unix pipes for any user-defined task (maybe even the javascript engine could be built as an external application?) would be enough for a modest hobbyist coder like me, I guess. Something like the nmh mail handler, but for web browsing: https://www.unix.com/man-page/linux/7/nmh/ https://www.unix.com/man-page/linux/7/nmh/ This is by no means meant as a harsh critique or whining, though. As indicated, the project, Karl Dahlke, Chris Bannon and other devs do deserve tremendous respect. It is quite possible that a heavily pipes-oriented web browsing would be too fragmented for actual, day-to-day usage. The line-editing paradigm nonetheless has its beauty and a compelling simplicity, even in 2022. Keep it going!
siraben 4y ago
For a command line web browser that doesn't compromise on modern web features, I use browsh[0] especially in low-bandwidth situations. In conjunction with tmux and mosh it's just fantastic because you get the full colors and features of the web with minimal bandwidth, since the rendering happens server-side and all the information that is sent to your computer is terminal half-width characters. [0] https://www.brow.sh/ https://www.brow.sh/
- sylware 4y ago
  You missed one important point: edbrowse does not depend on those absurdely and grotesquely massive and complex c++ web engines (blink/geeko, financed by gogol, and webkit, financed by apple). It is lean and simple C, including its deps.
  - deleted 4y ago
    [deleted]
shudza 4y ago
Anyone managed to build this on arch? The AUR package fails to build.
stanleychink 4y ago
- alpaca128 4y ago
  Why is each of your comments a slightly modified copy of another comment in the respective thread?
  - nerdponx 4y ago
    Upvote fishing in order to develop a legitimate-looking account for later sale or other forms of abuse like astroturfing?
  - sylware 4y ago
    yep, I wonder what's happening here. That said, I do compile edbrowse with quickjs and pcre on my hand-rolled glibc/linux distro. I have even removed the build systems from edbrowse, quickjs and pcre to use my own. Quickjs is aiming the one-compilation-unit model (good!). Almost no javascript-ed sites run with it (but I am still testing, for instance on gogol noto font download page), but I like to keep track of its advancement, loosely, but I do it anyway. The bottom of the pb is big tech javascript-ed web engines: blink/geeko (financed by gogol) and webkit (apple). We all know the only reasonably fix for this issue is to regulate HARD towards the availability of a noscript/basic (x)html portal for all pertinent web sites (which are close to 100% anyway).
tpoacher 4y ago
Could this be used to browse/send email from a 2fa domain? I'm desperate to use a cli/tui client but my organisation's 2fa requirement has me stuck on web outlook. I'm trying to get alpine to work, but I'm not sure that it can ...

57 points by gjvc 4y ago

Funny seeing this here as I've been thinking a lot about text-based browsers lately. Just a couple days ago I tried to build this one from source, but I put it aside due to the dependencies on PCRE and a JavaScript engine. (I am running a hand-rolled Linux "distro" so I can't just install ready-made binary packages.) I do really appreciate that this one uses libcurl on the backend. Surprisingly few browsers do this--Lynx, Links, and w3m all have their own networking code. They have bespoke HTML parsing and rendering as well. I'm lately thinking I want to see a text-mode browser that just glues together libcurl, curses, simple HTML rendering, and maybe an existing HTML parsing library. No text-based HTML rendering library exists that I'm aware of. Also these classic text browsers have their own implementations of FTP, NNTP, and some other legacy cruft. I'm thinking most of this could easily be provided by libcurl (if at all).

augusto-moura 4y ago
It depends on quickjs for the JavaScript implementation, which should be fairly simpler to compile on a hand rolled Linux. I'm not so sure about PCRE though
- ttgurney 4y ago
  Oh I'm sure the actual work to compile those packages is not much. It's more to do with keeping the number of packages on my system to a minimum. Actually I would not be surprised if the JavaScript engine can be omitted with just a little bit of patching work... assuming there's not actually a build configuration that leaves it out. I've found that with some software projects and their dependencies, "required" does not always mean required.
shiomiru 4y ago
> I'm lately thinking I want to see a text-mode browser that just glues together libcurl, curses, simple HTML rendering, and maybe an existing HTML parsing library. I had a similar idea a while ago, except mine was to glue together components from the nim stdlib. So I wrote something like that, then I thought "hey, why not implement some CSS too?" and that sent me down the rabbit hole of writing an actual CSS-based layout engine... I eventually also realized that the stdlib html parser is woefully inadequate for my purposes. In the end, I wrote my own mini browser engine with an HTML5 parser and whatnot. Right now I'm trying to bring it to a presentable state (i.e. integrate libcurl instead of using the curl binary, etc.) so I can publish it. Anyways, if there's a moral to this story it's that writing a browser engine is surprisingly fun, so go for it :)
- ttgurney 4y ago
  I look forward to seeing your 1st release of this program! > Anyways, if there's a moral to this story it's that writing a browser engine is surprisingly fun, so go for it :) Good to know. I'd been fairly intimidated by the idea.
smaudet 4y ago
Call it Unixy or something - unix philosophy of having each program do something separate. Makes more sense, that's what this guy does anyways with the js engine? > Surprisingly few browsers do this--Lynx, Links, and w3m all have their own networking code I think people are suspicious of curl because it is a common utility, and they think it can't possibly have got it right - plus there's something mildly fun about figuring out how to monitor a socket and send/receive IP packets for the first time. I have played around a bit with the Curl code a bit, in part I also suspect other programs do it to get "closer" i.e. being able to manage/dispatch events from a thread directly instead of some signal from a curl thread, probably something about security and thread safety too...
- shiomiru 4y ago
  The main reason for the aforementioned browsers not using libcurl is mostly historical, as it simply didn't exist back when they were created. (The newest of them is links, first released in 1999 - and according to the curl website, the first libcurl release with a proper interface was in 2000.) w3m even uses its own regex engine for search, because there was no free regex engine with Japanese support the author could've used back then.
1vuio0pswjnm7 4y ago
https://github.com/google/oss-fuzz-vulns/tree/main/vulns/curl https://github.com/google/oss-fuzz-vulns/tree/main/vulns/cur... https://github.com/curl/curl/commit/68ffe6c17d6e44b459d60805813f646d244a186b https://github.com/curl/curl/commit/68ffe6c17d6e44b459d60805... https://www.cvedetails.com/product/25084/Haxx-Curl.html?vendor_id=12682 https://www.cvedetails.com/product/25084/Haxx-Curl.html?vend... Instead of only "thinking a lot about text-based browsers", I have been actively using them on a daily basis for the past 26 years. Links already uses ncurses. I am glad that it does not use libcurl and that it has its own "bespoke" HTML rendering. In over 25 years time, I still have yet to see any other program produce better rendering of HTML tables as text. I have had few if any problems with Links versions over the years. I am quite good at "breaking" software and for me Links has been quite robust. The source code is readable for me and I have been able to change or "fix" things I do not like, then quickly recompile. I can remove features. Recently I fixed a version of the program so that a certain semantic link would not be shown in Wikipedia pages. No "browser extension" required. Links' rendering has managed to keep up with the evolution of HTML and web design sufficiently for me. Despite the enormous variation in HTML acrosse the www, there are very few cases where the rendering is unsatisfactory.^1 I cannot say the same for other attempts at text-only clients. W3C's libwww-based line-mode browser still compiles and works,^2 although I would not be satisifed with its rendering. Nor would I be satisfied with edbrowse, or something simpler such as mynx.^3 I use Links primarily for reading and printing HTML. I use a variety of TCP clients for making HTTP requests, including djb's tcpclient which I am quite sure beats libcurl any day of the week in terms quality, e.g., the programming skill level of the author and the care with which it was written. This non-libcurl networking code is relatively small and does not need oss-fuzz. I do not intentionally use libcurl. It is too large and complex for my tastes. For TLS, I mainly use stunnel and haproxy. 1. One rare example I can recall is https://archive.is https://archive.is 2. https://github.com/w3c/libwww https://github.com/w3c/libwww 3. https://github.com/SirWumpus/ioccc-mynx https://github.com/SirWumpus/ioccc-mynx
- ttgurney 4y ago
  Hey thanks for your perspective and a couple of mentions of software I'd not heard of (like tcpclient). I agree that curl is pretty big and bloated. I would not call it a deficiency that Links et al. don't depend on it. I mostly just was thinking that since I already have curl on my system, it'd be nice to have a browser that reuses that code. Especially since curl has upstream support for the much smaller BearSSL rather than depending on OpenSSL/LibreSSL.

jhallenworld 4y ago

This is very cool, but dedicated newsreaders (like trn) were easier to use to browse USENET than edbrowse is to browse Hacker News.. If you were forced to surf the web using a teletype, this is your tool. Edit: actually it's growing on me. My one complaint is that I wish it would retain the printing mode when you follow a link. For example, type "g2" to follow the 2nd link on the line. Right now I should be able to hit Enter or z to display lines from this link, but annoyingly you have to enter something like "1" to start it going.

marttt 4y ago
I think the main "selling point" of edbrowse for sighted people might be its scripting capabilities. You can automate a lot of browsing with it if you take the time. See the user guide: http://edbrowse.org/usersguide.html#script http://edbrowse.org/usersguide.html#script I've been curious about edbrowse for a long time, finding it a fascinating and inspiring project. It's a somewhat hidden gem for sure. Then again, I failed to build it on my (tiny) distro, and I occasionally think that maybe the internal scripting language, mail client, etc create "too many" temptations for a casual user. A simple line-mode website pager that encourages Unix pipes for any user-defined task (maybe even the javascript engine could be built as an external application?) would be enough for a modest hobbyist coder like me, I guess. Something like the nmh mail handler, but for web browsing: https://www.unix.com/man-page/linux/7/nmh/ https://www.unix.com/man-page/linux/7/nmh/ This is by no means meant as a harsh critique or whining, though. As indicated, the project, Karl Dahlke, Chris Bannon and other devs do deserve tremendous respect. It is quite possible that a heavily pipes-oriented web browsing would be too fragmented for actual, day-to-day usage. The line-editing paradigm nonetheless has its beauty and a compelling simplicity, even in 2022. Keep it going!

siraben 4y ago

For a command line web browser that doesn't compromise on modern web features, I use browsh[0] especially in low-bandwidth situations. In conjunction with tmux and mosh it's just fantastic because you get the full colors and features of the web with minimal bandwidth, since the rendering happens server-side and all the information that is sent to your computer is terminal half-width characters. [0] https://www.brow.sh/ https://www.brow.sh/

sylware 4y ago
You missed one important point: edbrowse does not depend on those absurdely and grotesquely massive and complex c++ web engines (blink/geeko, financed by gogol, and webkit, financed by apple). It is lean and simple C, including its deps.
- deleted 4y ago
  [deleted]

shudza 4y ago

Anyone managed to build this on arch? The AUR package fails to build.

stanleychink 4y ago

alpaca128 4y ago
Why is each of your comments a slightly modified copy of another comment in the respective thread?
- nerdponx 4y ago
  Upvote fishing in order to develop a legitimate-looking account for later sale or other forms of abuse like astroturfing?
- sylware 4y ago
  yep, I wonder what's happening here. That said, I do compile edbrowse with quickjs and pcre on my hand-rolled glibc/linux distro. I have even removed the build systems from edbrowse, quickjs and pcre to use my own. Quickjs is aiming the one-compilation-unit model (good!). Almost no javascript-ed sites run with it (but I am still testing, for instance on gogol noto font download page), but I like to keep track of its advancement, loosely, but I do it anyway. The bottom of the pb is big tech javascript-ed web engines: blink/geeko (financed by gogol) and webkit (apple). We all know the only reasonably fix for this issue is to regulate HARD towards the availability of a noscript/basic (x)html portal for all pertinent web sites (which are close to 100% anyway).

tpoacher 4y ago

Could this be used to browse/send email from a 2fa domain? I'm desperate to use a cli/tui client but my organisation's 2fa requirement has me stuck on web outlook. I'm trying to get alpine to work, but I'm not sure that it can ...