7 ms·
Show HN: A Web-to-RSS Parser in Common Lisp
- faleidel 10y agoWe should do a web-to-mardown thing since so much websites are unreadable. The funny thing is that chrome for cellphone already has something like that with the "make this website mobile friebdly". Anyway, nice project!
- danappelxx 10y agoSafari (for macOS and iOS) also has this with reader mode. Works well for most websites.
- throwanem 10y agoFirefox in recent versions, too.
- bhrgunatha 10y agoI've been disappointed so much with the changes to firefox UI over the last couple of years, so it's great to have something positive. Reader is a resounding success with me, given the terrible low contrast, difficult to read, web design trends over the same time span.
- rhabarba 10y agoThanks. :-)
- bootload 10y ago"We should do a web-to-markdown thing since so much websites are unreadable." THE FASCINATOR: that cheeky hacker Aaron beat you to this ~ http://www.aaronsw.com/2002/html2text/html2text.py http://www.aaronsw.com/2002/html2text/html2text.py and http://www.aaronsw.com/2002/html2text/ http://www.aaronsw.com/2002/html2text/ ... old, interesting to see if it's still usable after almost 15yrs. Latest code at: https://github.com/aaronsw/html2text https://github.com/aaronsw/html2text
- rhabarba 10y agoThere are quite some forks, this could be interesting to watch, thanks.
- sedachv 10y ago> We should do a web-to-mardown thing since so much websites are unreadable. Opera used to have really great settings (something like 20 different presets) for overriding CSS. Really made things a lot more legible and was a quick fix for broken websites. Sometimes I still use Lynx.
- StreamBright 10y agoI am still amazed sometimes how naturally Lisp handles data and code. You can just represent html with css and all as a nested data structure and use it when you need it. It does look or feel weird to have that next to your business logic.
- rhabarba 10y agoInline HTML/CSS would work too, but writing half of the software as a string would feel wrong.
- junke 10y agoThis request handler, in particular, produces HTML/CSS and Javascript in the same scope: https://bitbucket.org/tux_/rssparser.lisp/src/7a9f5ed45aca8b1734181be5d637b5bac745c47c/rssparser.lisp?at=default&fileviewer=file-view-default#rssparser.lisp-97 https://bitbucket.org/tux_/rssparser.lisp/src/7a9f5ed45aca8b...
- wtbob 10y agoWhat blows my mind is how, after seeing that, some folks would want to program in anything other than Lisp. Code, HTML, CSS, JavaScript, all with a single uniform representation! Seriously, anyone who's not taken a look: take a look.
- nextos 10y agoOfftopic, I use Clojure these days. Is Common Lisp still worth considering?
- rhabarba 10y agoDoes your runtime still barf JVM stack traces? (attr.: http://www.loper-os.org/?p=42&cpage=2#comment-16767 http://www.loper-os.org/?p=42&cpage=2#comment-16767) Leaving this aside (after all, it might or might not be a matter of taste), it totally is. The Common Lisp ecosystem is pretty well alive, despite of the rise of Racket and Clojure. You should really give it a try one day. :-)
- noobermin 10y agoRandom question, how many people still use RSS? Just out of curiosity. EDIT: And while I'm at it, are his/er packages what one usually uses in common lisp for web development?
- nyolfen 10y agomany? here's a thread from last night with dozens of comments about rss readers https://news.ycombinator.com/item?id=12932429 https://news.ycombinator.com/item?id=12932429
- rhabarba 10y agoat EDIT: I don't care as long as it does the job. There's Caveman2 for a full-features web stack but I thought it was overkill. After all, the web UI is - more or less - an "add-on". :-)
- eth0up 10y ago"...how many people still use RSS?" +1
- shakna 10y agoI use RSS everywhere I can. It allows me to build a central source for everything I'm interested in. Quicklisp is par for the course with Common Lisp nowadays, and the dependencies all look fairly normal to me. If you were going for production code, I'd throw Huchentoot behind Nginx.
- netghost 10y agoI find it really valuable, and follow a few dozen sites. I'm curious, if folks don't use an RSS reader, how do they find articles and follow sites they're interested in?
- symlinkk 10y agoTwitter / Facebook / Reddit
- rcarmo 10y ago
- williamle8300 10y agoWhich RSS reader do you use?
- armitron 10y agoThis is far from ideal (and a bad idea in general): (load "~/quicklisp/setup.lisp") (ql:quickload '(:datafly ;; for database access ;; WEB SERVER: :hunchentoot ;; for providing a web server :cl-who ;; for building the HTML output :parenscript ;; for the avoiding of the horrible JS syntax :smackjack ;; for AJAX requests :lass ;; for building the (S)CSS styles You should not use Quicklisp to silently install dependencies as part of your normal operation sequence, because: + It imposes its own view on how libraries are retrieved, stored, managed and updated. + It is vulnerable to man-in-the-middle attacks. + Used in this fashion, it effectively becomes part of your program. The more people that do this, the more ingrained it becomes. + Plenty of us do not use Quicklisp at all, preferring other management schemes. A better scheme for your project would be to have an install-deps-via-quicklisp.lisp script file, that pulls down and installs the dependencies via Quicklisp, for those that want to go that route and also a README that lists all the dependencies (associated github repos or homepages) . That way you decouple normal execution from dependency installation and those of us that do not use Quicklisp are satisfied.
- rhabarba 10y agoThank you. But what would be the preferred way to load the installed libraries then? (Sorry, I rarely read style books.)
- PuercoPop 10y agoOne can use quicklisp bundles so that the user doesn't have to install quicklisp and download dependencies themselves https://www.quicklisp.org/beta/bundles.html https://www.quicklisp.org/beta/bundles.html
- rhabarba 10y agoThat would basically include Quicklisp again. I thought I shouldn't?
- srgseg 10y agoFYI this functionality is 'automagically' implemented by specifying just a web site URL within the http://www.protopage.com http://www.protopage.com RSS reader. It scans a web site and figures out which are the article headlines and links.
- rhabarba 10y agoWhich might (and probably will) fail for a lot of websites.
- bootload 10y ago"written because a disappointing number of websites still does not have an RSS or Atom feed" Or a way to programatically parse websites, process with a bit of code, to view sites off-line. Really interesting piece of code.
- rhabarba 10y agoThanks. :)
- cheiVia0 10y agoDoes this work with Facebook? They turned off RSS recently and it was very annoying.
- rhabarba 10y agoIt can't log in for you, sorry. :-) But it should be able to process publicly visible contents.
- narrowrail 10y agoThis could be paired with a browser extension or selenium?
- rhabarba 10y agoTechnically, yes.
- evmar 10y ago> This software was written because a disappointing number of websites still does not have an RSS or Atom feed so I could subscribe to their updates, e.g. the KiTTY website. It seems the KiTTY website does have an RSS feed at http://www.9bis.net/kitty/data/rss/rssen.xml http://www.9bis.net/kitty/data/rss/rssen.xml
- rhabarba 10y agoYes, it does. But you might see that this does not just list new KiTTY updates but anything that happens on that website; which is not exactly handy, is it?
- nreece 10y agoPretty cool example of Lisp's capabilities. When trying to create feeds from webpages in the real-world, there are plenty of pitfalls involved, for example: handling JavaScript content, graceful retries, bypass IP blocks, throttle & rate-limit requests, accessing public social media (Facebook, Twitter, Google) etc. At Feedity (https://feedity.com https://feedity.com), we've developed our own little system over the years using .NET (C#) and node.js, with a bunch of tweaks and optimizations, for generating custom feeds from public webpages.
- k1m 10y agoThis looks really nice. We've got something similar (written in PHP) at FiveFilters.org http://createfeed.fivefilters.org http://createfeed.fivefilters.org
- rhabarba 10y agoI had a look at your software a while ago. I would probably use it if it was free. (I totally respect you for writing it though!)
- burtonator 10y agoI posted here the other day about the death of RSS: https://www.spinn3r.com/blog/2016/11/09/The-Death-of-RSS-Long-Live-Open-Web.html https://www.spinn3r.com/blog/2016/11/09/The-Death-of-RSS-Lon... And this kind of codifies my point even though a number of people in the comments here were calling me crazy ;)