5 ms·

Lessons learned from 6 months of operating a tiny news archive

124 points by hiAndrewQuinn 2y ago

metaph6 2y ago
i get 404 :)
- hiAndrewQuinn 2y ago
  Strange, it's working for me :0 Github is my backup. You can find the article here as well, along with everything else I've written at my TIL site: https://github.com/hiAndrewQuinn/til/blob/main/lessons-learned-from-6-months-of-operating-a-teensy-tiny-news-archive.md https://github.com/hiAndrewQuinn/til/blob/main/lessons-learn...
  - metaph6 2y ago
    Tabs Archive and Search return 404, fyi
    - hiAndrewQuinn 2y ago
      Joo, Tags works fine for me but I'm aware of Archive and Search being broken. I gotta get around to removing them one of these days.
aeurielesn 2y ago
It would be best to not hotlink the images. Linking back to the source would be great too.
- hiAndrewQuinn 2y ago
  Re/ not hotlinking the images: Not gonna happen :) I'm not going to balloon the size of the repo, or add the overhead of a database, just so I can locally serve those images. I'd rather take them out entirely at that point. Re/ linking back to the source: Great idea! I look forward to your PR. Start with https://github.com/hiAndrewQuinn/selkouutiset-scrape/ https://github.com/hiAndrewQuinn/selkouutiset-scrape/ and see if you can figure out the logic behind how YLE.fi chooses its archive tags, maybe there's a pattern in there I was unable to see when I tried this myself.
- draw_down 2y ago
  [dead]
karma_pharmer 2y ago
Bare HTML never goes out of style. My archive doesn’t use any JavaScript at all, it’s just HTML with the smallest amount of classless CSS I could get away with I love you, will you marry me? Can you make the rest of the web like this? That would be awesome.
- hiAndrewQuinn 2y ago
  <3 The bad news is market forces are such that Web 1.0 will never again be profitable. The good news is market forces are such that Web 1.0 is effectively unkillable.
  - walterbell 2y ago
    > Web 1.0 will never again be profitable .. if not already profitable due to historical network effects, like https://craigslist.org https://craigslist.org which was ~$10M annual revenue per employee, north of $500M/year total.
- cranberryturkey 2y ago
  checkout https://brisk.news https://brisk.news
  - karma_pharmer 2y ago
    I checked it out, and it doesn't seem to do anything at all if you have javascript turned off.
    - cranberryturkey 2y ago
      no, it won't. you need js
- xenodium 2y ago
  > Can you make the rest of the web like this? That would be awesome. I'm working on something like that for blog hosting: https://lmno.lol https://lmno.lol No JS needed to read anything. You can browse from termina/lynx if you fancy Here's mine blog on it: https://lmno.lol/alvaro https://lmno.lol/alvaro ps. Service hasn't officially launched, though happy to share invites. Already got a handful of happy users. Ping me (help at lmno.lol).
luxuryballs 2y ago
my language skills and I have been profiting off of it ever since
- sdwr 2y ago
  Language skills, not language skills!
meiraleal 2y ago
Why is no-JS so appealing to some people? Honest question, I wanna understand. Edit: by the replies what I understood is that people that dislike JS think the web is a medium exclusively to read? Apps should be desktop executables. Fair enough. It's a good thing this is a negligible % of internet users.
- Dalewyn 2y ago
  Downloads quicker, responds instantly, less load on system resources, no country mile wide attack vector.
- PaulRobinson 2y ago
  Page bloat is one reason, tracking/privacy-invading JS is a pest, but for me at least, my main problem is that most JS is used to break how people want to use the web. The web is a document-centric medium. I load a URL, I want to look at the document presented at that page. Things I likely don't want to do is deal with banners, pop-overs or other code grabbing my attention (if you don't show me a cookie banner you've done the right thing: not use cookies that aren't strictly necessary, thanks!). I probably don't want infinite scroll, "funky" animations or other stuff that tries to break the document medium. I think the only exception is if I'm looking at a document that wants to use an interactive chart, that's fine. Form validation might be helpful, but do you really need anything other than built-in HTML client-side validation?
  - Y_Y 2y ago
    I like this explanation. If I was trying to be pithy I might rewrite it as something like: JS code is usually run by people I don't necessarily trust, for interests other than my own.
- ripe 2y ago
  It's not because of a religious opposition to JS, but because usually Javascript-laden sites take a long time to load, require the latest hardware and browsers, and come with pop-up banners, third-party ads, and bloatware.
brookst 2y ago
> Simplicity is underrated I agree, or at least simplicity is often given lip service while complexity is implemented because it’s easier. > 3 Git repos, each submoduling and iterating upon the last Not throwing stones, just curious — is that really the simplest approach? I would have thought one repo, with folders for source, raw articles, and website. You’ve obviously given it a lot of thought. Why this model?
- jauntywundrkind 2y ago
  I was going to call this out as something I super loved. > The first repo archives the page as-is with a cronjob. The second repo turns the first repo into cleaned-up Markdown with an English translation with another cronjob. The third repo turns the cleaned-up Markdown into a Hugo website with a third cronjob. This is a super awesome pattern, imo. You all never have two systems try to make commits onto the same head, no clashing. If one process goes bad, you don't have to meddle with interweaved history, your source history is pure. It also makes the event sourcing easy; when a repo changes the next process starts, where-as you have to start filtering down your edge triggering if you conflate concerns together. In this case the author is using cron jobs, but this would be a viable medium-throughput modest-latency event-source system with little change! This to me is much simpler. There's little appealing to me about conflating streams. Having the history show up as a continuous record within a repo could be seen as an advantage, but even still I'd rather make a fourth repo that merges the streams after the fact. Simplicity is in the eye of beholder. Not conflating different concerns feels to me much simpler.
  - skeeter2020 2y ago
    This mirrors how many big data / data warehousing pipelines work. Originally a lot of transformations tended to get pulled into the same flow as moving data around, but storage has gotten progressively cheaper and this is so much simpler and durable.
    - hiAndrewQuinn 2y ago
      That's essentially why I gravitated to it, yeah. To me, Git submodules are a pretty generalized and elegant solution to the question of "how do I model a directory of files as an immutable data structure". If you know the commit hash of the submodule, you know exactly what is going to be in there. That kind of certainty is very helpful when trying to code up tricky data cleaning and the like.
sed3 2y ago
I have a bit different workflow: - in first step I "tag" urls I like (in rss reader, HN, twitter, pdfs, podcast...). It is very quick, I usually spend like 2 hours per week filtering news. - latter I collect tagged urls (from email, notes and bookmarks) and feed them into batch job - batch job downloads text form, and converts them into markdown. HTML are converted with Reader mode, videos have transcripts with time marks... - this gets converted into new Obsidian vault and synced with Syncthing across my devices. - there is AI job that does some annotation and linking with existing notes. - over next weeks (usually two) I slowly "digest" content, read it, and annotate. At end it becomes integrated into my Obsidian notes. I can copy&paste and write directly into articles. It is very natural process for learning. I spend minimal time dealing with junk content, clickbait, ads... And it works for all types of content including video and PDFs. And offline absolutely rocks!
- hobo_mark 2y ago
  Are you some sort of writer/reporter or is this all for entertainment?
  - sed3 2y ago
    Software consultant. I like to group topics (like new Kafka release) together and read it all at once, rather than slow trickle over two weeks. I also like to have my sources directly linked with notes. Offline archive is just easiest way. I started working on my own system after Google Reader shutdown in 2013. And I really hate ads.
- potatoball 2y ago
  Do you mind elaborating on your process for ‘tagging’? I’ve been looking for some type of solution to send a url to something which can then scrape the URL. Currently my process is to just send it to myself on a self hosted Zulip server (Slack like alternative) but I find it a bit cumbersome. I’ve been considering an email based system after reading a blogpost from Stephen Wolfram someone posted recently here but I‘ve been looking for what others have done as well. Aforementioned blogpost: https://writings.stephenwolfram.com/2019/02/seeking-the-productive-life-some-details-of-my-personal-infrastructure/ https://writings.stephenwolfram.com/2019/02/seeking-the-prod...
deleted 2y ago
[deleted]

124 points by hiAndrewQuinn 2y ago

metaph6 2y ago

i get 404 :)

hiAndrewQuinn 2y ago
Strange, it's working for me :0 Github is my backup. You can find the article here as well, along with everything else I've written at my TIL site: https://github.com/hiAndrewQuinn/til/blob/main/lessons-learned-from-6-months-of-operating-a-teensy-tiny-news-archive.md https://github.com/hiAndrewQuinn/til/blob/main/lessons-learn...
- metaph6 2y ago
  Tabs Archive and Search return 404, fyi
  - hiAndrewQuinn 2y ago
    Joo, Tags works fine for me but I'm aware of Archive and Search being broken. I gotta get around to removing them one of these days.

aeurielesn 2y ago

It would be best to not hotlink the images. Linking back to the source would be great too.

hiAndrewQuinn 2y ago
Re/ not hotlinking the images: Not gonna happen :) I'm not going to balloon the size of the repo, or add the overhead of a database, just so I can locally serve those images. I'd rather take them out entirely at that point. Re/ linking back to the source: Great idea! I look forward to your PR. Start with https://github.com/hiAndrewQuinn/selkouutiset-scrape/ https://github.com/hiAndrewQuinn/selkouutiset-scrape/ and see if you can figure out the logic behind how YLE.fi chooses its archive tags, maybe there's a pattern in there I was unable to see when I tried this myself.
draw_down 2y ago
[dead]

karma_pharmer 2y ago

Bare HTML never goes out of style. My archive doesn’t use any JavaScript at all, it’s just HTML with the smallest amount of classless CSS I could get away with I love you, will you marry me? Can you make the rest of the web like this? That would be awesome.

hiAndrewQuinn 2y ago
<3 The bad news is market forces are such that Web 1.0 will never again be profitable. The good news is market forces are such that Web 1.0 is effectively unkillable.
- walterbell 2y ago
  > Web 1.0 will never again be profitable .. if not already profitable due to historical network effects, like https://craigslist.org https://craigslist.org which was ~$10M annual revenue per employee, north of $500M/year total.
cranberryturkey 2y ago
checkout https://brisk.news https://brisk.news
- karma_pharmer 2y ago
  I checked it out, and it doesn't seem to do anything at all if you have javascript turned off.
  - cranberryturkey 2y ago
    no, it won't. you need js
xenodium 2y ago
> Can you make the rest of the web like this? That would be awesome. I'm working on something like that for blog hosting: https://lmno.lol https://lmno.lol No JS needed to read anything. You can browse from termina/lynx if you fancy Here's mine blog on it: https://lmno.lol/alvaro https://lmno.lol/alvaro ps. Service hasn't officially launched, though happy to share invites. Already got a handful of happy users. Ping me (help at lmno.lol).

luxuryballs 2y ago

my language skills and I have been profiting off of it ever since

sdwr 2y ago
Language skills, not language skills!

meiraleal 2y ago

Why is no-JS so appealing to some people? Honest question, I wanna understand. Edit: by the replies what I understood is that people that dislike JS think the web is a medium exclusively to read? Apps should be desktop executables. Fair enough. It's a good thing this is a negligible % of internet users.

Dalewyn 2y ago
Downloads quicker, responds instantly, less load on system resources, no country mile wide attack vector.
PaulRobinson 2y ago
Page bloat is one reason, tracking/privacy-invading JS is a pest, but for me at least, my main problem is that most JS is used to break how people want to use the web. The web is a document-centric medium. I load a URL, I want to look at the document presented at that page. Things I likely don't want to do is deal with banners, pop-overs or other code grabbing my attention (if you don't show me a cookie banner you've done the right thing: not use cookies that aren't strictly necessary, thanks!). I probably don't want infinite scroll, "funky" animations or other stuff that tries to break the document medium. I think the only exception is if I'm looking at a document that wants to use an interactive chart, that's fine. Form validation might be helpful, but do you really need anything other than built-in HTML client-side validation?
- Y_Y 2y ago
  I like this explanation. If I was trying to be pithy I might rewrite it as something like: JS code is usually run by people I don't necessarily trust, for interests other than my own.
ripe 2y ago
It's not because of a religious opposition to JS, but because usually Javascript-laden sites take a long time to load, require the latest hardware and browsers, and come with pop-up banners, third-party ads, and bloatware.

brookst 2y ago

> Simplicity is underrated I agree, or at least simplicity is often given lip service while complexity is implemented because it’s easier. > 3 Git repos, each submoduling and iterating upon the last Not throwing stones, just curious — is that really the simplest approach? I would have thought one repo, with folders for source, raw articles, and website. You’ve obviously given it a lot of thought. Why this model?

jauntywundrkind 2y ago
I was going to call this out as something I super loved. > The first repo archives the page as-is with a cronjob. The second repo turns the first repo into cleaned-up Markdown with an English translation with another cronjob. The third repo turns the cleaned-up Markdown into a Hugo website with a third cronjob. This is a super awesome pattern, imo. You all never have two systems try to make commits onto the same head, no clashing. If one process goes bad, you don't have to meddle with interweaved history, your source history is pure. It also makes the event sourcing easy; when a repo changes the next process starts, where-as you have to start filtering down your edge triggering if you conflate concerns together. In this case the author is using cron jobs, but this would be a viable medium-throughput modest-latency event-source system with little change! This to me is much simpler. There's little appealing to me about conflating streams. Having the history show up as a continuous record within a repo could be seen as an advantage, but even still I'd rather make a fourth repo that merges the streams after the fact. Simplicity is in the eye of beholder. Not conflating different concerns feels to me much simpler.
- skeeter2020 2y ago
  This mirrors how many big data / data warehousing pipelines work. Originally a lot of transformations tended to get pulled into the same flow as moving data around, but storage has gotten progressively cheaper and this is so much simpler and durable.
  - hiAndrewQuinn 2y ago
    That's essentially why I gravitated to it, yeah. To me, Git submodules are a pretty generalized and elegant solution to the question of "how do I model a directory of files as an immutable data structure". If you know the commit hash of the submodule, you know exactly what is going to be in there. That kind of certainty is very helpful when trying to code up tricky data cleaning and the like.

sed3 2y ago

I have a bit different workflow: - in first step I "tag" urls I like (in rss reader, HN, twitter, pdfs, podcast...). It is very quick, I usually spend like 2 hours per week filtering news. - latter I collect tagged urls (from email, notes and bookmarks) and feed them into batch job - batch job downloads text form, and converts them into markdown. HTML are converted with Reader mode, videos have transcripts with time marks... - this gets converted into new Obsidian vault and synced with Syncthing across my devices. - there is AI job that does some annotation and linking with existing notes. - over next weeks (usually two) I slowly "digest" content, read it, and annotate. At end it becomes integrated into my Obsidian notes. I can copy&paste and write directly into articles. It is very natural process for learning. I spend minimal time dealing with junk content, clickbait, ads... And it works for all types of content including video and PDFs. And offline absolutely rocks!

hobo_mark 2y ago
Are you some sort of writer/reporter or is this all for entertainment?
- sed3 2y ago
  Software consultant. I like to group topics (like new Kafka release) together and read it all at once, rather than slow trickle over two weeks. I also like to have my sources directly linked with notes. Offline archive is just easiest way. I started working on my own system after Google Reader shutdown in 2013. And I really hate ads.
potatoball 2y ago
Do you mind elaborating on your process for ‘tagging’? I’ve been looking for some type of solution to send a url to something which can then scrape the URL. Currently my process is to just send it to myself on a self hosted Zulip server (Slack like alternative) but I find it a bit cumbersome. I’ve been considering an email based system after reading a blogpost from Stephen Wolfram someone posted recently here but I‘ve been looking for what others have done as well. Aforementioned blogpost: https://writings.stephenwolfram.com/2019/02/seeking-the-productive-life-some-details-of-my-personal-infrastructure/ https://writings.stephenwolfram.com/2019/02/seeking-the-prod...

deleted 2y ago

[deleted]