3 ms·

- since we are on the topic i got a question for you - what is the best way to store rss data inside postgres - will you store it as raw xml directly? or will

- since we are on the topic i got a question for you - what is the best way to store rss data inside postgres - will you store it as raw xml directly? or will you process it and store it? - what format will you store it as if stored inside postgres? - how will you handle updates to the feed? rss doesnt do websocket push as far as i can tell so you have to poll which gives you the entire feed with a few items removed and a few new items added in the best case scenario. how do you diff this for storing?

mmakeev 3mo ago
[dead]
Kwpolska 3mo ago
If you're building a RSS reader, you'd probably want to process the data and store all relevant fields as columns for storage. Not much point to storing the XML. Entries in feeds should have a guid you can use to detect updates/entries you have already seen. RSS readers typically don't delete entries that disappeared from the feed - entries disappear from feeds to limit their size. You can get live updates to some RSS feeds using WebSub.
- vivzkestrel 3mo ago
  - the idea behind storing the raw feed items instead of processed ones is - what happens if your processing logic changes 6 months down the line? - for example today you decide you want to remove all extra spaces and lowercase all titles before storing them - 6 months down the line you want to revert, what now?
  - setr 3mo ago
    For that purpose, you’re perfectly well off just leaving it as an opaque blob; compress, base64 encode and stuff it into a string/varbinary column and call it a day. You don’t even need to store it in pg itself; stuff it into a cheaper datastore like s3 and just have the locations stored in pg. The only thing to optimize for is cost & storage. Access/retrieval doesn’t matter for a once in 6 months process. Also what is this revolting formatting strategy you’ve found?
  - munch117 3mo ago
    Absolutely store the raw data. For the reasons you state. But also store derived data. Titles, authors, dates, article texts. You need those for whatever your application does. You don't want your application logic to be working with the raw text. > how will you handle updates to the feed? When polling, consider using HTTP HEAD to check for changes before GET. What you do when an article ID reappears with different content, that up to you. I think readers usually replace the old entry with the new content, silently. But it's not the only option.
    - vivzkestrel 3mo ago
      so how do you store raw xml data in postgres?
      - paulryanrogers 3mo ago
        As text or XML column type. IME a document store or columnar backend is significantly cheaper if you're not querying it from SQL.
        vivzkestrel 3mo ago
        - there is definitely one dilemma here - if you split the rss document into its "items" i am not sure if you can store each item separately inside postgres using that XML data type - if you store the whole document, you end up with a problem when say the feed partially updates after a few minutes - one suggestion i hear from r/PostgreSQL is that you store the XML blob somewhere else like hstore or something and somehow index it back to postgres. I wonder how that works conceptually / architecturally speaking
        paulryanrogers 3mo ago
        Pg can do array columns, though I don't think it's a good fit for common use cases
        zaptheimpaler 3mo ago
  - inigyou 3mo ago
    A reader does things in realtime. You don't care what happened 6 months ago. If you did something 6 months ago you don't want to gaslight the user by pretending you did something different 6 months ago.
inigyou 3mo ago
For polling, the ultimate test is whether Rachel likes it. Check this and her other posts: https://rachelbythebay.com/w/2026/02/23/readers/ https://rachelbythebay.com/w/2026/02/23/readers/

mmakeev 3mo ago

[dead]

Kwpolska 3mo ago

If you're building a RSS reader, you'd probably want to process the data and store all relevant fields as columns for storage. Not much point to storing the XML. Entries in feeds should have a guid you can use to detect updates/entries you have already seen. RSS readers typically don't delete entries that disappeared from the feed - entries disappear from feeds to limit their size. You can get live updates to some RSS feeds using WebSub.

vivzkestrel 3mo ago
- the idea behind storing the raw feed items instead of processed ones is - what happens if your processing logic changes 6 months down the line? - for example today you decide you want to remove all extra spaces and lowercase all titles before storing them - 6 months down the line you want to revert, what now?
- setr 3mo ago
  For that purpose, you’re perfectly well off just leaving it as an opaque blob; compress, base64 encode and stuff it into a string/varbinary column and call it a day. You don’t even need to store it in pg itself; stuff it into a cheaper datastore like s3 and just have the locations stored in pg. The only thing to optimize for is cost & storage. Access/retrieval doesn’t matter for a once in 6 months process. Also what is this revolting formatting strategy you’ve found?
- munch117 3mo ago
  Absolutely store the raw data. For the reasons you state. But also store derived data. Titles, authors, dates, article texts. You need those for whatever your application does. You don't want your application logic to be working with the raw text. > how will you handle updates to the feed? When polling, consider using HTTP HEAD to check for changes before GET. What you do when an article ID reappears with different content, that up to you. I think readers usually replace the old entry with the new content, silently. But it's not the only option.
  - vivzkestrel 3mo ago
    so how do you store raw xml data in postgres?
    - paulryanrogers 3mo ago
      As text or XML column type. IME a document store or columnar backend is significantly cheaper if you're not querying it from SQL.
      - vivzkestrel 3mo ago
        - there is definitely one dilemma here - if you split the rss document into its "items" i am not sure if you can store each item separately inside postgres using that XML data type - if you store the whole document, you end up with a problem when say the feed partially updates after a few minutes - one suggestion i hear from r/PostgreSQL is that you store the XML blob somewhere else like hstore or something and somehow index it back to postgres. I wonder how that works conceptually / architecturally speaking
        paulryanrogers 3mo ago
        Pg can do array columns, though I don't think it's a good fit for common use cases
        zaptheimpaler 3mo ago
- inigyou 3mo ago
  A reader does things in realtime. You don't care what happened 6 months ago. If you did something 6 months ago you don't want to gaslight the user by pretending you did something different 6 months ago.

inigyou 3mo ago

For polling, the ultimate test is whether Rachel likes it. Check this and her other posts: https://rachelbythebay.com/w/2026/02/23/readers/ https://rachelbythebay.com/w/2026/02/23/readers/