I’ve done a similar thing with APIs that return JSON. I write a small script which strictly queries the API and dumps the raw JSON into Postgres. Any structured

I’ve done a similar thing with APIs that return JSON. I write a small script which strictly queries the API and dumps the raw JSON into Postgres. Any structured data can be derived from the raw results (optionally with triggers) but what’s key is you will never reject a response from the API - anything goes. A nice of extension of this is having a distinct process using listen/notify on a derived table to drive notifications. I used this design to notify me of apartment listings changing and it was quite reliable.