3 ms·

I have just built a Notion integration that pulls pages into our statically built API documentation website, and it was, frankly, horrible. While the end result

by 9dev 3y ago

I have just built a Notion integration that pulls pages into our statically built API documentation website, and it was, frankly, horrible. While the end result works (the team can write docs in the tool they know, the site is built and released from the structure there automatically), it was a lot of pain to even discern children from their parent pages, parse attributes or let alone get databases right. Considering I’ll need to get other data in there soon, probably, I’m in the market for Psychic. The question I have, though, is: can you really reconcile the Schema of several apps into one, without settling for the smallest common denominator? What do you do about platforms like Notion, that don’t even provide webhooks? We settled on polling, but obviously that won’t scale.

jasonwcfan 3y ago
Reconciling schemas -> this will be hard. We're starting with just two data models (Documents and Conversations) that are relatively universal, but there's no way to avoid a lossy transformation from Notion because things like tables and embeds aren't neatly captured as a Document without making our data models just as complex. I suspect LLMs can help finding a good balance between generalization/depth since they're very good at automating work that typically requires a lot of customization. Data syncs -> If the source doesn't offer webhooks, we just poll daily, do a diff on our side, and send the updated data to our customer. I'm not aware of any way to avoid polling when webhooks aren't available, but we plan to do the polling ourselves so we can provide a webhook like experience for customers.
- enos_feedler 3y ago
  do you think you'll have issues with rate limiting and security blocks more generally if you are doing lots of polling from your side on behalf of many integrations? One time i wrote a script to poll for changes to my frequent flyer mile account and they banned me and froze my miles. just from a polling script.
  - jasonwcfan 3y ago
    Yes - we will run into this problem eventually with a single OAuth application. Having our customers use their own credentials is a workaround but won't work for some platforms that have a really onerous approval policy for new apps. If you or anyone else knows of a solution to this problem let me know, because we haven't crossed that bridge yet.
    - drekk 3y ago
      I worked at [undisclosed Fintech] and their solution was pretty funny. They had a rack of mobile phones on 3G/4G running Android to poll [undisclosed throttle-happy bank]. Was an incredible hackathon project that lasted years.
  - nickphx 3y ago
    Even with oauth applications you will experience rate limiting 'at scale' when many users have authorized your single "application" to perform actions. In my experience, some cases make it easier to have the customer create and provide the oauth application details to help mitigate rate limits, review requirements, and access restrictions.
    - jasonwcfan 3y ago
      Yeah we do offer that actually! For customers with significant volume it'll help lighten the load

by 9dev 3y ago

jasonwcfan 3y ago

Reconciling schemas -> this will be hard. We're starting with just two data models (Documents and Conversations) that are relatively universal, but there's no way to avoid a lossy transformation from Notion because things like tables and embeds aren't neatly captured as a Document without making our data models just as complex. I suspect LLMs can help finding a good balance between generalization/depth since they're very good at automating work that typically requires a lot of customization. Data syncs -> If the source doesn't offer webhooks, we just poll daily, do a diff on our side, and send the updated data to our customer. I'm not aware of any way to avoid polling when webhooks aren't available, but we plan to do the polling ourselves so we can provide a webhook like experience for customers.

enos_feedler 3y ago
do you think you'll have issues with rate limiting and security blocks more generally if you are doing lots of polling from your side on behalf of many integrations? One time i wrote a script to poll for changes to my frequent flyer mile account and they banned me and froze my miles. just from a polling script.
- jasonwcfan 3y ago
  Yes - we will run into this problem eventually with a single OAuth application. Having our customers use their own credentials is a workaround but won't work for some platforms that have a really onerous approval policy for new apps. If you or anyone else knows of a solution to this problem let me know, because we haven't crossed that bridge yet.
  - drekk 3y ago
    I worked at [undisclosed Fintech] and their solution was pretty funny. They had a rack of mobile phones on 3G/4G running Android to poll [undisclosed throttle-happy bank]. Was an incredible hackathon project that lasted years.
- nickphx 3y ago
  Even with oauth applications you will experience rate limiting 'at scale' when many users have authorized your single "application" to perform actions. In my experience, some cases make it easier to have the customer create and provide the oauth application details to help mitigate rate limits, review requirements, and access restrictions.
  - jasonwcfan 3y ago
    Yeah we do offer that actually! For customers with significant volume it'll help lighten the load