3 ms·

ASK HN Help with capturing API xhr request automatically

2 points by charlesrach 2y ago

So I’ve been making a website that automates making schedules for my university. The way it works now is I paste the json data of the courses into my website and it does the rest. The way i get this json is by going on my school website, inspecting element, going to fetch/xhr, and when i press a button on the site i go into the request and find the json in the response. Now for people that aren’t too tech savvy (even for just inspect element) is there a way to automate this within my website so that all they do is login press a button and my site does the rest. Any alternative solutions are also appreciated. Thanks!

solardev 2y ago
Can you make it a chrome extension instead?
- charlesrach 2y ago
  Even if i could make a chrome extension that automates copying that json so they can paste it later would be beneficial but is that even possible?
  - solardev 2y ago
    They wouldn't paste it, your entire app would be in the chrome extension. Is what possible, exactly?
    - charlesrach 2y ago
      Oh I see. I was wondering if that would make it possible to capture that json being sent
      - solardev 2y ago
        Should be. Extensions generally have a lot more privilege. https://medium.com/@ddamico.125/intercepting-network-response-body-with-a-chrome-extension-b5b9f2ef9466 https://medium.com/@ddamico.125/intercepting-network-respons... Not sure how that will change with manifest V3 though. Instead of intercepting the request, you might also be able to monkey patch the submit button or just scrape their cookies / localstorage for auth and do your own request. Or maybe read their state from memory. Not sure though. A JS bookmarklet is another option. Not as good UX and can't auto update, but better than trying to manually get them to open the inspector or console.
cebert 2y ago
Is authentication required to make a schedule? You might want to look into using Playwright if you’re looking to programmatically parse these pages as a service. Also, you should add the Ask HN prefix to this submission title.
- charlesrach 2y ago
  Authentication is required to get the course info from the school website since each student has different courses but my website is a single use and doesn’t require any logins. I will look into playwright though what is it exactly?
  - solardev 2y ago
    Playwright is a headless browser automation framework. It lets you script browser actions in an offscreen window, simulating user activity in a real browser. It's not really the issue here though. If you want to capture real user auth and requests, you have to man in the middle them somehow. A Chrome extension is probably the easiest to do that, in terms of UX, because you can interact with the university website directly and intercept its requests. A third party website is subject to XSS protections so your user will have to jump through hoops.

2 points by charlesrach 2y ago

solardev 2y ago

Can you make it a chrome extension instead?

charlesrach 2y ago
Even if i could make a chrome extension that automates copying that json so they can paste it later would be beneficial but is that even possible?
- solardev 2y ago
  They wouldn't paste it, your entire app would be in the chrome extension. Is what possible, exactly?
  - charlesrach 2y ago
    Oh I see. I was wondering if that would make it possible to capture that json being sent
    - solardev 2y ago
      Should be. Extensions generally have a lot more privilege. https://medium.com/@ddamico.125/intercepting-network-response-body-with-a-chrome-extension-b5b9f2ef9466 https://medium.com/@ddamico.125/intercepting-network-respons... Not sure how that will change with manifest V3 though. Instead of intercepting the request, you might also be able to monkey patch the submit button or just scrape their cookies / localstorage for auth and do your own request. Or maybe read their state from memory. Not sure though. A JS bookmarklet is another option. Not as good UX and can't auto update, but better than trying to manually get them to open the inspector or console.

cebert 2y ago

Is authentication required to make a schedule? You might want to look into using Playwright if you’re looking to programmatically parse these pages as a service. Also, you should add the Ask HN prefix to this submission title.

charlesrach 2y ago
Authentication is required to get the course info from the school website since each student has different courses but my website is a single use and doesn’t require any logins. I will look into playwright though what is it exactly?
- solardev 2y ago
  Playwright is a headless browser automation framework. It lets you script browser actions in an offscreen window, simulating user activity in a real browser. It's not really the issue here though. If you want to capture real user auth and requests, you have to man in the middle them somehow. A Chrome extension is probably the easiest to do that, in terms of UX, because you can interact with the university website directly and intercept its requests. A third party website is subject to XSS protections so your user will have to jump through hoops.