This is an interesting proof of concept. I'm impressed at how simple the SMS signup code is. For many universities though, it won't be this simple. Many hide

This is an interesting proof of concept. I'm impressed at how simple the SMS signup code is. For many universities though, it won't be this simple. Many hide the open seats behind a login, which means you need to be a student there to see them. Many also use student information systems that are a real pain to scrape like this. Once you get to the point where interacting with the student information system is done via AJAX requests mediated by a mess of JavaScript, it starts to push the limits of what you can do with this technique. When it gets to this point you either have to carefully dissect everything on the Network tab of your developer tools, or use a real browser. I've used PhantomJS (basically a headless WebKit) and Selenium WebDriver with great success on sites that are not amenable to scraping. The neat thing is that you really only need to use that for the interaction to get to where the information is. Once you've navigated there you can just have it dump the rendered page as HTML and parse it using the same techniques shown here.