12 ms·
How I Hacked My University's Registration System with Python and Twilio
- anirudt 9y agoProbably the word "hacked" made it click baity, atleast in my opinion. Automation, yes. Not sure there was a vulnerability involved.
- jacquesm 9y agoA 'neat hack' is the original use of the word hack. Hacking does not at all require the involvement of vulnerabilities.
- anirudt 9y agoAgreed, but it's just automation at play here. Clever, and will save time though, just thought that the headline was a little misleading, considering a registration system was involved.
- old_chap 9y agoEh, I haven't read the article but I'm pretty sure hacking is doing something outside its intended purpose.
- anirudt 9y agoAgreed, but it's just automation at play here. Clever, and will save time though, just thought that the headline was a little misleading, considering a registration system was involved.
- saamm 9y agoApologies for the mislead! When I titled this, I imagined talking a lot more about stepping through the registration system in something like mitmproxy to find the exact requests to send, which feels a lot closer to hacking/reverse engineering. I ended up not really talking about that, but neglected to change the title.
- deleted 9y ago[deleted]
- Cyph0n 9y agoI'd be worried if it was actually about "hacking", because the result is usually very severe. Universities are still quite backwards went it comes to security unfortunately.
- rasmi 9y agoIt's also a bit risky, as some University administrators may not understand that he isn't actually being malicious. Automated scraping can be considered a violation of some IT policies though, so OP should be careful about something like this.
- ian_d 9y agoI'm in higher-ed infosys and we see a number of automated registration sniping hacks in our monitoring / metrics. We usually don't run it up the chain unless it's malicious or impacting the service for other students. Otherwise we just investigate it a little bit and share it on Slack with the group if it's clever. The only time we've ever reached out to a student is to ask him to stop sharing it with others (and fix it) because it was performing some really aggressive polling. Some other schools in our system are really reactionary, though, and consider any automation a ToS violation and will freak out. e: And if you know the URL pattern / platform of your Uni's registration system, there's probably already a couple of examples on github of a registration bot.
- coleca 9y agoI've seen where it could also be seen as a violation of the academic honor code. Georgia Tech has some serious warnings about trying this sort of stuff.
- dsfyu404ed 9y agoBurp is probably the better tool for the job in most cases since you're usually just looking to repeat a request until you get the response you're looking for. If you just want to automate registration to ensure you get a seat in a course you should be looking into your school's network topology to minimize latency.
- Mister_Snuggles 9y agoThis is an interesting proof of concept. I'm impressed at how simple the SMS signup code is. For many universities though, it won't be this simple. Many hide the open seats behind a login, which means you need to be a student there to see them. Many also use student information systems that are a real pain to scrape like this. Once you get to the point where interacting with the student information system is done via AJAX requests mediated by a mess of JavaScript, it starts to push the limits of what you can do with this technique. When it gets to this point you either have to carefully dissect everything on the Network tab of your developer tools, or use a real browser. I've used PhantomJS (basically a headless WebKit) and Selenium WebDriver with great success on sites that are not amenable to scraping. The neat thing is that you really only need to use that for the interaction to get to where the information is. Once you've navigated there you can just have it dump the rendered page as HTML and parse it using the same techniques shown here.
- saamm 9y agoAuthor here--you're totally right. When we did this for our actual registration site, we spent hours inspecting requests in a proxy[0] to get to ~194 lines of PHP. 0: We used https://www.charlesproxy.com/ https://www.charlesproxy.com/
- Mister_Snuggles 9y agoYeah, that's more what I'd expect to see with a standard student information system. I can only imagine the amount of work it took - the last time I encountered a site like that that I wanted to scrape I gave up (this was before I discovered PhantomJS). It's really too bad that universities don't foster creativity like this.
- monksy 9y agoIf you were able to do that prior to taking CS101, you might not need CS101.
- vesrah 9y agoIf you want to get that piece of paper that some companies require...
- zeusk 9y agoand also immigration..
- tnecniv 9y agoAt my school, they would waive requirements if you take a higher level course in the same track. For example, if I just took CS102 that requires CS101, I wouldn't have to go back to satisfy the CS101 requirement. Obviously everywhere is different.
- JoblessWonder 9y agoHow... are you taking CS102 that requires CS101... without taking CS101? Are the requirements not actually required?
- deathanatos 9y agoIDK about the parent's school, but my at university (RPI), when I was there, students could simply opt to skip "CS101", and move to "CS102". I think it was also encouraged (but again, not required) to have taken one of the AP CS tests if you did so. I think you could also opt out of "CS102" if you desired. It was on the student to determine how much they knew. The school just understood that a number of CS students had been doing this for some time on their own, and had mastered some of the basics. I skipped the "CS101" course, which was mostly intro to programming, control flow, etc. I decided against skipping the "CS102" course; the first half wasn't that exciting (I knew how linked lists, vectors, hash tables worked) but trees got me (and some of the more exotic stuff at the end of the course), so I'm glad I took it.
- NiceGuy_Ty 9y agoI did this for one of my classes as well. But since it was just a one off, I just put this bash script in a cron job on a free-tier AWS ec2. It just did a quick and dirty parse of the html, which for my university wasn't behind a login wall. $seats="$(curl $URL | sed \"140qd\" | sed -nE \"s/<TD CLASS=\\\"$CLASS\\\">|<\\/TD>//gp\")" if [ $seats -gt 0 ]; then echo "Go register for class $URL" | msmtp -a "default" $EMAIL fi
- EE84M3i 9y agoYep. This is what I did. Also, I just used the email address for my phone's SMS. I'm not sure why you need Twilio for this, seems like over kill to me.
- thrav 9y agoDid the same thing for an apartment hunt several years ago (one particular realtor had all the best stuff going up on a personal site) and also just notified via email. This is done the way that it is by their content marketing department.
- aw3c2 9y agoIt's an ad for Twilio, of course Twilio is needed for that.
- AdamJacobMuller 9y agoThe email-to-sms gateway for AT&T was very unreliable for me.
- eggie5 9y agoI remember I made this same thing when I was in undergrad b/c my uni didn't have waitlists. So people could waitlist on my app. Their priority in the waitlist is determined by a market. I remember getting SMS's in the middle of the night alerting me that a seat opened up!
- specialist 9y ago"Their priority in the waitlist is determined by a market." I'm very surprised registration systems aren't (generally) some kind of phased auction. There's got be some research, experiments somewhere exploring different strategies. I worked on student facing software in higher ed. Admins were obsessed with fairness. Our registration systems had all sorts of arcane rules. Which were hard to explain, validate, troubleshoot. And probably neither fair or effective. Some kind of auction, where blocks of seats were made available to different populations, progressively over time, would greatly simplify the implementation and understanding. For (a greatly simplified) example, GRADUATION REQUIREMENT ABC has 200 seats. 50 seats are released every hour. First hour, seniors have first shot. Second hour, opens up for juniors and program enrollees. Third hour, sophomores. Fourth hour, freshmen. Of course, there are other factors, like multiple sections (same class offered at different times). -- We also talked about predicting demand (capacity planning). A novel solution there might moot the entire registration stampede. Perhaps a "buying club" type solution, where students state which classes they'll "buy" and roughly when. Then registrars form up sections to satisify the largest number of students. This could reduce the twin connundrums of waitlists and over capacity. Anywho. It's an interesting problem, ripe for innovative optimization, matchmaking, auction/market solutions.
- asr 9y agoI'm very surprised prereigstration systems exist at all for college level lecture courses. Personally, I prefer the system where anyone can sign up for (almost) any class and then the university hires/reassigns teaching assistants as needed: http://www.thecrimson.com/admissions/article/2013/9/10/course-selection-shopping-week/ http://www.thecrimson.com/admissions/article/2013/9/10/cours...
- agentd00nut 9y agoWondering why you didn't also automate the process of actually signing up for the seat? Getting notified that a seat is open is helpful but automatically taking the open seat sounds a lot more helpful. Though at that point you'd have a lot more luck with selenium or some other web driver.
- superdaniel 9y agoSometimes universities have weird rules about automating the actual sign up process versus automating being notified.
- TheGRS 9y agoI wondered that too until I realized it was the Twilio blog, so they would need the use case to fit their particular service.
- 0xdeadbeefbabe 9y agoWould you rather fail to register for an open seat or fail to find an open seat? Maybe I'm biased because I made the same choice as an undergrad, I stopped short of registering for the class automatically.
- XaspR8d 9y agoYeah, automating that next step is making the failure mode much worse.
- saamm 9y agoWe thought about doing that but were scared of the repercussions from the school. We felt like it was much easier to argue for notification than for automatic sign-up. Luckily, we didn't get in any trouble, so we didn't have to argue it anyway.
- ollin 9y agoI was able to get auto-registration working with mechanize a few quarters ago (similar setup to OP); I got into two classes that way! I was just using SMS/emails from UW's provided notification system to trigger the script, though, so I don't think I was actually any faster than other students-just less stressed about watching for notifications.
- ardacinar 9y agoI remember a classmate of mine doing the exact same 'hack' and distributed it to the whole school (Well, it was posted on school's major Facebook group with 10k-ish members) but it didn't have the SMS bit (It notified you via E-Mail, AFAIK).
- thaumasiotes 9y agoI don't know about you, but for me receiving an email and receiving an SMS are identical notifications.
- kernelmachine 9y agoI tried building a similar service at my university when I was in undergrad, and it led to lots of administrative conflicts, because people using my service were thought to have an unfair advantage in the registration process.
- makr99 9y agoI was in the US and want to immigrate to Canada because of US broken H1B immigration system ( for me it will take 9 years to get a greencard). Here is what I did - The immigration website of Saskatchewan province opens up randomly to apply for immigration. I missed it many times because there is no indication other than the "Apply" button being enabled and a small text in their homepage which says "Applications are now 'open' ". They will close the application intake when they have reached X number of applicants. So timing is very important. So I hacked up a script which diffs their home page every 10 minutes for "open" regex. When there is an "open" keyword in the diff, the python script calls twilio API to make a phone call to me along with an SMS. So this script was running in AWS for many weeks and one day I got the call. Logged in to Saskatchewan's immigration homepage and applied. Now I am in Canada as a permanent resident. Thanks to Twilio. edit: add H1B to make clear which type of immigration is broken IMO.
- gregorymichael 9y agoGreg from Twilio here. Congrats on your immigration! Glad we could play a small part in that journey, but credit goes to you for your courage and creativity. Would love to hear more of your story. Would you mind dropping me a note at gb@twilio.com?
- nodesocket 9y agoSounds like an amazing case study. Just looked/stalked your profile and ended up watching your talk on "Developers, entrepreneurs and depression"[1]. Really inspiring and awesome! [1] https://vimeo.com/78419167 https://vimeo.com/78419167
- fapjacks 9y agoThat's along the lines of what I do when I am waiting for a dedicated machine from OVH, since they list all of their clearance machines on a special page and you have to keep checking them. So I just used my script with Twilio to send me SMS when it's available to buy. Done that all three times I needed a box. I would not call it "hacking" though.
- imron 9y agoNow just figure out a way to sell the whole solution back to the university, and you'll have paid for your tuition fees.
- saagarjha 9y agoSounds like the quickest way to get a call from the university's IT department…
- artursapek 9y agoI've done a similar thing with Southwest Airline's check-in. They let people get seated based on the order in which they check in for their flight. So right before the 24 hour period began I would run a script that tried to check me in repeatedly until it succeeded. Usually got really good seats. EDIT: to those asking for the source, it was just a bash script that I have probably lost. The GitHub linked in this subthread looks a lot better than what I had.
- gregorymichael 9y agoGreg from Twilio here. I love Southwest[1] and have wanted to do something like this forever. Would you be up for sharing the source? Can drop me a note at gb@twilio.com. [1]: http://baugues.com/southwest http://baugues.com/southwest
- deleted 9y ago[deleted]
- necrodome 9y agoI used this last year for ~ 10 flights. https://github.com/aortbals/southwest-checkin https://github.com/aortbals/southwest-checkin
- gregorymichael 9y agoThis is great. And RUBY!! Thank you.
- elyrly 9y agoI would love to check this out as well. richard@bugcrowd.com
- IVDV 9y agoNeed to be careful about this one. A student at my alma mater was suspended for doing the same thing: https://www.reddit.com/r/ucf/comments/xo5ye/ucf_student_penalized_for_writing_program_to https://www.reddit.com/r/ucf/comments/xo5ye/ucf_student_pena... You have to love the education system...
- jjnoakes 9y agoIn your example, he was charging for the service, which was against the terms of use of the web site he was scraping. So in general, do be sure to not violate a site's terms, but I don't see what that point has to do with your final comment...
- soulchild37 9y agoI did a similar thing (free mobile app with notification + backend to scrap), gotten a cease and desist letter and threatening not to let me graduate.
- kerbalspacepro 9y agoThis blew up at all the Florida unis when it happened. I wonder how much it leaked out to the rest of the Internet.
- teddyc 9y agoI used to work in University IT and registration time is the worst. The systems never handle the load well and everyone freaks out. I think your work might anger the sysadmins b/c of the increased load.
- Mister_Snuggles 9y agoThe sad thing is that with a little bit of effort on the part of University IT, something like this could be built to put a lot less load on the system. Using this example, couldn't someone (University IT) write an SQL script to dump out the current seat count/capacity for every "current" class and throw it on a web server somewhere? Assuming that the underlying database schema isn't completely insane, that would probably be about a half-day of effort. The student writing the scraper could then point it at the output of this script. With a half-day of effort, you've reduced the load on the application, you've provided a service that the students probably want, and best of all you've got someone else to do most of the hard work.
- tomjen3 9y agoCouldn't you just let anybody sign up for $LONG_ENOUGH_PERIOD and then randomize who gets in the classes? It sounds like a much better solution in general.
- foepys 9y agoMy university switched to that for some (mostly optional) classes. This resulted in a lower attendance rates because as a student you end up registering for pretty much everything instead of one or two classes you really want. There was a system in place to fill the empty seats with students still in the waiting queue by the second session but most students already got into other classes and just didn't remove themselves from the queues. So the students got an email that a seat was free for them which they ignored and because you can only miss two sessions before failing the classes, the seats stayed mostly empty.
- aqsalose 9y agoOkay, so what about this: Students rank their course preferences in an order and submit it. For each course, free seats are first allocated to the students who ranked it as their first choice, then what is left to the students who ranked it as a second choice, and so on until the round n and there are less seats available than people who ranked the course as their nth choice: the remaining seats are allocated by a lot to them. (For a very popular course, they would have to start with the random selection right away on the round n=1). However, instead of enrolling the lucky students right away, everyone gets notified about which classes they have been tentatively admitted, and they must confirm their attendance before they can take their seat. If they don't confirm in a timely manner, they lose the opportunity and the free seats are propagated to students left in the waiting list until someone accepts (again using random lot). The one problem I can think of: the process (especially the propagation of rejected offers in the waiting list) may be too long-winded to be practical. So maybe skip the confirmation part: there's a set maximum number of courses you can take, M, and slots are filled until you're enrolled to maximum of M courses. But in any case, if the overbooking is enough of a problem so that you are looking at methods like these, the real problem is that have too many students compared to the teaching resources. All other attempts can only mitigate the symptoms of the root cause. So maybe staff should choose limit the amount of students eligible to take some very popular courses by an entrance exam, or imposing certain mandatory minimal GPA requirement in the prerequisite courses. This option should be especially considered if the attrition rate is high (there's a problem of students taking much-coveted seats in the class but who fail to show up or otherwise don't put in enough effort to pass). Usually that would have been done on the admission level already, though.
- noer 9y agoThis is pretty cool. I did something similar with adwords scripts a few years ago. I was doing facebook advertising a few years ago and the ads featured products that would occasionally sell out, but there wasn't a way to monitor them so if something sold out, the ad would go to an empty product page. I rigged up a system that would take a list of URLs, look for "sold out" on the content of the page and then send me a text if it found a sold out product. I used Adwords scripts because it was able to run jobs hourly and all I really knew at the time was JS. It was a fun little project.
- kevsully5 9y agoI did a similar thing my freshman year of college, and by junior year I created an API based around the class registration system and an iPhone/Android app for it that sends push notifications to students when the class they want opens up (shameless plug: http://www.eaglescribe.com/ http://www.eaglescribe.com/). This past semester, the active users was between 1/3-1/2 of the undergrad students in the school. As you can imagine, popular classes had dozens of people "subscribed" to receive notifications when it opened, so it became a race to sign up once the push went out. On the plus side, this gave us a treasure trove of data on the most popular courses, and we've been in communication with the school to see if they would be interested in this data.
- zaidf 9y agoI built a similar service when I was in school at UNC. Unfortunately, the IT dept blocked my server IP and asked me to shut down the service after it got press (and lots of usage.)
- jorblumesea 9y agoIs there a generic page scraper library or toolset? I can see a ton of demand here and it seems like something you could use on your day to day. Things like this actually seem like an outlier, I'd imagine airfare would be a huge one.
- saamm 9y agoThe nearest thing I can think of is Scrapy (https://scrapy.org/ https://scrapy.org/). For airfare, I think there's already a few players in the "this flight got cheaper" space (Kayak and Hopper come to mind), but they don't have data on some airlines (because of the airlines' ToS IIRC).
- Paul-ish 9y agoSomeone started charging for this service near the end of my time at University of Washington. I think the university implemented their own notification system so that class registration didn't become pay to win.
- noonespecial 9y agoFor the curious, if you just want to one-off this for yourself, you probably don't need twilio to do it. Most carriers have a special domain that if you send email with the recipient's phone number, it turns into a text. I've got Verizon and its as simple as this from the command line: echo 'Text message text' | mail -s 'Subject will display in braces in the message' -a 'From: Myserver@Mydomain.com' 'number@vtext.com' This ended up working faster than Twilio for me. (Less than 10 seconds from enter key to text on phone).
- jaboutboul 9y agoYour number is now on hacker news!
- noonespecial 9y agoJust saw that. Yowzer... damn clumsy of me! (Thankfully a published work number but still!)
- taco_emoji 9y agoYep, they're called SMS gateways. I don't think there's a single canonical source for them, unfortunately, but I keep this file bookmarked for when I need to look it up: https://github.com/cubiclesoft/email_sms_mms_gateways/blob/master/sms_mms_gateways.txt https://github.com/cubiclesoft/email_sms_mms_gateways/blob/m...
- rocqua 9y agoIs the SMS free? If not, who pays for it?
- JosephRedfern 9y agoI've written a similar thing for the Dell Outlet: https://outletmeknow.redfern.me/ https://outletmeknow.redfern.me/. SMS notification isn't actually working at the moment as I've run out of Twilio credit, but Email should be fine.
- mrpippy 9y agoI did the same thing at UCSB 10 years ago to constantly poll for seats opening up in a class. This was back when GOLD was a primitive web interface that was clearly screen-scraping an IBM AS/400 terminal app. I used python and twill--put values into forms (no JavaScript and the form IDs never changed), login, search for the class, and scrape.
- WaxProlix 9y agoSame at UCSC, I even considered trying to monetize the script given what an atrocious system it was.
- nsb1 9y agoI'm dating myself, but when I went, it was a standard phone AVR, with touch tone menus. I used a war dialer to get into all my classes. Ahh, the good ole days.
- clamprecht 9y agoWhich wardialer?
- dalacv 9y agoCool, I'm dating myself too. We should double-date
- nsb1 9y agoI'm dating myself, but when I went, it was a standard phone AVR, with touch tone menus. I used a war dialer to get into all my classes. Ahh, the good ole days.
- sdca 9y agoTwilio is great and all but an SMS or other notification doesn't actually guarantee you a class that's available to all students. A class was sniped from me one time while I was still on the confirmation screen. So unlike OP I fully automated the process. EDIT: A notification & response is good for solving a captcha if they have one, if you don't want to outsource that to mechanical turk (or trust their timing / accuracy).
- ArlenBales 9y agoIf you're using Python, please don't reinvent the wheel and write your own scraper. Use Scrapy (https://scrapy.org/ https://scrapy.org/). It's one of the most tested and powerful scrapers out there.
- raybb 9y agoFor a use case like in the article would it make more sense to use beautifulsoup or is there an advantage to scrapy?
- assafmo 9y agoCool! But I'd use a Telegram bot for free. :-)
- j45 9y agoOdd that this is still an issue 20 years after the first time I heard about this. This reminds me of the story of beartracks, and how it was shut down by the University of Alberta to make everyone register by phone .. and eventually was brought back by the university. Tried to find a link I had read about the story, but can't seem to find it.
- pszczurko 9y agoThere exists something like this for Rutgers University. Anyone can submit a course to be tracked along with email, and once a course pops up as open, an email is sent. Text messages are not used since they are not free and this project is maintained by students, not university. But it can be easily modified and self-hosted since the source is on Github: https://github.com/v/sniper https://github.com/v/sniper
- scottydelta 9y agofor sending yourself updates when a bug is encountered or a condition is met, isn't it better to use something like PushBullet instead of normal text messages?
- deleted 9y ago[deleted]
- ChrisCinelli 9y agohttps://github.com/ChrisCinelli/scraperNotifier https://github.com/ChrisCinelli/scraperNotifier - This can check if regex exist or do not exist. It send an email or a message on Slack using a hook. I used successfully with Eventbrite for hackthons sign up.
- ptr_void 9y agoA lot of things here smells like pure advertisement. hmm...
- soulchild37 9y agoI did a similar thing like this back in university, I made it a mobile app with notification and passed around. It got popular and I received cease and desist from my university shortly afterwards, now the system has a captcha on every page haha.
- tahabi 9y agoSome time ago Berkeley switched from Telebears to a new registration system developed by PeopleSoft/Oracle. Needless to say, it's garbage. I wrote a couple scripts to pull class data from the API in an effort to help me get into our heavily oversubscribed CS courses by searching for open sections, but the system seems to be designed to make using it as painful and worthless as possible. Luckily this semester the EECS department has put dummy sections in for all classes so students who are able to register for a lecture don't get permanently stuck on the waiting list because of a full section. College administrators who are considering Oracle for your registration needs, please consider anyone else instead, for everyone's sake.
- Justin_K 9y agoCan we stop calling screen scraping scripts a hack?
- cja 9y agoWhy not just send yourself an email? Much simpler than using Twilio. Your smartphone will alert you either way.