14 ms·
Frustration project: Automate data entry into PeopleSoft with Selenium
- edoceo 6y agoWebdriver is for more than just testing! Very cool demonstration. My biggest headache was in getting the right selectors for the various form fields - some services have shifting IDs and classes (ugh xpath)
- bickeringyokel 6y agoWhen you start using xpath that's 20 parents deep and every line of code scrolls off the page you quickly realize how important it is for your devs to to write good accessible and unique element IDs.
- lifeisstillgood 6y agoHa! I see your crappy UI in PeopleSoft, with hard to automate edge cases, and I raise you multiple SharePoint instances! Ante up at your peril in this game :/)
- bash-j 6y agoI've managed to automate SharePoint data uploads and downloads using the SOAP API.
- lifeisstillgood 6y agoI have too, and the OP got Peoplesoft to work. But ... my goodness the effort compared to a proper jason APInis huge and compared to just web scraping HTML. I mean there is a word file someone just uploaded. That's three clicks
- beatthatflight 6y agoLove it. I've built a flight search tool in selenium too, and for isolating deals. (I've also used selenium for testing). The biggest problem is the speed and the unexpected (the spinner you experienced, intermittent page load times, asynchronous requests or when elements are stale as a result). But it is possible to do it. Next you can thread the sucker, so you could have 2-4 PS windows open and enter at a faster speed ;)
- tbensky 6y agoYes, multi-threads of it...that would be scary!
- julianeon 6y agoI'm always interested in clever use cases for Selenium. It seems like "automating boring stuff using Selenium" should be a bigger category than it is - and use cases that can do it profitably are always fascinating.
- sleet 6y agoRPA (Robotic Process Automation) is a big category of tools that can be summarised as ‘automating boring stuff’. You often see it used for repetitive data entry, e.g. 1. Load excel sheet from X:\... 2. Login to Y 3. For each row navigate to ... then enter data 4. Submit, if there’s a validation error add it to and excel workbook 5. Email exception report to ...
- halo 6y agoWhat software is generally used for this? Is there anything available for the average person that doesn’t require spending a small fortune?
- janstice 6y agoUipath and automation anywhere seem to be big players with community/free starter editions. PWC (who are spearheading the RPA at $work) seem to have decreed that all their staff are going to use uipath worldwide. There are a handful of open source RPA tools, but I don’t think there’s a breakthrough project yet.
- mahalol 6y agoAzure LogicApps are good for these kind of silly integrations, if it has a connector for it then it's a piece of cake. They have a pretty limited use case though.
- sbierwagen 6y agoThe fact that you have to update Chromedriver with every Chrome release now doesn't make things easier.
- dfee 6y agoThere is an entire class of business problems that can be solved using Selenium. I don’t love that raw http connections are the universal API, but at least it’s a tool in our belts when these companies provide lackluster or zero proper interfaces.
- sqrt17 6y agoThere's a whole class of solutions beyond Selenium that automates not just web apps but all apps. It's going under the name of "Robotic Process Automation" and people really like it because they're not dependent on their software vendor to get basic integrations done.
- jimmyswimmy 6y agoI use selenium similarly to drive a similarly awful interface known as costpoint. I would DEARLY love to know how to use the browser developer tools to figure out what is actually happening under the hood of websites without having to read every single Javascript module. The network tab shows you what is sent and received but it's not easy to figure out how those messages are created and parsed. It would be nice to learn how to debug and step through some of that. If I could learn the underlying api, I could write a tool that would never change. The ui changes much more frequently than the interface between client and server code. Besides, it would be much faster this way.
- PetahNZ 6y agoIf you click on the "initiator" column in the network tab it will jump to the code that created the request. Can also see stack traces, set breakpoints, etc.
- mistermann 6y agoIf anyone happens to know of a particularly good article on this for non-javascript (but technical) people, please post.
- sk5t 6y agoPlease clarify--are you looking to understand how browser events work without learning about the DOM and/or Javascript?
- mistermann 6y agoI suppose something along the lines of Chrome debugging for programmers who aren't javascript developers...undoubtedly there are some fundamentals you would have to become knowledgeable in, as with any technology.
- jimmyswimmy 6y agoAmazing - how did I miss these? Thanks!
- hugs 6y ago#FunFact: I started the Selenium project. My first job out of college was at PeopleSoft. I left PeopleSoft and went "in-house" as the PeopleSoft admin at ThoughtWorks. A few years later, Selenium came out as a side-project for another internal system we needed. But when we were rolling out Selenium and dreaming up "world domination plans", one of my mine was to get a copy of Selenium to ship with every copy of PeopleSoft (supplanting the free copy of SQA Robot that PeopleSoft shipped by default). Anyway, I never really pursued that particular "plan", but here we are.
- tbensky 6y agoTHANK YOU for giving us Selenium. When I first ran my script and saw the browser pop up start "running on it's own," I was pretty wowed.
- j45 6y agoThank you for starting selenium. It has helped me in so many ways over the years.
- claudeganon 6y agoSelenium is amazing and saved my sanity when I was fresh out of college in a boring office job with terrible software. Thank you for it!
- jon_elbrook 6y ago> Selenium came out as a side-project for another internal system we needed. What kind of system? I had thought that selenium was created to TDD web development front-end.
- trhway 6y agoadd AI on top of that which learns what/where/how to enter (or at least suggests) and you get the modern office automation wave.
- stm17 6y agoI would love to hear any more you have to say about the genetic algorithms you use to help with scheduling.
- tbensky 6y agoAh yes, the genetric algorithm (GA). Well, in short, scheduling personnel is a very visible, policital, and important job. It affects people's lives, and has to be done farily and correctly. I did my best (by hand) for a few years (scheduling 50ish professors into 200 class sections), and I knew I was doing a bad job, but had no choice. It would take 2 weeks to do, because I could only mentally handle working on such a job for so much in a given day before I had to set it aside. So I came up with a preference form, that each could fill out dealing with things like preferred time of the day to teach, daily schedule compactness, preferred class levels, etc. These preferences could also be weighted (0..1) by importance to each person. So the GA starts out with a long, two-part inital random sequence that consists IDs to all professors followed by IDs to all possible classes. Using crossover and mutation (the usual GA steps), the order of the professor IDs vs the order of the classes (prof 0 gets class, prof 1 gets class 1, etc.) are shuffled by the GA. (There are a lot of strange constraints to this you'd only see in academia, related to pay, and contractual teaching requirements.) A fitness is calculated for each professor's latest schedule, by computing a weighted sum of their preferences, based on the assigned classes pulled from the long GA sequence. It's incredible how well it works. It is done in under an hour, people seem happy, everyone's schedule fitness is minimzed, and each's schedule fitness is explained to them (i.e. you got +3 points because you wanted an 8am class, and got one). Everyone can also see everyone else's fitness so it's a "share the pain" kind of resulting schedule. At times when we go in to tweak things by hand, it is clear how tightly the GA optimized things, as any human intervention we try usually and obviously makes things worse. Bottom line, the GA generates the CSV, which Selenium types into PS. After using the GA, I'm convinced that neither humans (or spreadsheets) should be invovled in scheduling at all. I also think GAs are also pretty incredible.
- mleonard 6y agoInterest! Could you put the genetic algorithm code on GitHub? Would love to learn from it :)
- oandrei 6y agoSelenium leaves trace in HTTP requests, so its use is quite detectable. The authors of PeopleSoft can include Selenium detection in their products, disallowing your automation scripts. Why not use GreaseMonkey instead?
- 1024core 6y agoDo you have any more information about this trace in HTTP requests? My understanding was that it was the _browser_ doing the requests; Selenium was just driving the browser.
- unreal6 6y ago"Peoplesoft is a curse on humanity" I would love to know the story of how this wretched piece of software seems to have embedded itself into nearly every college and university in the United States.
- derwiki 6y agoBelieve it or not, Peoplesoft was way better than what it replaced at my school.
- xyst 6y agoNo surprise that it is backed by Oracle. The sales people at Oracle are really good at getting technology inept people/businesses into multi year contracts. Once they started citing the list of features in their hundred slide presentation, the administrator stood no chance.
- tallanvor 6y agoPeopleSoft was one of the larger players, at least in education, well before Oracle bought them. There's really (or at least there was when I was involved in higher education) only 3 main players at that level. And trust me, each option you had was bad in its own way. The problem is that these products started out much smaller, and grew to fill the needs, although not necessarily as nicely as you might like. So now when it's time to put out a new bid, there's just the handful of vendors who can check off all the boxes for what the current system does. And integration with other systems is always a pain - even today it often includes csv or tab-delimited files that get sent to ftp sites for batch processing!
- patrickk 6y agoThe 4 ways to spend money[1], in this case, spending someone else's money on someone else. Likely a procurement person with no IT knowledge, who never has to deal with the painful solution, but got wined and dined by the salesman. [1] https://investorjunkie.com/economics/milton-friedman-ways-spend-money/ https://investorjunkie.com/economics/milton-friedman-ways-sp...
- 6y ago
- kfk 6y agoAny tools to do this with native applications too not just browsers? We have been flooded by this RPA companies the past 2 years and I'd love a way to do this with Python. It is actually a potentially very profitable market since the impact of automation is very visible (people see cursors do stuff for them...).
- cranekam 6y agoAnother +1 for Selenium. I used it to bag tickets for a show I wanted to see. The seller’s site was terrible and apparently had no protection against ticket spinners so all 5,000 tickets sold out in minutes. Fortunately their resale site also had no protection at all so it was easy to write something to reload the page often and put tickets into my shopping cart as soon as they became available. When that happened it notified me and handed back control for me to enter the credit card details (I was too lazy to automate it all). I’m not proud of this brute force approach but if the reseller’s site wasn’t so awful to start with it wouldn’t be necessary.
- devit 6y agoIMHO it's usually better to write a WebExtension instead of using Selenium, since you have more control over the browser, can intercept and modify network requests, can access the DOM from JavaScript while being protected from the page altering the environment, and can ship the code as a browser add-on, while still being able to communicate with native apps. Only downside is that you can't drive old IE and mobile browsers except for Firefox for Android.