12 ms·
Learning to code kept me sane when I was a diplomat
- SocksCanClose 12y agoThis is a fantastic article and illustrates a HUGE problem in government systems, where data gets aggregated, but not associated.
- Retra 12y agoI had a government job and my boss loved me for doing automation stuff. All of the documentation and data requirements seemed to be written as though a person had to manually do everything -- like they expected a human brain to be making decisions at every step of the process. It just reeks of a lack of care not only for the power of automation systems, but disrespect for the people who have to do that work. Design your processes in such a way that easy stuff can be done easily and important stuff can be done by people who have some authority. This is my biggest (personal) problem with bureaucracy. "You have to do it this way because it's the only way _I_ know how to do it, and I don't care if you know a better way, because you don't get paid enough to have good ideas."
- cportela 12y agoYea my university IT job was like that too. Except automating things was part of my job description and I still wasn't given much lee-way to automate things much.
- cportela 12y agoYea my university IT job was like that too. Except automating things was part of my job description and I still wasn't given much lee-way to automate things much.
- yaddayadda 12y agoA few years ago I had a brain-numbingly monotonous "navigate here" then "click here" then "record output here" type job. Unfortunately, mine was on a military base and the "record output here" was an Access database. I tried to automate the whole process in Access, but military restrictions prevented me from automating most of the process.
- elwin 12y agoThe article shows an interesting illustration of the disconnect between programmers and normal computer users. Like the author says at the beginning, any experienced developer knows that automating interaction with a GUI is a crazy idea. It's a painful, fragile way to move data around. But to the author and anyone who's done data entry, it's an improvement to their workday that demonstrates the power of programming. So much programming instruction seems designed to teach people who already understand programming how to solve problems that nonprogrammers wouldn't even understand. A tutorial that started with automating tasks like this might be more effective than all the ones that start by demonstrating how to do basic arithmetic in $LANGUAGE and go on to a detailed explanation of $LANGUAGE's object system.
- yason 12y agoI know several places where software components or systems software is tested against real-world applications, games, and whatnot, and... guess how it's often done? Yup, with a combination of injecting artificial keyboard/mouse inputs and looking things at the screen. And these guys aren't beginners but professionals. You can't really automate something that doesn't have the APIs for it.
- deleted 12y ago[deleted]
- lstamour 12y agoThis is why I like http://www.sikuli.org/ http://www.sikuli.org/ and Mechanize-style DSLs so much. They translate programming, web requests and parsing/processing into plain english "click [here]" concepts.
- beachstartup 12y ago> all the ones that start by demonstrating how to do basic arithmetic in $LANGUAGE and go on to a detailed explanation of $LANGUAGE's object system. the fact that i knew exactly what you're talking about as soon as i read it is pretty good evidence that you're right. the best tutorials i've ever seen are ones that start with a moderately complex program and step through it line by line, explaining in plain english why things are being done. this is how both human beings and computers parse programs, so i'm not sure why a tutorial writer would do it any other way! i think the current trend of embedding documentation within your program as markdown on github is a pretty close approximation of that.
- alexbecker 12y ago> I decided to write a program that clicked on things. I cringed a little. I'm glad he's learned enough to know now that this is not the right approach.
- vikp 12y agoI thought this way before, and looked down on what I had made, but I'm slowly coming around to the idea that its more about how well the approach works and what it teaches you than about the technical details. I definitely wouldn't code it the same way today, but it worked well and taught me a lot. I wouldn't stop anyone who's in a similar position from doing the same.
- bluedino 12y agoThe worst that could have happened was that you talked yourself out of automating it because there wasn't an 'easy way'
- vxNsr 12y agoI also have to wonder if you even could have used a api-approach, especially for that second db which was it's own program and was likely monkeyed together by interns over a summer.
- dnautics 12y agoI cringed too, but then I realized this is probably the best way to start out. It's only a 'problem' because it's a bit more glue code (but less 'figuring things out' time). The only truly cringey part is not knowing about CSV!
- stefs 12y agothat said, CSV is not without its own problems - i mean, it's not well defined. if you're down to automate a bunch of windows programs by moving around the cursor programatically and faking keyboard events, using excel over ODBC is far from cringey.
- 12y ago
- flashgordon 12y agoBrings up old memories reading this. I remember my first job about 20 years ago at NEC as a data entry operator. It was 8 hours each day of taking a bunch of mal-formatted CSV files from some reports the previous day and importing it into excel and drawing graphs. I was just getting into learning C at the time and ended up writing an importer (parsing and formatting in C and ODBC to write back to excel along with excel macros to draw graphs). Very hacky and ugly but got the job done. It got the 8 hour job done in about 30 minutes - most of which was detecting patterns in the input and adding rules for those, which was fun in itself.
- mschuster91 12y agoThe biggest problem is not the lack of code and automation. The biggest problem is that apparently Guyana allows the US embassy to search their criminal records system: > Search a different database with a separate program (“DB2”) for criminal and other history So either Guyana allows the US wholesale access to its criminal records system or the US systems gather this data more or less illegally from somewhere. edit: downvotes? seriously?
- maxlybbert 12y agoI have no idea what DB2 was. It's my understanding that every country in the world wants to know the criminal history of anyone asking for a visa for any reason (the host country may ignore that history, if it believes the crimes were politically motivated). And it's my understanding that INTERPOL allows countries to get at least some criminal records information. And there is clearly some system in place for countries to make extradition requests -- which would require some criminal history information. So it's not clear to me that the US must have wholesale access to Guyana's criminal records system, or that it must be illegally gathering data to populate DB2. Even without INTERPOL, I would expect countries to have treaties to share some criminal records information. If Guyana is trying to capture a criminal, clearly it's in Guyana's interest to ask countries to help prevent that criminal from fleeing.
- mschuster91 12y ago> It's my understanding that every country in the world wants to know the criminal history of anyone asking for a visa for any reason I'm actually fine with that. And if I ever apply for a visa, I gladly sign a waiver allowing the respective country to request the data from my country. But I am not fine with any government in the world being able to research my criminal record without authorization! Push, not pull. > If Guyana is trying to capture a criminal, clearly it's in Guyana's interest to ask countries to help prevent that criminal from fleeing. There's a system for this: take away the criminal's passport and don't let him out of the country. That's a job of regular Guyana border police. edit: oh, and who guarantees that (e.g.) the US respects expiration dates? In Germany, records have to be entirely wiped after certain dates (5-20 years, http://www.gesetze-im-internet.de/bzrg/__46.html http://www.gesetze-im-internet.de/bzrg/__46.html) - but I highly doubt that the US respect this and just keep the data instead.
- jdbernard 12y agoBefore you knock on "writing code that clicks things" remember that on Windows clicking things is the primary if not solitary interface to most of the system. Even the administrative interfaces are secondary to the visual interface. A lot of programs don't offer an API available to outside processes. The most important thing about any program is correctness: does it work and do what it is supposed to do. The only thing cringe-worthy about having to script GUI interactions is that fact that the operating system and environment requires it.
- detaro 12y agoFor the server versions, Windows finally leaves this path, making PowerShell (that admittedly also takes some getting used to) a first class configuration system. You even can do something in the GUI and then check in a log what PowerShell commands were executed under the hood.
- pjc50 12y agoFrom which version of Server? Where does the log go? I have a use for this.
- detaro 12y agoI think it's from 2012? Here is a link that describes it: http://www.learnmsexchange.com/windows-servers/windows-server-2012/593-what-is-windows-powershell-history-viewer-in-windows-server-2012 http://www.learnmsexchange.com/windows-servers/windows-serve...
- sinwave 12y agoOof. Had a hilariously similar experience doing data entry at my first internship, which was at a financial services firm. I knew nothing about programming, but I had taken an intro stats class that had a few assignments in R. A super simple R script did half of my day's work in seconds (plus some debugging because of course I did next to no error handling). In retrospect, I've realized that if I knew a thing of two about regular expressions at the time, I would have been able to automate my whole summer internship. So it goes.
- techtalsky 12y agoI haven't seen much discussion of a more ideal way to solve this problem (to say: as a one-off like the author was doing, not reusable software), so I thought I'd just think about how I'd do something similar. Luckily there are libraries that are great at interacting with a web page or multiple pages at the same time without needing to actually have a browser instance open. There's selenium and its various scripting environments, which are meant for browser testing but work just fine for many automation tasks: http://www.seleniumhq.org/ http://www.seleniumhq.org/ And, newer, are headless technologies like phantom.js: http://phantomjs.org/ http://phantomjs.org/ and Dalek.js: http://dalekjs.com/ http://dalekjs.com/ This is SO much easier than using Javascript to find purple links on a page, and so much more reliable. He could have looked up each link on DB1, then used the data to look up the DB2 information. In any case, what he did saved him hours and hours of time and started him on a lifelong interest in programming. Also, obligatory relevant XKCD's: http://xkcd.com/1319/ http://xkcd.com/1319/ & http://xkcd.com/1205/ http://xkcd.com/1205/
- Jezparov 12y agoThis is a really great post. For me, the recurring theme is knowing what's possible. From a class he took in high school, Vik knew that some sort of automation was possible. That spurred some reading and exploration, which led to some success. Success led to thinking about other opportunities for automation and more exploration of what's possible, which in turn led to more success. To me, it absolutely doesn't matter that things could have been done in other ways. Nothing to cringe about here. A developer can only use approaches that he/she knows or suspects are possible. In the day-to-day work of a developer, knowing what's possible is perhaps the most important thing. Anything else can be googled.
- seanmcdirmid 12y agoEqually important to knowing what is possible is to not know what is considered impossible, or at least considered very hard. Automation is often considered futile, and knowing that can be intimidating. We are often limited by what we think is impossible, but really isn't. It is said that most Nobel prizes are earned for research done before 30, since any older and they would know the breakthrough they are trying to accomplish is impossible :)
- stefs 12y agothis reminds me of when i started to code - find the first solution to the problem that comes to mind and hack it until it works, even if was something as abstruse as manipulating the UI. of course the outcomes were incredibly fragile and broke at the first possible moment, but in the end they still did the job they were supposed to do most of the time. today i couldn't do that anymore; i've grown too uptight to try any of those inefficient "solutions". i guess i'd rather give up and admit defeat if there were no APIs or passable hacks (i'd say using phantom.js would be a passable hack) i could work with than solve the problems this way. i'm usually unable to follow the advice given by the xkcd automatisation timetable described here: http://xkcd.com/1205/ http://xkcd.com/1205/ "is it worth the time?" - if a dirty hack is destined to run more often than once, i just have to make it passably elegant, readable and efficient. i'm sure nine out of ten times the effort is pretty much wasted. i guess i've just grown old.
- thebenedict 12y agoLove it. I started my career in International Development and one of my team's first projects was figuring out how many miles our organization's 1000+ staff flew in a year (why is a separate story). The travel agent could only provide records in PDF. 1000's of them. The intern I shared an office with was assigned to go through each record, type the origin and destination airport codes into an online flight distance calculator, and enter the results in a spreadsheet. After two days of watching this I volunteered to take over and spent a few nights learning enough Java to read/write CSV and make the HTTP request. Saved days of pain for the intern and was a satisfying accomplishment as a novice.
- eru 12y agoAnd you even learned some Java in the process!
- yason 12y agoThis is golden. I've wondered more often than once myself what would it be like to have a "normal" office job. For instance, take some city department or some big firm where things generally flow at a slow pace and require a lot of time spent on work items, doing possibly repeated tasks each week. What would a programmer do? Exactly as described in the article: - learn what kind of tasks the job consists of - identify repeated actions and patterns in work - think of a way to automate the easiest parts, still requiring human to finalise the task - gradually learning more and automating the harder and harder parts and building tools bottom-up based on earlier blocks - finally ending up with a software mechanisation puppet that I can instruct and program to do my work - gradually spend first some of my time doing the automation and as it would slowly take over I could devote more and more of my time automating the rest, and when all is done, I would feel the job is "completed" and get bored. This is something that feels so natural to a programmer: I have dozens of little tools to aid in doing my programming job at work. The goal of a programmer is to program anything that can be automated and only work himself on the hard tasks that can't. It's intriguing to imagine how you could apply those skills outside programming jobs.
- eru 12y agoOften you have to change the tasks slightly to make them properly automatable. The fancy term is process improvement. See https://en.wikipedia.org/wiki/Business_process_improvement https://en.wikipedia.org/wiki/Business_process_improvement
- mjklin 12y agoWhy has no one mentioned Autohotkey? (http://www.autohotkey.com http://www.autohotkey.com) It was practically made for this use case, and has an active community behind it. It even has its own script editor (SciTE4AutoHotkey) and was recommended as a Kevin Kelly-approved Cool Tool (http://kk.org/cooltools/archives/4147 http://kk.org/cooltools/archives/4147). My go-to program for my desk job.