16 ms·

Ask HN: How did you automate your day job?

96 points by amerf1 7y ago

Curious to hear stories of people who automated their 9 to 5 jobs

FearNotDaniel 7y ago
I actually got my first dev job by doing exactly that. While I was searching for a new career and vaguely interested in something to do with IT, I was doing short-term temp work as an office admin. One was in a startup - not tech-related - where I spent most of my day searching for customers' names and addresses in a spreadsheet, then copy-pasting them into a bunch of Word templates and printing them out. I already knew about mail merge, but in this job the particular set of docs to be generated would be different according to the specific circumstances of each individual case, along with one or two customer-specific paragraphs that came in an email and had to be pasted into the document. Having been a hobby programmer as a teenager, I had a hunch this could be made more efficient. Once I discovered VBA (what? Excel has its very own programming language BUILT IN?!) I quickly threw together some macros and an Excel form that generated the whole personalised document set in a few clicks (what? you can write code in Excel to control Word? This is AWESOME!) At the time I was sharing a house with a couple of CS students and one of them let me borrow a very academic textbook about this wacky thing called SQL. I fell in love with the idea of tables and joins and stored procs, skipped over all the difficult maths parts, and discovered back at work that the funny icon next to Word and Excel called "Access" was actually a SQL database. Feeling like I had all the power of the universe at my fingertips, I hacked together a custom CRM system which pretty soon the whole office was using, and pretty soon it started grinding to a halt. Round about then the company hired its first IT Manager, he liked what I had done and said: maybe you should check out this new thing called SQL Server 2000, do you want to be our Database Developer? So I shifted the back end over to SQL then discovered .NET 1.1 and started writing better front ends and hired a small team to take it over. So yeah, I kind of fell into business app development by accident just by automating my very repetitive office admin job. That was 15 years ago, and I'm still getting paid to automate other people's 9 to 5 jobs.
- shyn3 7y ago
  You should thank that IT Manager today. Most of them would just ask you to get approval to use a database on your own PC at this point.
davchana 7y ago
Not a day job, but while looking for a day job, I came across a process, where a state run department has multiple openings; you upload your resume things like name, details, previous jobs, address etc to a standardized form, & then at job search page, just click the listing, review the text, click Apply, attach your standardized auto generated resume by click of a button, agree to their t&c with a checkbox, & click Apply, & note down the information from thank you page about job number, city, position, department, type etc. All of these 6-7 pages had unique URL, & all html was nicely generated with proper css id class names etc. After spending about 5 Sundays doing this manually, I quickly wrote a JavaScript bookmarklet where it gets the current URL, & switch case calls the required function. Each page had different function to look for element, focus, highlight for me, & click it. The last page all required information was loaded & sent to a Google script for spreadsheet.
c22 7y ago
My first job was doing restock for a small independent retailer. The only report we could print showing inventory was inclusive of all the items we had ever carried, regardless of current status. The software was very limited and there was not much in the way of filtering. The best you could do was break the report up into ranges by vendor. So my job was to go through the alphabet printing reports (e.g. "every item from a vendor that starts with 'A'"). I'd take this report, which for some letters was over a hundred pages long (about 20 items per page) then, using a highlighter and a ballpoint pen I'd go through the report highlighting the items that had stock available in the "backstock" column and had stock in the "sales floor" column that was lesser than the "restock level". I also had to compute the restock amount by subtracting the items on the floor from the restock level and taking the max of that number or the available backstock. I'd write this number in the margin of the report, then I'd travel to the backstock area, pick the highlighted items, and restock them onto the sales floor. After this was done I'd move on to the next letter. The goal was to cycle through the alphabet at least once a week and during the off-peak season this worked fine, but during the busy season, even with extra help, this system was showing signs of stress. The first thing I did was to write a small perl script which I placed at the path the POS software was expecting to find a line printer. My script took the print-formatted report as input and simply stripped out the lines with no available backstock before forwarding the remaining lines to the printer. This alone reduced the number of pages printed out by an order of magnitude in most cases (from ~100 pages to ~10). At first I would copy this script to the printer path, run the report, then re-link the printer, but later I modified the script to recognize that specific report and pass-through other print jobs unmolested. Over the next three years or so I added onto the script a lot. First I made it compute the restock amount for me, then I had it start skipping lines where the restock was trivial (e.g. restock amount was 1 and there were still >10 on the floor). Eventually the script was maintaining its own parallel inventory database, tracking rate of sale for each item and only suggesting restock for things that had very low inventory on the floor or were selling very quickly. I got to the point where I could restock the entire alphabet every morning and have time leftover in the afternoons to help with receiving. By the time I left I had a dedicated terminal set up in the back during the busy season that ran my scripts every 10 minutes and displayed in near real-time items that required critical restock throughout the day. I heard a few years after I left they had switched back to the old methodology because they couldn't get their POS vendor to support my setup.
geocrasher 7y ago
I don't automate my job specifically (I'm a tech trainer, currently) but when I have optimized myself heavily using AutoHotKey. I spend most of my time these days between Slack and a bash prompt, and so I have created macros for myself. I lean on it pretty hard, and have cut down the amount of typing I do simply because I can now type "yyy" to get "Yup." and other silly optimizations that save me time and effort, mostly cumulatively. There are few big optimizations to be gained this way, but they add up. I also have entire phrases and paragraphs macro'd for the same reasons, as well as common typos. It doesn't sound like much but these things save me a lot of time.
s09dfhks 7y ago
Haven't totally automated my entire job but a small portion of what I do is configuring a fresh aws account some core services (SMTP, NTP, LDAP etc etc) and the associated VPC configurations before handing it off to the devs. Wrote a pretty simple python script that takes some inputs, cidr range, account name, jira ticket etc, and sets up all the terraform, ansible groupvars, ssh key generation, secret storing and a few version control check-ins. Reduced the footprint from a few hours to literally less than one minute.
audiometry 7y ago
I am a trader. A million different factors drive commodity prices, and their significance ebbs and flows over time in unpredictable ways. So typically you'll have a spreadsheet that monitors one of these occasional factors. You run it 98 days in a row and it's irrelevant. This lulls you into complacency on the 99th day, where you don't look at it, and on the 100th day suddenly it becomes relevant again and you didn't notice it in time because you hadn't checked the spreadsheet. So my agenda has been to move the analysis corpus into Python, directly talking to Bloomberg and other data sources. Then these things run automatically and can flag me when they're significant. Rather than trusting me to check scores of spreadsheets and wading through endless True Negative results. When I started working, Excel was the powertool that allowed a junior to outperform dinosaurs using calculators, and make an outsize impact. My feeling now is that Python and similar tools are the equivalent of my Excel 20+ years ago, and Excel is the 'desk calculator'.
- inakarmacoma 7y ago
  Would you be open to sharing more? What sort of things are you doing in Python that are new and unique, distinct from capabilities you had in spreadsheets?
  - dbancajas 7y ago
    isn't python used for automating? I guess excel can do autochecking if you really pushed it
  - audiometry 7y ago
    What things are new/unique? Hmmm, I'll bet someone could implement almost all of what I do inside excel sheets one way or another, but it would just be super-cumbersome to do. One big example is regression. All Excel really does is simple linear regression. Add more variables you get a better r2'd and that's it. Comparatively, with scikit it's trivial to avoid all horrid overfitting problems that come with simple regressions in excel. I use of lasso-cv model, and better practices like cross-validation, etc. Another big improvement is I replaced Excel -> Tableau visualizations pipeline with using Altair-Viz. Altair makes almost-Tableau-like charts pretty easily and cuts out lots of intermediary steps -- jump right from dataframe to a chart. It's much easier to add sanity/safety assertion checks in python than in Excel. Everyone knows examples of excel spreadsheets that are subtly poisoned. I took someone else's technical charting ideas and reimplemented them in python, which then just alerts me for key events (changes in trend, potential exciting issues, etc). Now I don't have to look through the data myself, which is a chore, and which is susceptible to my mood. It just spits out its objective 'truth'. One thing about (at least commodities trading) that outsiders might not realize is -- none of the sexy FAANG big data stuff is particularly useful or appropriate. The data is way too sparse and the quality varies immensely and is normally quite lagged. As a concrete example perhaps you think the price of oil = f(oil demand in china, oil production in saudi arabia). Great. At best you get monthly data, probably lagged, and probably manipulated by the government. So that's 24 pieces of data per year....not great candidate for machine learning ideas. Perhaps very high-frequency, ultra short term automated trading can use tick data for things like this, but not for general prop trading that holds risk for days or weeks. I also expect outsiders imagine oil price = f(factor_a, factor_b, ...factor_z) and it's just the job to figure out that ka, kb, ... kz are. But markets aren't physical phenomena like a spring where repeated measurements of length and weight can eventually can estimate what the K-factor is. If markets are springs, then their K-factor changes all the time. Factors that sometimes matter for one month don't matter for the next six or twenty four. That's why it's important to monitor the many spaces that can drive a market -- underlying fundamentals, 'flows' from the financial/hedging markets, and myriad non-specialist investors who behave based totally different drivers than oil specialists. As much as oil trading could be trying to accurately count barrels of oil, it's also important to understand the 'meta' of who is buying/selling and why. To be clear, my trading is not automated. At end of the day I'm making a bet myself, not a machine deciding. It's more like I have machine exoskeleton helping me to see a little bit more clearly and maybe a little less emotionally, which hopefully leads to slightly smarter bets.
hajimex 7y ago
I think, before automation, konmari method is more important. If the job doesn't spark joy, we should get rid of it
nairobi222 7y ago
s09dfhks 14 hours ago [-] Haven't totally automated my entire job but a small portion of what I do is configuring a fresh aws account some core services (SMTP, NTP, LDAP etc etc) and the associated VPC configurations before handing it off to the devs. Wrote a pretty simple python script that takes some inputs, cidr range, account name, jira ticket etc, and sets up all the terraform, ansible groupvars, ssh key generation, secret storing and a few version control check-ins. Reduced the footprint from a few hours to literally less than one minute.
jonny789 7y ago
I have been assigned a work in which data is uploaded every month into a web portal for hundreds of items, manually doing so takes 1-1.5 hour. But after automating the uploading task using python beautifulsoup and requests module, it gets done in few minutes.
s9876121 7y ago
A major part of my job is to create reports. These are mind numbingly created in Powerpoint / Word with graphs from Excel. I've been told that automating any of the reports I do is basically impossible. I'm expected to do overtime as needed to get reports done on time. This happens most weeks. Unfortunately scripting and devops aren't a thing where I am even if they could be applied. I'll need to find another corp. The next position I'm going for in is DevOps. But I haven't got enough of the silly certs / experience yet. If only I had time off to study and wasn't creating reports all day. Then I'd be able to build my project portfolio and do the exam preparation. I've been here 18 months doing nothing but reports and a little system administration. My plan is to be here for no more than 36 months as this level of manually creating reports is too mind numbing. Also me: I generate a very important report for presentation each Monday afternoon and it takes around 4-6 hours of data gsthering and preparation. For the first four weeks I did it manually then realised I could automate some parts and then gradually up to all of it. Now it takes 30 minutes of script runtime with no manual intervention. There's a patchwork of many other 1-2 hour reports I do across other days. They are now all fully automated by me. They take around 5 minutes of script runtime but no interaction. I have another big report about to be assigned for Fridays. Its expected to take around 5-7 hours of manual collation. My estimation is that I will be able to run this in around 20 minutes. In all I generate multiple word documents and powerpoint documents using excel spreadsheets with most of the heavy lifting being done using python, powershell, R and an ecosystem of little elixir critters. As soon as I reach 24 to 36 months I'll have everything lined up and I'm out of here. I've done enough networking with various people to know that my portfolio of experience is relevant and useful. I've gotten my interview skills up. Plenty of my contacts agree.
TheGabeMan 7y ago
Last year in school I also worked for a car dealership one day a week. One of their employees transfered a task to me. She showed me a spreadsheet in Lotus 1-2-3, in which they had a list of cars with the day the received the car and they day the sold the car. She was counting the number of days in between and entered that number into the next cell. Took here 2hrs each week. During the explanation by her I already showed here how to do this with a simple formule. She was ecstatic. Second thing I automated was automating their invoices. There was AS400 system that generated a CSV file which they printed. Then they opened a template Word-document and manualy copied the data from the print into the document, printed that and continued with the next. With some simple marco's I saved them about a days work every week.
japhyr 7y ago
I was a high school teacher in a school that worked primarily with underserved students, who often had transcripts with many academic gaps. So we had to spend more time than usual examining transcripts to make sure students were taking the right classes to graduate. It was a small school, with three teachers and a few support staff. We literally spent hundreds of hours each year poring over text-based transcripts with students, parents, and each other. I spent one staff training day writing a script that parsed the transcripts and generated a visual transcript for each student. It had gray bars showing how much credit was required in each subject area and subdiscipline area, and green bars showing how much credit students had earned. For students close to graduation, it also generated a list of the classes they needed to take. What used to take 10-15 minutes for each student, and was error-prone, now took 30 seconds to 1 minute. The really interesting and satisfying part wasn't just noticing how much time was saved. We ended up spending just as much time with each student and parent/guardian, but the conversations focused much more on how students could get through to graduation. What used to be a frustrating period of analyzing past failures became a focus on what students could do now that they were focused on graduation.
- throw03172019 7y ago
  We need more teachers like this! Optimizing to spend time on getting their students to their goals instead of optimizing to go home early. Thank you!
- peterbozso 7y ago
  These small, simple things are the reason why I truly love technology.
gadders 7y ago
I had to compile reports from Jira of tickets in a release that weren't "in standard". I wrote some Excel VBA that got the details via the Jira rest interface, formatted a report, and sent it round to the "offenders".
ruairidhwm 7y ago
When I worked as a lawyer, one of my tasks as a junior was to check the court lists every day against the spreadsheet of clients we had who were involved in litigation. If their name appeared then I had to send an email to the relevant lawyer containing the information. This took about 30 - 60 mins each day and was monotonous. Just copying and pasting the client name into the browser search tool. I automated the process, making a small website which lawyers could add their client and would receive an email if the name appeared in the court lists. I ended up turning it into a small SaaS business for a while and sold a licence to my company. Not a huge victory but it felt great at the time!
- TheRealWatson 7y ago
  There are several businesses that grew out of people that worked in law firms or corporate legal departments and spotted the numerous deficiencies and opportunity for automation. I know this is true for most businesses but having worked in that area for a few years made me think they're specially lacking process automation considering how bug the Legal industry is in the US.
codingdave 7y ago
I once took a temp job to get some quick spare cash after a failed startup attempt that wiped out my savings, when I still wanted a bit more time before getting another coding job. I ended up helping with some general office work in a mortgage bank, in a department where some accountants would look at all the applications that fell in the grey areas where it wasn't automatically accepted or rejected. They entered the data in a spreadsheet, then looked at other pages of the spreadsheet to see if it was OK to approve or not. Ultimately, there was no grey area... just humans in the middle when it was close. So I wrote a button (VBA, in Excel) for them that would just do the lookups, and send the correct email response back to the original loan officer. It worked well. They sent my button to all the loan officers, and that department was no longer needed, at least not for anything other than some basic communications asking applicants for more info once in a while. I had a fun conversation with their IT department, too, when they found out that a temp had automated a department.
- x2f10 7y ago
  Was there a job offer / spot bonus?
  - codingdave 7y ago
    Yep. I got a job offer. It was nowhere near competitive with other opportunities, though, so the talks didn't go far.
    - deleted 7y ago
      [deleted]
    - serpix 7y ago
      This checks out. Banks have no problem spending millions on poor solutions but zero interest in rewarding optimisations.
      - shyn3 7y ago
        That's because you are a threat to the manager if you implement something like this without approval.
darrenwestall 7y ago
I became a manager and delegated everything.
- quickthrower2 7y ago
  Did you delegate managing too?
  - darrenwestall 7y ago
    Yes, but let me elaborate. I quickly learned the way to get promoted is to be replaceable. If you’re relied upon too much or if you’re the guy who always fixes things - it’s actually too risky for a business to promote you. Throughout my career I always made sure I was giving people below me the opportunity to step up, which allowed me to step up too. I quickly appointed team leaders as soon as a team I managed had more than 4 people for example. I had to manage the team leaders but they had a lot of freedom. Some days I had so little to do you wouldn’t believe it, but the department was running the best it had in years and everyone was happy.
    - quickthrower2 7y ago
      Nice answer. The bosses I have been most happy to work for had such systems in place. Scrum/Agile (while sometimes derided) if well executed can help with this. If the team is 'self-organizing'. Although you had little to do, I imagine you probably have to nudge things in the right direction every now and then. Another aspect I think is important is hiring good people. Hire bad people and you will be a busy manager!
      - darrenwestall 7y ago
        Absolutely, if you truly want to automate your day job, you need the best people propping you up. Luckily I did, and if I didn’t I removed the rot fast - that can spread and really make your life difficult.
    - mandeepj 7y ago
MaanuAir 7y ago
I deal quite a lot with JIRA at my job, as this is a corporate tool. Often, I get questions about the content of these JIRA issues, by issue key: either orally ("hey, what about PRJ-12345 ?"), or in plain text format in email, i.e. "PRJ-12345" _without_ the hyperlink of course. That could result in going to a https://jira.domain.tld/browse/PRJ-12345 https://jira.domain.tld/browse/PRJ-12345, or even https://anotherjira.domain.tld/browse/QSK-54321 https://anotherjira.domain.tld/browse/QSK-54321 (yes we have two, separated, instances running). Like dozens of time a day. As the JIRA UI does not make it fast enough for me to lookup for such issues so often (and having two different URLs do not help), I wrote a little Hammerspoon script that listen to a keyboard shortcut key. When I press it: - if a JIRA issue key is currently highlighted on screen: it just points my browser to it - if no, an interactive auto-suggester appears and I can type in the issue key, so that I can browse to it immediately as well. Only a very small automation, but I literally save several precious minutes per day. Mentioning here since the idea might be helpful for others...
- bberenberg 7y ago
  Your method will be faster, but fyi, you can also put the issue key in the search and it will go directly to the issue. We typically configure search shortcuts in the browser so if you type jABC-123 it will auto expand the j to https://somejira.tld/browse/ABC-123 https://somejira.tld/browse/ABC-123
  - TheRealWatson 7y ago
    That's what I do too but I liked the idea of highlighting and hitting a shortcut, from anywhere. I also configured Sourcetree to match Jira IDs and hyperlink them.
- xemdetia 7y ago
  I do this by using the JIRA search integration with a prefix, and I have one prefix per relevant JIRA instance I have to talk to. 'j PRJ-12345' is what goes in the search bar and I zip right there.
compliance_tech 7y ago
I work in compliance for a major bank. Our process to investigate odd spending/payment patterns could not have been more manual. Essentially, cases alert according to a variety of rules and you investigate accordingly. Everything starts in Excel and you know what to look for based on why the case alerts. I automated everything (analysis) required in Excel using VBA. Did not know any programming prior to this but I did know this was the point of using computers ("bicycle for the mind"). Learning VBA was a little painful but eventually I figured out how to control a web browser with it so I would pull the required documents from the web. From there, I got Excel and Word to talk and auto-generated our summary memos. So, pretty effectively automated myself out of my position. The lesson is, while I thought it was all pretty straight-forward, many of my colleagues found my new work flow challenging to use and were pretty concerned my path would put us out of work eventually, which was only kind of the goal. Have since rewritten everything in Python, which is a comparative joy, and I'm now working on using machine learning, NLP, NLG to replicate some of the same work to be more intelligently automated.
alsysadmin 7y ago
I was a VMware admin primarily for years in engineering shops and everywhere I worked constantly had issues with resource allocation, which team had the most resources deployed, who had the most idle vCPU and memory running... At one company pretty much my sole job for 2 days a week was to look over utilization by team, shutdown idle VM's and check for any that were provisioned with too many resources. I'd have to e-mail the team, power them down and resize the resources and boot it back up, etc. I automated all that with scripts, from the team e-mails with a nice HTML table report of the pending changes to actually running the operations to resize and power off the machines. I was pretty proud of all that work.

96 points by amerf1 7y ago

Curious to hear stories of people who automated their 9 to 5 jobs

I actually got my first dev job by doing exactly that. While I was searching for a new career and vaguely interested in something to do with IT, I was doing short-term temp work as an office admin. One was in a startup - not tech-related - where I spent most of my day searching for customers' names and addresses in a spreadsheet, then copy-pasting them into a bunch of Word templates and printing them out. I already knew about mail merge, but in this job the particular set of docs to be generated would be different according to the specific circumstances of each individual case, along with one or two customer-specific paragraphs that came in an email and had to be pasted into the document. Having been a hobby programmer as a teenager, I had a hunch this could be made more efficient. Once I discovered VBA (what? Excel has its very own programming language BUILT IN?!) I quickly threw together some macros and an Excel form that generated the whole personalised document set in a few clicks (what? you can write code in Excel to control Word? This is AWESOME!) At the time I was sharing a house with a couple of CS students and one of them let me borrow a very academic textbook about this wacky thing called SQL. I fell in love with the idea of tables and joins and stored procs, skipped over all the difficult maths parts, and discovered back at work that the funny icon next to Word and Excel called "Access" was actually a SQL database. Feeling like I had all the power of the universe at my fingertips, I hacked together a custom CRM system which pretty soon the whole office was using, and pretty soon it started grinding to a halt. Round about then the company hired its first IT Manager, he liked what I had done and said: maybe you should check out this new thing called SQL Server 2000, do you want to be our Database Developer? So I shifted the back end over to SQL then discovered .NET 1.1 and started writing better front ends and hired a small team to take it over. So yeah, I kind of fell into business app development by accident just by automating my very repetitive office admin job. That was 15 years ago, and I'm still getting paid to automate other people's 9 to 5 jobs.

shyn3 7y ago
You should thank that IT Manager today. Most of them would just ask you to get approval to use a database on your own PC at this point.

davchana 7y ago

Not a day job, but while looking for a day job, I came across a process, where a state run department has multiple openings; you upload your resume things like name, details, previous jobs, address etc to a standardized form, & then at job search page, just click the listing, review the text, click Apply, attach your standardized auto generated resume by click of a button, agree to their t&c with a checkbox, & click Apply, & note down the information from thank you page about job number, city, position, department, type etc. All of these 6-7 pages had unique URL, & all html was nicely generated with proper css id class names etc. After spending about 5 Sundays doing this manually, I quickly wrote a JavaScript bookmarklet where it gets the current URL, & switch case calls the required function. Each page had different function to look for element, focus, highlight for me, & click it. The last page all required information was loaded & sent to a Google script for spreadsheet.

c22 7y ago

My first job was doing restock for a small independent retailer. The only report we could print showing inventory was inclusive of all the items we had ever carried, regardless of current status. The software was very limited and there was not much in the way of filtering. The best you could do was break the report up into ranges by vendor. So my job was to go through the alphabet printing reports (e.g. "every item from a vendor that starts with 'A'"). I'd take this report, which for some letters was over a hundred pages long (about 20 items per page) then, using a highlighter and a ballpoint pen I'd go through the report highlighting the items that had stock available in the "backstock" column and had stock in the "sales floor" column that was lesser than the "restock level". I also had to compute the restock amount by subtracting the items on the floor from the restock level and taking the max of that number or the available backstock. I'd write this number in the margin of the report, then I'd travel to the backstock area, pick the highlighted items, and restock them onto the sales floor. After this was done I'd move on to the next letter. The goal was to cycle through the alphabet at least once a week and during the off-peak season this worked fine, but during the busy season, even with extra help, this system was showing signs of stress. The first thing I did was to write a small perl script which I placed at the path the POS software was expecting to find a line printer. My script took the print-formatted report as input and simply stripped out the lines with no available backstock before forwarding the remaining lines to the printer. This alone reduced the number of pages printed out by an order of magnitude in most cases (from ~100 pages to ~10). At first I would copy this script to the printer path, run the report, then re-link the printer, but later I modified the script to recognize that specific report and pass-through other print jobs unmolested. Over the next three years or so I added onto the script a lot. First I made it compute the restock amount for me, then I had it start skipping lines where the restock was trivial (e.g. restock amount was 1 and there were still >10 on the floor). Eventually the script was maintaining its own parallel inventory database, tracking rate of sale for each item and only suggesting restock for things that had very low inventory on the floor or were selling very quickly. I got to the point where I could restock the entire alphabet every morning and have time leftover in the afternoons to help with receiving. By the time I left I had a dedicated terminal set up in the back during the busy season that ran my scripts every 10 minutes and displayed in near real-time items that required critical restock throughout the day. I heard a few years after I left they had switched back to the old methodology because they couldn't get their POS vendor to support my setup.

geocrasher 7y ago

I don't automate my job specifically (I'm a tech trainer, currently) but when I have optimized myself heavily using AutoHotKey. I spend most of my time these days between Slack and a bash prompt, and so I have created macros for myself. I lean on it pretty hard, and have cut down the amount of typing I do simply because I can now type "yyy" to get "Yup." and other silly optimizations that save me time and effort, mostly cumulatively. There are few big optimizations to be gained this way, but they add up. I also have entire phrases and paragraphs macro'd for the same reasons, as well as common typos. It doesn't sound like much but these things save me a lot of time.

s09dfhks 7y ago

Haven't totally automated my entire job but a small portion of what I do is configuring a fresh aws account some core services (SMTP, NTP, LDAP etc etc) and the associated VPC configurations before handing it off to the devs. Wrote a pretty simple python script that takes some inputs, cidr range, account name, jira ticket etc, and sets up all the terraform, ansible groupvars, ssh key generation, secret storing and a few version control check-ins. Reduced the footprint from a few hours to literally less than one minute.

audiometry 7y ago

I am a trader. A million different factors drive commodity prices, and their significance ebbs and flows over time in unpredictable ways. So typically you'll have a spreadsheet that monitors one of these occasional factors. You run it 98 days in a row and it's irrelevant. This lulls you into complacency on the 99th day, where you don't look at it, and on the 100th day suddenly it becomes relevant again and you didn't notice it in time because you hadn't checked the spreadsheet. So my agenda has been to move the analysis corpus into Python, directly talking to Bloomberg and other data sources. Then these things run automatically and can flag me when they're significant. Rather than trusting me to check scores of spreadsheets and wading through endless True Negative results. When I started working, Excel was the powertool that allowed a junior to outperform dinosaurs using calculators, and make an outsize impact. My feeling now is that Python and similar tools are the equivalent of my Excel 20+ years ago, and Excel is the 'desk calculator'.

inakarmacoma 7y ago
Would you be open to sharing more? What sort of things are you doing in Python that are new and unique, distinct from capabilities you had in spreadsheets?
- dbancajas 7y ago
  isn't python used for automating? I guess excel can do autochecking if you really pushed it
- audiometry 7y ago
  What things are new/unique? Hmmm, I'll bet someone could implement almost all of what I do inside excel sheets one way or another, but it would just be super-cumbersome to do. One big example is regression. All Excel really does is simple linear regression. Add more variables you get a better r2'd and that's it. Comparatively, with scikit it's trivial to avoid all horrid overfitting problems that come with simple regressions in excel. I use of lasso-cv model, and better practices like cross-validation, etc. Another big improvement is I replaced Excel -> Tableau visualizations pipeline with using Altair-Viz. Altair makes almost-Tableau-like charts pretty easily and cuts out lots of intermediary steps -- jump right from dataframe to a chart. It's much easier to add sanity/safety assertion checks in python than in Excel. Everyone knows examples of excel spreadsheets that are subtly poisoned. I took someone else's technical charting ideas and reimplemented them in python, which then just alerts me for key events (changes in trend, potential exciting issues, etc). Now I don't have to look through the data myself, which is a chore, and which is susceptible to my mood. It just spits out its objective 'truth'. One thing about (at least commodities trading) that outsiders might not realize is -- none of the sexy FAANG big data stuff is particularly useful or appropriate. The data is way too sparse and the quality varies immensely and is normally quite lagged. As a concrete example perhaps you think the price of oil = f(oil demand in china, oil production in saudi arabia). Great. At best you get monthly data, probably lagged, and probably manipulated by the government. So that's 24 pieces of data per year....not great candidate for machine learning ideas. Perhaps very high-frequency, ultra short term automated trading can use tick data for things like this, but not for general prop trading that holds risk for days or weeks. I also expect outsiders imagine oil price = f(factor_a, factor_b, ...factor_z) and it's just the job to figure out that ka, kb, ... kz are. But markets aren't physical phenomena like a spring where repeated measurements of length and weight can eventually can estimate what the K-factor is. If markets are springs, then their K-factor changes all the time. Factors that sometimes matter for one month don't matter for the next six or twenty four. That's why it's important to monitor the many spaces that can drive a market -- underlying fundamentals, 'flows' from the financial/hedging markets, and myriad non-specialist investors who behave based totally different drivers than oil specialists. As much as oil trading could be trying to accurately count barrels of oil, it's also important to understand the 'meta' of who is buying/selling and why. To be clear, my trading is not automated. At end of the day I'm making a bet myself, not a machine deciding. It's more like I have machine exoskeleton helping me to see a little bit more clearly and maybe a little less emotionally, which hopefully leads to slightly smarter bets.

hajimex 7y ago

I think, before automation, konmari method is more important. If the job doesn't spark joy, we should get rid of it

nairobi222 7y ago

s09dfhks 14 hours ago [-] Haven't totally automated my entire job but a small portion of what I do is configuring a fresh aws account some core services (SMTP, NTP, LDAP etc etc) and the associated VPC configurations before handing it off to the devs. Wrote a pretty simple python script that takes some inputs, cidr range, account name, jira ticket etc, and sets up all the terraform, ansible groupvars, ssh key generation, secret storing and a few version control check-ins. Reduced the footprint from a few hours to literally less than one minute.

jonny789 7y ago

I have been assigned a work in which data is uploaded every month into a web portal for hundreds of items, manually doing so takes 1-1.5 hour. But after automating the uploading task using python beautifulsoup and requests module, it gets done in few minutes.

s9876121 7y ago

A major part of my job is to create reports. These are mind numbingly created in Powerpoint / Word with graphs from Excel. I've been told that automating any of the reports I do is basically impossible. I'm expected to do overtime as needed to get reports done on time. This happens most weeks. Unfortunately scripting and devops aren't a thing where I am even if they could be applied. I'll need to find another corp. The next position I'm going for in is DevOps. But I haven't got enough of the silly certs / experience yet. If only I had time off to study and wasn't creating reports all day. Then I'd be able to build my project portfolio and do the exam preparation. I've been here 18 months doing nothing but reports and a little system administration. My plan is to be here for no more than 36 months as this level of manually creating reports is too mind numbing. Also me: I generate a very important report for presentation each Monday afternoon and it takes around 4-6 hours of data gsthering and preparation. For the first four weeks I did it manually then realised I could automate some parts and then gradually up to all of it. Now it takes 30 minutes of script runtime with no manual intervention. There's a patchwork of many other 1-2 hour reports I do across other days. They are now all fully automated by me. They take around 5 minutes of script runtime but no interaction. I have another big report about to be assigned for Fridays. Its expected to take around 5-7 hours of manual collation. My estimation is that I will be able to run this in around 20 minutes. In all I generate multiple word documents and powerpoint documents using excel spreadsheets with most of the heavy lifting being done using python, powershell, R and an ecosystem of little elixir critters. As soon as I reach 24 to 36 months I'll have everything lined up and I'm out of here. I've done enough networking with various people to know that my portfolio of experience is relevant and useful. I've gotten my interview skills up. Plenty of my contacts agree.

TheGabeMan 7y ago

Last year in school I also worked for a car dealership one day a week. One of their employees transfered a task to me. She showed me a spreadsheet in Lotus 1-2-3, in which they had a list of cars with the day the received the car and they day the sold the car. She was counting the number of days in between and entered that number into the next cell. Took here 2hrs each week. During the explanation by her I already showed here how to do this with a simple formule. She was ecstatic. Second thing I automated was automating their invoices. There was AS400 system that generated a CSV file which they printed. Then they opened a template Word-document and manualy copied the data from the print into the document, printed that and continued with the next. With some simple marco's I saved them about a days work every week.

japhyr 7y ago

I was a high school teacher in a school that worked primarily with underserved students, who often had transcripts with many academic gaps. So we had to spend more time than usual examining transcripts to make sure students were taking the right classes to graduate. It was a small school, with three teachers and a few support staff. We literally spent hundreds of hours each year poring over text-based transcripts with students, parents, and each other. I spent one staff training day writing a script that parsed the transcripts and generated a visual transcript for each student. It had gray bars showing how much credit was required in each subject area and subdiscipline area, and green bars showing how much credit students had earned. For students close to graduation, it also generated a list of the classes they needed to take. What used to take 10-15 minutes for each student, and was error-prone, now took 30 seconds to 1 minute. The really interesting and satisfying part wasn't just noticing how much time was saved. We ended up spending just as much time with each student and parent/guardian, but the conversations focused much more on how students could get through to graduation. What used to be a frustrating period of analyzing past failures became a focus on what students could do now that they were focused on graduation.

throw03172019 7y ago
We need more teachers like this! Optimizing to spend time on getting their students to their goals instead of optimizing to go home early. Thank you!
peterbozso 7y ago
These small, simple things are the reason why I truly love technology.

gadders 7y ago

I had to compile reports from Jira of tickets in a release that weren't "in standard". I wrote some Excel VBA that got the details via the Jira rest interface, formatted a report, and sent it round to the "offenders".

ruairidhwm 7y ago

When I worked as a lawyer, one of my tasks as a junior was to check the court lists every day against the spreadsheet of clients we had who were involved in litigation. If their name appeared then I had to send an email to the relevant lawyer containing the information. This took about 30 - 60 mins each day and was monotonous. Just copying and pasting the client name into the browser search tool. I automated the process, making a small website which lawyers could add their client and would receive an email if the name appeared in the court lists. I ended up turning it into a small SaaS business for a while and sold a licence to my company. Not a huge victory but it felt great at the time!

TheRealWatson 7y ago
There are several businesses that grew out of people that worked in law firms or corporate legal departments and spotted the numerous deficiencies and opportunity for automation. I know this is true for most businesses but having worked in that area for a few years made me think they're specially lacking process automation considering how bug the Legal industry is in the US.

codingdave 7y ago

I once took a temp job to get some quick spare cash after a failed startup attempt that wiped out my savings, when I still wanted a bit more time before getting another coding job. I ended up helping with some general office work in a mortgage bank, in a department where some accountants would look at all the applications that fell in the grey areas where it wasn't automatically accepted or rejected. They entered the data in a spreadsheet, then looked at other pages of the spreadsheet to see if it was OK to approve or not. Ultimately, there was no grey area... just humans in the middle when it was close. So I wrote a button (VBA, in Excel) for them that would just do the lookups, and send the correct email response back to the original loan officer. It worked well. They sent my button to all the loan officers, and that department was no longer needed, at least not for anything other than some basic communications asking applicants for more info once in a while. I had a fun conversation with their IT department, too, when they found out that a temp had automated a department.

x2f10 7y ago
Was there a job offer / spot bonus?
- codingdave 7y ago
  Yep. I got a job offer. It was nowhere near competitive with other opportunities, though, so the talks didn't go far.
  - deleted 7y ago
    [deleted]
  - serpix 7y ago
    This checks out. Banks have no problem spending millions on poor solutions but zero interest in rewarding optimisations.
    - shyn3 7y ago
      That's because you are a threat to the manager if you implement something like this without approval.

darrenwestall 7y ago

I became a manager and delegated everything.

quickthrower2 7y ago
Did you delegate managing too?
- darrenwestall 7y ago
  Yes, but let me elaborate. I quickly learned the way to get promoted is to be replaceable. If you’re relied upon too much or if you’re the guy who always fixes things - it’s actually too risky for a business to promote you. Throughout my career I always made sure I was giving people below me the opportunity to step up, which allowed me to step up too. I quickly appointed team leaders as soon as a team I managed had more than 4 people for example. I had to manage the team leaders but they had a lot of freedom. Some days I had so little to do you wouldn’t believe it, but the department was running the best it had in years and everyone was happy.
  - quickthrower2 7y ago
    Nice answer. The bosses I have been most happy to work for had such systems in place. Scrum/Agile (while sometimes derided) if well executed can help with this. If the team is 'self-organizing'. Although you had little to do, I imagine you probably have to nudge things in the right direction every now and then. Another aspect I think is important is hiring good people. Hire bad people and you will be a busy manager!
    - darrenwestall 7y ago
      Absolutely, if you truly want to automate your day job, you need the best people propping you up. Luckily I did, and if I didn’t I removed the rot fast - that can spread and really make your life difficult.
  - mandeepj 7y ago

MaanuAir 7y ago

I deal quite a lot with JIRA at my job, as this is a corporate tool. Often, I get questions about the content of these JIRA issues, by issue key: either orally ("hey, what about PRJ-12345 ?"), or in plain text format in email, i.e. "PRJ-12345" _without_ the hyperlink of course. That could result in going to a https://jira.domain.tld/browse/PRJ-12345 https://jira.domain.tld/browse/PRJ-12345, or even https://anotherjira.domain.tld/browse/QSK-54321 https://anotherjira.domain.tld/browse/QSK-54321 (yes we have two, separated, instances running). Like dozens of time a day. As the JIRA UI does not make it fast enough for me to lookup for such issues so often (and having two different URLs do not help), I wrote a little Hammerspoon script that listen to a keyboard shortcut key. When I press it: - if a JIRA issue key is currently highlighted on screen: it just points my browser to it - if no, an interactive auto-suggester appears and I can type in the issue key, so that I can browse to it immediately as well. Only a very small automation, but I literally save several precious minutes per day. Mentioning here since the idea might be helpful for others...

bberenberg 7y ago
Your method will be faster, but fyi, you can also put the issue key in the search and it will go directly to the issue. We typically configure search shortcuts in the browser so if you type jABC-123 it will auto expand the j to https://somejira.tld/browse/ABC-123 https://somejira.tld/browse/ABC-123
- TheRealWatson 7y ago
  That's what I do too but I liked the idea of highlighting and hitting a shortcut, from anywhere. I also configured Sourcetree to match Jira IDs and hyperlink them.
xemdetia 7y ago
I do this by using the JIRA search integration with a prefix, and I have one prefix per relevant JIRA instance I have to talk to. 'j PRJ-12345' is what goes in the search bar and I zip right there.

compliance_tech 7y ago

I work in compliance for a major bank. Our process to investigate odd spending/payment patterns could not have been more manual. Essentially, cases alert according to a variety of rules and you investigate accordingly. Everything starts in Excel and you know what to look for based on why the case alerts. I automated everything (analysis) required in Excel using VBA. Did not know any programming prior to this but I did know this was the point of using computers ("bicycle for the mind"). Learning VBA was a little painful but eventually I figured out how to control a web browser with it so I would pull the required documents from the web. From there, I got Excel and Word to talk and auto-generated our summary memos. So, pretty effectively automated myself out of my position. The lesson is, while I thought it was all pretty straight-forward, many of my colleagues found my new work flow challenging to use and were pretty concerned my path would put us out of work eventually, which was only kind of the goal. Have since rewritten everything in Python, which is a comparative joy, and I'm now working on using machine learning, NLP, NLG to replicate some of the same work to be more intelligently automated.

alsysadmin 7y ago

I was a VMware admin primarily for years in engineering shops and everywhere I worked constantly had issues with resource allocation, which team had the most resources deployed, who had the most idle vCPU and memory running... At one company pretty much my sole job for 2 days a week was to look over utilization by team, shutdown idle VM's and check for any that were provisioned with too many resources. I'd have to e-mail the team, power them down and resize the resources and boot it back up, etc. I automated all that with scripts, from the team e-mails with a nice HTML table report of the pending changes to actually running the operations to resize and power off the machines. I was pretty proud of all that work.