5 ms·
Google Sheets can be surprisingly handy as a UI for editing data. For example, I have a job on AWS that scrapes a handful of URLs and snapshots them. Instead of
by paulgb 7y ago
Google Sheets can be surprisingly handy as a UI for editing data. For example, I have a job on AWS that scrapes a handful of URLs and snapshots them. Instead of creating a database and hosting it somewhere, or hard-coding them, I put them in a spreadsheet that the AWS job reads at the beginning of every run.
- AllanHoustonSt 7y agoI continually find myself amazed with what people can do with Excel.
- peschu 7y agohaha yeah and I always wonder why especially engineers are so creative in using Excel as a graphics/painting tool :) Looks good, but a nightmare to read in a programmatic way ^^
- thrownaway954 7y agoHow do you get around the number of fetch limitations?
- paulgb 7y agoI'm hitting at a rate of 2/hour, it hasn't been an issue.
- thrownaway954 7y agohow are you accomplishing that? Does Google sheets have a scheduler built into it? Long running timer?
- paulgb 7y agoThe scheduled job runs on AWS, it just fetches the sheet when it runs.
- mneubrand 7y agoGoogle Apps Script has a scheduler built in: https://developers.google.com/apps-script/reference/script/clock-trigger-builder https://developers.google.com/apps-script/reference/script/c... I use this to automate a lot of tasks. E.g. automatically clip supermarket coupons every week, check for tax loss harvesting opportunities in my portfolio, etc. It's really powerful
- valtism 7y agoAren't you forced to use Google Apps Script to interact with Sheets? I remember being forced to use it for interacting with Google Forms API.
- dragonwriter 7y ago> Aren't you forced to use Google Apps Script to interact with Sheets? No, it has a REST API; you need Apps Script if you want automation within Sheets, but you can interact with Sheets without it.
- paulgb 7y agoThat's the best part, I don't even use an API! I just go File -> Publish to the web, then select "Comma-separated values" instead of "Web page" from the dropdown. Then I use Pandas to read the CSV in Python.
- freeqaz 7y agoI just made an example for how to do this using Refinery (most of the pieces were off the shelf using "saved blocks"). https://app.refinery.io/import?q=9lvohdvqd50g https://app.refinery.io/import?q=9lvohdvqd50g This project will: - Run every hour (via timer) - Grab URLs from a Google Sheet - Go screenshot all of the URLs - Write each screenshot to an S3 bucket - Email you at the end after the jobs finish There is more I could do with this (like trigger an email if something fails, etc). But at least does what you describe above in a (hopefully) succinct fashion! Also, shill alert as I'm an author for this service (but this was a fun project to build over a few minutes, so thank you).