4 ms·
Most folks here are focusing on the scheduling part, but change detection means you need some way to persist state (or have a long running process that keeps st
by hbrn 4y ago
Most folks here are focusing on the scheduling part, but change detection means you need some way to persist state (or have a long running process that keeps state in memory).
While you could do FaaS (e.g. AWS Lambda) + another service for state (e.g. S3/RDS/EFS), that seems like overkill.
Even today I would still do it the old way: store state in a filesystem (maybe SQLite if you have more complex needs), and configure a plain old cronjob on any of the servers you have access to.
Maybe put it in docker/dockerhub to make it easier to run.
Surprisingly it is an interesting question. You have a very simple task, but ideally you want the solution to include: running environment (let's say a container), scheduler, persistent state, monitoring (you want to know when it is down). Not to mention deployment.
There are solutions for each of those, but given the simplicity it would be nice to have a lightweight solution that includes all of that with minimal configuration. I doubt it exists.
I.e. you can do all of that on AWS, but I can't help but wonder if the infrastructure setup is going to be more complicated than your actual scraper.
Edit: a good way to think about it is: imagine that a person just learned how to scrape websites, they wrote a script that works locally. How much do they need to learn to move it to the cloud?
If your answer is: ok, so you just need to create an AWS account, dockerize your script, put it in a Lambda, create RDS instance, trigger your Lambda through EventBridge and setup CloudWatch... then there's something wrong with you. There must be a better way to reliably run a fucking 10 line script in the cloud.
There's been many times over the years when I wanted to write a simple scraper that does exactly what author describes and what stopped me is the amount of infrastructure bullshit I would have to deal with.
- VLM 4y agoI thought about AWS MQTT "IoT" as the pricing contains gems like a million MQTT packets for a dollar. You'd use a MQTT "retained message", push the current SHA1 hash of the page or whatever will fit (maybe only a CRC32 LOL), then connect an hour later to receive that message containing the hash. The cost would seem to be $1 per 5 years because AFAIK AWS meters on input and output... I'm not bored enough to audit the AWS pricing structure to see if this will work. I think $2 per decade would be hard to beat for a billed service. I think the cheapest "cheaty" way to store data for free would be to pollute tracking sites. Open an account at some company, change your mailing address every hour, etc.
- TylerE 4y agoHash isn’t nearly enough. Web pages love to generate random ids, cache busters, etc.
- throwaway894345 4y ago> If your answer is: ok, so you just need to create an AWS account, dockerize your script, put it in a Lambda which is hooked up to RDS, trigger lambda through EventBridge and connect it to CloudWatch... then there's something wrong with you. There must be a better way to reliably run a fucking 10 line script in the cloud. I mean, you don’t have to dockerize it, and I would use S3 or Dynamo for state, but otherwise it’s a pretty simple solution and it can run entirely in the free tier on AWS. It’s way better than managing your own host, especially when you consider that you get monitoring, alerting, logging, etc for free. EventBridge is just the standard AWS thing for triggering stuff (in this case, a cron schedule)—don’t let the branding scare you.
- cebert 4y agoI do a lot of serverless development and agree with you on this. This is a very simple and cost effective problem for lambda and EventBridge.
- cash22 4y agoI don't have much experience with AWS Glue, but would this fit the use case here? https://docs.aws.amazon.com/glue/latest/dg/monitor-data-warehouse-schedule.html https://docs.aws.amazon.com/glue/latest/dg/monitor-data-ware...
- deleted 4y ago[deleted]
- archsurface 4y ago"you need some way to persist state (or have a long running process that keeps state in memory" Hash the data, update the script with the hash.
- paradox242 4y agoPut the 10 lines in a Lambda and invoke it from a schedule in EventBridge. Done. If you follow the default settings you'll get Cloudwatch logging as well.