5 ms·
Great to see it move on from just JS (although I know you can run binaries, it's a little awkward). There was a 60s execution limit in preview, anyone know if
by IanCal 11y ago
Great to see it move on from just JS (although I know you can run binaries, it's a little awkward).
There was a 60s execution limit in preview, anyone know if thats still around / likely to change? I've got a task to solve a the moment which is basically
"When a user puts a file on S3, hit an API for each line, store all responses in S3"
Lambda would be awesome for the low end of that, small files would generally be alright, but as the filesize grows I'd need to try and setup workflows to split the file & rejoin the results (and guess how many requests I can make in 60s) which is starting to get into a more complicated setup than I'd really want so I'd be likely to just rent a machine and setup my own queue processing (again more complicated, but more in my control). Maybe I'm actually just looking for simple grid software and I should just rent some servers.
Essentially, what I'd like to use lambda for is "Run this code. It's not big or heavy, or requires much power, but run it and stop charging me when it finishes". I don't want to request machines, setup disks, compare multiple reserved costs, orchestrate them starting & stopping or handle updating them, and I don't want to worry about starting 100 5s jobs and that costing me 100 hours. I'm aware that's rather "moon on a stick", but I do feel like a lot of this is almost there.
I'm missing a big hole in my toolbox since PiCloud shut down. Millisecond pricing, tasks, queues, workflows, custom environments. Anyone here an old picloud user? What have you moved to?
GCE is interesting, with fast boot times and a 10 minute min, 1 minute increment pricing model. Ironworker is interesting, but the potential delay for a standard job (15 min) along with the cost vs computing time it's hard to justify for my tasks over just renting a machine from OVH ($50/mo would pay for a decent 32G machine, so I could run a lot more than the 25/500 limit for the smallest tier) and the 1 hour limits on the smaller plans cause the same issues as with the 60s limits on lambda.
I'm not knocking these services, they're all really impressive, just unfortunately have various restrictions / limitations that limit their application to the problems I currently have. If I had slightly different problems, I'd already be using one of these options. If people are using other services, I'd love to hear what you're using!
Edit - offtopic, but every post I've made recently has gone through twice. Chrome 43.0.2357.124 on a mac here. Anyone else having the same problem?
- BillinghamJ 11y agoSeems like it would be good to take the file, then dump each line onto a queue, then have lambda run for each thing on that queue. It should parallelize really nicely.
- IanCal 11y agoThat does sound like quite a good approach, there are a few issues though: 1. Some vague throttling is useful, but I can only do this on an account level with Lambda rather than per function. Trying to call the API in bursts of 10000 requests may be problematic, which would limit our use of lambda for other tasks (for which we may be happy running that many in parallel). The default limit of 100 would probably serve well enough though. 2. I've now got to add splitting and recombining code on each end, with concerns around failed jobs being silently missed from the final output file. Although that extra work may leave me with a better approach to handling some failed jobs out of a large file. Hmm. Part of the issue is that without the execution limit, this is an amazingly simple script. I've got a 10-20 line python script which does the actual processing of a file (read, hit api, store with a pool of N concurrent requests). Lambda is impressive because it adds only a small level of complexity to a problem and gives you a lot in return, just because my use-case is so simple that small amount of complexity adds up to relatively quite a lot. Currently the setup doesn't hit the API, it just creates a dedicated instance to process a batch of data locally, but I'm hoping to simplify things to send everything through the API and just scale & load balance separately. Having code to automatically turn on & be responsible for turning off machines makes me a bit nervous :) I've already missed that I'd deleted the shutdown command in a script and left a box running for a day while developing. Thanks for the suggestion though, I'll try and work through it in more detail today, see if I can see a clean way of dealing with the recombination. I think that's the side that I'm less clear on at the moment.
- siscia 11y agoThe main problem that I see in implementing what you are asking is related to the bandwith cost. I still haven't figure out how to measure how much bandwith a trivial application is using, so my price is only based on time. If I let you using the service for as long as you like you either need to pay A LOT or I don't allow you to use the network connection...