13 ms·

The rise of embarrassingly parallel serverless compute

132 points by prostoalex 6y ago

ipnon 6y ago
It's going to take a new kind of software engineer to build these fully distributed systems. You can imagine calls for "Senior Serverless Engineers". Will conventional serverful engineers be left in the dust, or will the serverless engineers just break away and pioneer the apps on a new scale?
- deleted 6y ago
  [deleted]
- corty 6y ago
  Frankly, there isn't much engineering to be done. This only works for embarassingly parallel tasks, so you can just start a loop, distribute data, start compute, collect results. Done. For anything else, this model breaks down. Need aggregation of results? Communication among nodes? Computation subdivision that is not strictly predeterminable? Sorry, not embarrasingly parallel, won't be doable like this. You may be able to extract some embarrassingly parallel part, like compilation of independent object files, but very often you still have a longish, complex and timeconsuming serial step, like linking those object files. This kind of recognising different parts of a program is already state of the art, no need to invent a new field...
  - efnx 6y ago
    Speed and parallelism are good features but the killer app of serverless is the cost savings. You only pay for the time your function is running. Traditional server applications can be rewritten “serverless” so long as your pipeline is pretty functional, ie you’re not saving critical state in memory on the server process itself.
    - threeseed 6y ago
      For some use cases the cost saving is a big deal e.g. resizing an incoming stream of images. But for the majority of apps it doesn't save that much. And it still pales in comparison to the cost of the engineers. Which often spend significantly more time to build, debug, test etc a serverless app than one they just throw on an EC2 server.
      - efnx 6y ago
        “Apps” is a pretty vague description of an application, but if you’re writing a server it’s just as easy to “just throw” the app on lambda as it is on EC2. In general I ask myself if my new server can run on a lambda and if it can’t then I reach for something else. Most of the time it can.
    - tsimionescu 6y ago
      > Traditional server applications can be rewritten “serverless” so long as your pipeline is pretty functional, ie you’re not saving critical state in memory on the server process itself. And if you're storing things in DB, how long does it take for your lambda to start up now? In a project I was on, it could easily take 3s for a new request to be served, because we had a lambda that was doing and auth check with the DB, and then a different lambda that was doing the actual application stuff with another DB. So not only would we incur the cold start cost twice, but, since our lambdas needed a nic inside a VPC to talk to the DB, the cold start cost was huge. And of course, you pay that cost for each additional concurrent connection, not just once. Of course, if we had stored everything in some other managed service, and maybe used some managed auth scheme, this would have not been a problem. AWS likes it when they get you hook, line and sinker.
      - ElFitz 6y ago
        Was the first lambda waiting on the second's result before returning? If so, you really shouldn't rely on synchronously invoking à lambda from within another lambda. Also, things stored in the global scope (in Node.js functions) are kept between different invocations of a same container. If it can help...
- threeseed 6y ago
  Serverless has been around for many years now. It doesn't require a new kind of software engineer. It's just another software architecture to go alongside micro services, containerisation etc. And it hasn't changed the world because (a) it's the ultimate form of vendor lock in and (b) it makes even simple apps much more complex to reason about and manage.
  - arcturus17 6y ago
    > much more complex to reason about and manage I really dislike the local dev experience and deployment for serverless. But otherwise the model is pretty clear: a file is a function, data goes in, data comes out, just like any other server function. If one instance is busy, spin off a new one. What’s hard to reason about?
    - flak48 6y ago
      Yep. Inversion of control has always been the norm with whatever server/framework you choose. If you have a /myroute handler defined in express or flask or even form.php or form.cgi in Apache,you never had to write the code to make user requests trigger your handler anyway even in the old days. That's the entire point of using a server instead of listening to a socket yourself With serverless the same thing still happens with someone else managing the path from a request to a handler and back. In fact, if you ever used a cpanel host with PHP like in the good old days (110mb.com anyone?), you already used 'serverless'. You just uploaded .php files to a directory and your website just 'magically' worked.
- adev_ 6y ago
  > It's going to take a new kind of software engineer to build these fully distributed systems. What your are selling here is nothing else than a new proprietary-scheduler-runtime to run embarrassingly parallel jobs ( the easiest kind of parallelism) There were already plenty of solution to do that, the only difference here is that you run on AWS lambda. Why would you need an entire new type of engineer to do that ? There is nothing new here excepted buzzwords. Are engineer nowadays a script-kiddies bind to a technology there entire career ? (Tip: of course no)
mirimir 6y ago
> Basically grep for the internet. So is this a workaround for "censorship" by Google etc? And where would the crawl archives come from? Also, I wonder how this could be made usable and affordable for random individuals.
shoo 6y ago
there is a subset of embarrassingly parallel problems that are are heavily data intensive. E.g. suppose you have 100 TB of data files and you want to run some kind of keyword search over the data. If the data can be broken into 1000x100GB chunks then you can do some map-reduce-ish thing where each 100GB chunk is searched independently, then the search results from each of the 1000 chunks is aggregated. 1000x speedup! serverless! however, if you want to execute this across some fleet of rented "serverless" servers, a key factor that will influence cost and running time is (1) where is the 100 TB of data right now, (2) how are you going to copy each 100 GB chunk of the data to each serverless server, (3) how much time and money will that copy cost. I.e. in examples like this where the time required to read the data and send the data over the network is much larger than the time required to compute the data once the data is in memory, is going to be more efficient to move the code & the compute to where the data already is rather than moving the data and the code to some other physical compute device behind a bunch of abstraction layers and network pipes.
- gopalv 6y ago
  > there is a subset of embarrassingly parallel problems that are are heavily data intensive. There's an even smaller subset which is one-shot data access. > in examples like this where the time required to read the data and send the data over the network is much larger than the time required to compute the data once the data is in memory The annoying thing about lambda and other functional alternatives is that data-access patterns tend to be repetitive in somewhat predictable ways & there is no way to take advantage of that fact easily. However, if you don't have that & say you were reading from S3 for every pass, then lambda does look attractive because the container lifetime management is outsourced - but if you do have even temporal stickiness of data, then it helps to do your own container management & direct queries closer to previous access, rather than to entirely cold instances. If there's a thing that hadoop missed out on building into itself, it was a distributed work queue with functions with slight side-effects (i.e memoization).
  - adev_ 6y ago
    > If there's a thing that hadoop missed out on building into itself, it was a distributed work queue with functions with slight side-effects (i.e memoization). Is that not named spark ? :)
alexchamberlain 6y ago
> One of the challenging parts about this today is that most software is designed to run on single machines, and parallelization may be limited to the number of machine cores or threads available locally. Depending how you look at it, I don't think most software is designed to take advantage of multiple cores, let alone multiple machines.
- efnx 6y ago
  Both points are true, but the author is talking about writing new software specifically for serverless compute.
lachlan-sneff 6y ago
There's no way to distribute really lightweight thunks of arbitrary code. Maybe WASM can work here, especially if you shape the standard APIs in the right way. You'd also need built-in support in tooling and compilers, where you can compile specific functions or modules into something that can run separately without actually doing that manually.
- theamk 6y ago
  This depends on how lightweight are those chunks. If you goal is <0.1 sec startup -- yeah, then you'll need WASM. If you are OK with 1-5 second of startup, you have a ton of options. Apache Spark uses JVM magic to send out the the raw bytecode. You can start up docker container. If you are willing to rewrite stdio, you can exec machine code under seccomp/pledge. There are even full-blown VM solutions -- Amazon Firecracker, which claims that: "Firecracker initiates user space or application code in as little as 125 ms and supports microVM creation rates of up to 150 microVMs per second per host."
  - threeseed 6y ago
    And for people that don't know. AWS Lambda uses Firecracker under the hood.
- j88439h84 6y ago
  I dont really understand your claim. You can send code to another computer and it can execute it and send it back. What is the problem?
Uptrenda 6y ago
Except that none of these compute resources are free and the cost of cloud-based super computing is a complete rip-off. It's actually thousands of dollars cheaper to build your own super computer from used rack mount servers than it is to use AWS, Azure, or Google cloud for lengthy computations. As an example: you can buy a Dell PowerEdge M915 from ebay with 128 cores for ~$500 USD and a rack costs around the same. Five of them is 640 cores for a total cost of just 3k USD. That's 640 cores that you now own forever. Guess how much it would cost to use that many cores for a month on AWS? Well over 10k... and next month its another 10k... and so is next month... With this option you only pay for power and still have the resources for all future experiments. I think the m1000e rack can fit 8 blades in total so you could upgrade to a max 1024 cores in the future! The downside with this particular rack is that it's very, very loud. But I've run the numbers here and it's hard to beat high-core, used rack servers on a $/Ghz basis.
- bpodgursky 6y ago
  None of this math is correct. You absolutely cannot get 128 cores for $500. Please link to a single valid-looking eBay offer at that amount (you may have seen 128GB and thought it was 128 cores). I see 32 cores for $500, at best. Second, 640 cores on an m4.large for 3 years upfront (the equivalent purchase) is $8755/mo, not well over 10k. Third, you really underestimate electricity costs if you're running a server 24/7 at all cores. Last, and most importantly, most people simply don't run servers 24/7. They run batch computations for five hours a week, or a day, and then spend a week or two crunching the numbers. There's some case to make that for certain institutions with very particular compute needs, running on-premise might make short-term financial sense (let's not even get started on labor costs). But it's really inaccurate to call cloud computing a "rip-off" for anything but the niche-est customers.
  - sigstoat 6y ago
    > Third, you really underestimate electricity costs if you're running a server 24/7 at all cores. not to mention the cooling, or the per square foot cost of renting the space for this hardware. anything close to what's being described here is going to need some sound deadening (be it distance or material) from humans.
vsskanth 6y ago
Yeah but how expensive will it be ? Some numbers would be nice for something like hyper parameter tuning
tlarkworthy 6y ago
Don't overcomplicate it. Xargs and curl is often enough to drive big, ad hoc jobs.
- kristopolous 6y ago
  These things only have a handful of customers in the world. Datasets that are tens of gigabytes, or maybe 100mil records or so...this really covers most things. And for every 1 thing it doesn't, there's 20 more claimed that a single machine using simple tools could handle just fine. Being able to detect when things have been processed, have a way to set dirty flags, prioritize things, have regions of interest, be able to have re-entrant processing, caching parts of the pipeline and having nuanced rules for invalidating it, these in my mind are kinda basic things here. When they aren't done, sure, someone will need giant resources because they're doing foolish things. But that's literally the only reason. Substituting money for sense is an old hack.
  - logicchains 6y ago
    A nice paper about that idea: https://www.usenix.org/system/files/conference/hotos15/hotos15-paper-mcsherry.pdf https://www.usenix.org/system/files/conference/hotos15/hotos...
    - kristopolous 6y ago
      This is like an infoworld quality article in LaTeX. I don't know why this is beneficial. The LaTeX format scares me away most of the time. I think "oh god, I don't have all afternoon to study an academic document!" This thing is a 3 minute read
  - necovek 6y ago
    Opening your mirrorless camera SD card with images and the image thumbnailer takes... Forever? Doing a facial search on it? Matching a rhythm picked up by the mic to your local music collection? Hashing and/or encryption of data. There's plenty of desktop-like use cases that would benefit from massively parallel computation, but network (or even IO) bandwidth is currently going to be the limiting factor. IOW, we are not there yet! Currently, we can parallelize tasks which are low on data and high on computation. So how can we expand the IO bandwidth for everyone, even desktop or mobile users?
ChuckMcM 6y ago
Hilariously, in 2008 Google was hugely resistant to this idea. DARPA had just put out an RFP for a new way of running computationally dependent tasks that currently ran on super computers on a 'shared nothing' architecture (which is what Google ran at the time (and I believe they still do). I had done some research in that space when I was at NetApp looking at decomposing network attached storage services into a shared nothing architecture so I had some idea of the kinds of problems that were the "hard bits" in getting it right. I recall pointing out to Platform's management that if Google could provide an infrastructure that solved these sorts of problems with massive parallelism that currently required specialized switching fabrics and massive memory sharing we would have something very special. But at the time it was a non-starter, way too much money to be made in search ads to bother with building a system for something like the 200 customers in the world total. I didn't care one way or the other if Google did it so after running at the wall of "under 2s" a couple of times I just said "fine, your loss."
dstaley 6y ago
I'm a huge fan of using Lambda to perform hundreds of thousands of discrete tasks in a fraction of the time it'd take to perform those same tasks locally. A while back I used Lambda and SQS to cross check 168,000 addresses with my ISP's gigabit availability tool.[1] If I recall correctly each check took about three seconds, but running all 168,000 checks on Lambda only took a handful of minutes. I believe the scraper was written in Python, so I shudder to think about how long it would have taken to run on a single machine. [1] https://dstaley.com/2017/04/30/gigabit-in-baton-rouge/ https://dstaley.com/2017/04/30/gigabit-in-baton-rouge/
- kortilla 6y ago
  > I believe the scraper was written in Python, so I shudder to think about how long it would have taken to run on a single machine. Scraping is an embarrassingly perfect scenario for coroutines. Most asynchronous frameworks even use scraping as one of the examples. In short, it would probably be done in 15 minutes, assuming you don’t get throttled quickly. If the tool wasn’t already async capable, another 15 minutes to wrap some scraping in gevent/eventlet.
  - nijave 6y ago
    Even without async it's pretty easy to slap a concurrent.futures ThreadPool on something normally single threaded and get massive performance gains.
cranekam 6y ago
> Note: If you are comfortable with Kubernetes and scaling out clusters for big data jobs & the parallel workloads described below, godspeed! Probably my pedantic side showing through but I find reading text where ampersand is used in place of “and” really jarring (same for capitalised regular nouns). It seems somewhat common now so I guess I’ll have to get used to it.
- loopback_device 6y ago
  Kubernetes is a proper noun, and thus always to be capitalized in English, while the "&" is a ligature of the Latin "et", the word for "and". Not sure why that would be jarring, it's all fine and correct English.
  - cranekam 6y ago
    I wasn’t talking about Kubernetes being capitalised. I know that’s a proper noun and deserves capitalisation. I didn’t intend to confuse on this point. Although & means “and” they are generally used differently. & is used in places like company names where it’s part of a noun (e.g. B&Q, Smith & Wesson). “And” joins parts of a sentence. I find it jarring to use & because a) it looks like a punctuation mark and I naturally pause when reading, b) I expect to have read a noun, not a join in a sentence, and it takes some cognitive effort to re-parse the sentence using & in a way I didn’t first expect. Reading, especially quickly, relies a lot on expectation and pattern matching and I find this disrupts it. If you don’t, good for you. Obviously in informal speech people write whatever they want and it’s true that language evolves over time. But I’d argue that using & instead of and isn’t “correct”, at least by current standards — if it was we’d see this used in newspapers, books, and so on.
fxtentacle 6y ago
I strongly believe that the author never tried out his examples. One time, I wanted to process a lot of images stored on Amazon S3. So I used 3 Xeon quad-core nodes from my render farm together with a highly optimized C++ software. We peaked at 800mbit/s downstream before the entire S3 bucket went offline. Similarly, when I was doing molecular dynamics simulations, the initial state was 20 GB large, and so were the results. The issue with these computing workloads is usually IO, not the raw compute power. That's why Hadoop, for example, moves the calculations onto the storage nodes, if possible.
- papaf 6y ago
  That's why Hadoop, for example, moves the calculations onto the storage nodes, if possible You make a good point about I/O and I actually wanted to comment something along the lines of "why not Hadoop?" since the programming model looks very similar but with less mature tooling. However, now I think about it, the big win of serverless is that it is not always on. With Hadoop, you build and administer a cluster which will only be efficient if you constantly use it. This Serverless setup would suit jobs that only run occasionally.
  - fxtentacle 6y ago
    In my experience, the cloud is so slow and expensive for these tasks that even if your job only runs once per day, you're better off getting a few affordable bare metal servers. Plus, most tasks that only run occasionally tend to be not urgent, so instead of parallizing to 3000 concurrent executions, like the article suggests, you could just wait an hour instead. Serverless is only useful if you have high load spikes that are rare but super urgent. In my opinion, that combination almost never happens.
    - qeternity 6y ago
      This is us exactly. We pay around $2k for one of our analytic clusters. Hundreds of cores, over 1tb of ram, many tb of nvme. Some days when the data science team are letting a model train (on a different gpu cluster) or doing something else, the cluster sits there with a load of zero. But it’s still an order of magnitude cheaper than anything else we’ve spec’d out.
jonnypotty 6y ago
We can't even design systems to properly take advantage of multiple cpu cores yet.
rahimnathwani 6y ago
The scraping example seems poorly chosen. The original blog post describing that example is no longer online, but archive.org has a copy: https://web.archive.org/web/20180822034920/https://blog.seanssmith.com/posts/pywren-web-scraping.html https://web.archive.org/web/20180822034920/https://blog.sean... If the author just wanted to fetch pages in parallel, they could have done better than 8 hours even on their own laptop (you can run more than one chromium process at a time). The real benefit they got from using AWS Lambda is that the requests weren't throttled or ghosted by Redfin, probably because the processes were running on enough different machines, with different IP addresses.
choeger 6y ago
> Software compilation Well, no. Software compilation does not work massively parallel. Maybe parts of the optimization pipeline, but compilation of 1000 unit program (assuming your language of choice even has separate compilation) does normally require to put the units into a dependency graph (see OCaml, for instance), or puts most of the effort into the inherently serial tasks like preprocessing and linking (C++).
qeternity 6y ago
Lambda compute only seems to have been rising lately to those who never used it before. We’ve been running huge amounts of “serverless” style workloads in a Celery + RabbitMQ setup. Our workloads are fairly stable for that so bursting in a public cloud has no real value, but we regularly do batch jobs that burst capacity. And we spun up more workers as such. The author seems to think the paradigm is new (it isn’t) and claims that it hasn’t taken off massively (it has) because he incorrectly points to a number of workloads that aren’t embarrassingly parallel. On the other hand, in theory having a common runtime for these operations from a public cloud provider should enable them to keep their utilization of resources extremely high, such that it would be cheaper for us to use AWS/GCP/etc instead of rolling our own on OVH/Hetzner. But if anything, the per compute cost of FaaS is higher than it is for other compute models, which means the economics really only work for small workloads where the fixed overhead of EC2 is larger than the variable overhead of Lambda.
superjan 6y ago
It reminds me of this quote “a supercompter is a device that turns a compute bound problem into an IO bound problem” (Ken Batcher).
- bob1029 6y ago
  This is a very apt quote here. "Serverless" is basically equivalent to a supercomputer in that context, but then it goes on to exhibit latency characteristics that would be considered a non-starter for a supercomputer. Latency is one of the most important aspects of IO and is the ultimate resource underlying all of this. The lower your latency, the faster you can get the work done. When you shard your work in a latency domain measured in milliseconds-to-seconds, you have to operate with far different semantics than when you are working in a domain where a direct method call can be expected to return within nanoseconds-to-microseconds. We are talking 6 orders of magnitude or more difference in latency between local execution and an AWS Lambda. It is literally more than a million times faster to run a method that lives in warm L1 than it is to politely ask Amazon's computer to run the same method over the internet. This stuff really matters and I feel like no one is paying attention to it anymore. Your CPU can do an incredible amount of work if you stop abusing it and treating it like some worthless thing that is incapable of handling any sizeable work effort. Pay attention to the NUMA model and how cache works. Even high level languages can leverage these aspects if you focus on them. You can process tens of millions of client transactions per second on a single x86 thread if you are careful. Furthermore, the various cloud vendors have done an exceptional job at making their vanilla compute facility seem like a piece of shit too. These days, a $200/m EC2 instance feels like a bag of sand compared to a very low-end Ryzen 3300G desktop I recently built for basic lab duty. I'm not quite sure how they accomplished this, but something about cloud instances has always felt off to me. I can see how others would develop a perception that simply hosting things on one big EC2 instance would mean their application runs like shit. I am unsurprised that everyone is reaching for other options now. On-prem might be the best option if you have already optimized your stack and are now struggling with the cloud vendors' various layers of hardware indirection. Simply going from EC2 to on-prem could buy you an order of magnitude or more in speedup just by virtue of having current gen bare metal 100% dedicated to the task at hand. Obviously, this brings with it other operational and capital costs which must be justified by the business.
aquamesnil 6y ago
Stateless serverless platforms like Lambda force a data shipping architecture which hinders any workflows that require state bigger than a webpage like in the included tweet or coordination between function. The functions are short-lived and not reusable, have a limited network bandwidth to S3, and lack P2P communication which does not fit the efficient distributed programming models we know of today. Emerging stateful serverless runtimes[1] have been shown to support even big data applications whilst keeping the scalability and multi-tenancy benefits of FaaS. Combined with scalable fast stores[2][3], I believe we have here the stateful serverless platforms of tomorrow. [1] https://github.com/lsds/faasm https://github.com/lsds/faasm (can run on KNative, includes demos) [2] https://github.com/hydro-project/anna https://github.com/hydro-project/anna (KVS) [3] https://github.com/stanford-mast/pocket https://github.com/stanford-mast/pocket (Multi-tiered storage DRAM+Flash)
Lio 6y ago
This seems to come up a lot in these discussions but Moore’s Law (actually just an observation) says nothing about single thread performance. It only tells you that the number of transistors on on a silicon die will double every 18 months. If we’re still able to add parallel threads of execution at the same rate then Moore’s Law still holds.
raverbashing 6y ago
But how much is serverless efficient? Has anyone benchmarked the speed of running (let's say, on AWS) 1000x a lambda function vs. running the same function on a regular AWS instances? What about all the overhead (for example, k8s overhead, both in CPU and disk, etc) I'm afraid it would be very easy to get a repeat of this https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html https://adamdrake.com/command-line-tools-can-be-235x-faster-...
yangl1996 6y ago
Such a scheme depends heavily on whether the cloud providers can efficiently multiplex their bare-metal machines to run these jobs concurrently. Ultimately, a particular computing job takes a fixed amount of CPU-hours, so there's definitely no savings in such a scheme in terms of energy consumption or CPU-hours. At the same time, overhead comes when a job can't be perfectly parallelized: e.g. the same memory content being replicated across all executing machines, synchronization, the cost of starting a ton of short-lived processes, etc. These overhead all add to the CPU-hour and energy consumption. So, does serverless computing reduce the job completion time? Yes if the job is somewhat parallelizable. Does it save energy, money, etc.? Definitely no. The question is whether you want to make the tradeoff here: how much more energy would you afford to pay for, if you want to reduce the job completion time by half? It like batch processing vs. realtime operation. The former provides higher throughput, while the latter gives user a shorter latency. Having better cloud infrastructure (VM, scheduler, etc.) helps make this tradeoff more favorable, but the research community have just started looking at this problem.
mD5pPxMcS6fVWKE 6y ago
But Azure Functions existed for years, AWS only now got to that?

132 points by prostoalex 6y ago

It's going to take a new kind of software engineer to build these fully distributed systems. You can imagine calls for "Senior Serverless Engineers". Will conventional serverful engineers be left in the dust, or will the serverless engineers just break away and pioneer the apps on a new scale?

deleted 6y ago
[deleted]
corty 6y ago
Frankly, there isn't much engineering to be done. This only works for embarassingly parallel tasks, so you can just start a loop, distribute data, start compute, collect results. Done. For anything else, this model breaks down. Need aggregation of results? Communication among nodes? Computation subdivision that is not strictly predeterminable? Sorry, not embarrasingly parallel, won't be doable like this. You may be able to extract some embarrassingly parallel part, like compilation of independent object files, but very often you still have a longish, complex and timeconsuming serial step, like linking those object files. This kind of recognising different parts of a program is already state of the art, no need to invent a new field...
- efnx 6y ago
  Speed and parallelism are good features but the killer app of serverless is the cost savings. You only pay for the time your function is running. Traditional server applications can be rewritten “serverless” so long as your pipeline is pretty functional, ie you’re not saving critical state in memory on the server process itself.
  - threeseed 6y ago
    For some use cases the cost saving is a big deal e.g. resizing an incoming stream of images. But for the majority of apps it doesn't save that much. And it still pales in comparison to the cost of the engineers. Which often spend significantly more time to build, debug, test etc a serverless app than one they just throw on an EC2 server.
    - efnx 6y ago
      “Apps” is a pretty vague description of an application, but if you’re writing a server it’s just as easy to “just throw” the app on lambda as it is on EC2. In general I ask myself if my new server can run on a lambda and if it can’t then I reach for something else. Most of the time it can.
  - tsimionescu 6y ago
    > Traditional server applications can be rewritten “serverless” so long as your pipeline is pretty functional, ie you’re not saving critical state in memory on the server process itself. And if you're storing things in DB, how long does it take for your lambda to start up now? In a project I was on, it could easily take 3s for a new request to be served, because we had a lambda that was doing and auth check with the DB, and then a different lambda that was doing the actual application stuff with another DB. So not only would we incur the cold start cost twice, but, since our lambdas needed a nic inside a VPC to talk to the DB, the cold start cost was huge. And of course, you pay that cost for each additional concurrent connection, not just once. Of course, if we had stored everything in some other managed service, and maybe used some managed auth scheme, this would have not been a problem. AWS likes it when they get you hook, line and sinker.
    - ElFitz 6y ago
      Was the first lambda waiting on the second's result before returning? If so, you really shouldn't rely on synchronously invoking à lambda from within another lambda. Also, things stored in the global scope (in Node.js functions) are kept between different invocations of a same container. If it can help...
threeseed 6y ago
Serverless has been around for many years now. It doesn't require a new kind of software engineer. It's just another software architecture to go alongside micro services, containerisation etc. And it hasn't changed the world because (a) it's the ultimate form of vendor lock in and (b) it makes even simple apps much more complex to reason about and manage.
- arcturus17 6y ago
  > much more complex to reason about and manage I really dislike the local dev experience and deployment for serverless. But otherwise the model is pretty clear: a file is a function, data goes in, data comes out, just like any other server function. If one instance is busy, spin off a new one. What’s hard to reason about?
  - flak48 6y ago
    Yep. Inversion of control has always been the norm with whatever server/framework you choose. If you have a /myroute handler defined in express or flask or even form.php or form.cgi in Apache,you never had to write the code to make user requests trigger your handler anyway even in the old days. That's the entire point of using a server instead of listening to a socket yourself With serverless the same thing still happens with someone else managing the path from a request to a handler and back. In fact, if you ever used a cpanel host with PHP like in the good old days (110mb.com anyone?), you already used 'serverless'. You just uploaded .php files to a directory and your website just 'magically' worked.
adev_ 6y ago
> It's going to take a new kind of software engineer to build these fully distributed systems. What your are selling here is nothing else than a new proprietary-scheduler-runtime to run embarrassingly parallel jobs ( the easiest kind of parallelism) There were already plenty of solution to do that, the only difference here is that you run on AWS lambda. Why would you need an entire new type of engineer to do that ? There is nothing new here excepted buzzwords. Are engineer nowadays a script-kiddies bind to a technology there entire career ? (Tip: of course no)

mirimir 6y ago

> Basically grep for the internet. So is this a workaround for "censorship" by Google etc? And where would the crawl archives come from? Also, I wonder how this could be made usable and affordable for random individuals.

shoo 6y ago

there is a subset of embarrassingly parallel problems that are are heavily data intensive. E.g. suppose you have 100 TB of data files and you want to run some kind of keyword search over the data. If the data can be broken into 1000x100GB chunks then you can do some map-reduce-ish thing where each 100GB chunk is searched independently, then the search results from each of the 1000 chunks is aggregated. 1000x speedup! serverless! however, if you want to execute this across some fleet of rented "serverless" servers, a key factor that will influence cost and running time is (1) where is the 100 TB of data right now, (2) how are you going to copy each 100 GB chunk of the data to each serverless server, (3) how much time and money will that copy cost. I.e. in examples like this where the time required to read the data and send the data over the network is much larger than the time required to compute the data once the data is in memory, is going to be more efficient to move the code & the compute to where the data already is rather than moving the data and the code to some other physical compute device behind a bunch of abstraction layers and network pipes.

gopalv 6y ago
> there is a subset of embarrassingly parallel problems that are are heavily data intensive. There's an even smaller subset which is one-shot data access. > in examples like this where the time required to read the data and send the data over the network is much larger than the time required to compute the data once the data is in memory The annoying thing about lambda and other functional alternatives is that data-access patterns tend to be repetitive in somewhat predictable ways & there is no way to take advantage of that fact easily. However, if you don't have that & say you were reading from S3 for every pass, then lambda does look attractive because the container lifetime management is outsourced - but if you do have even temporal stickiness of data, then it helps to do your own container management & direct queries closer to previous access, rather than to entirely cold instances. If there's a thing that hadoop missed out on building into itself, it was a distributed work queue with functions with slight side-effects (i.e memoization).
- adev_ 6y ago
  > If there's a thing that hadoop missed out on building into itself, it was a distributed work queue with functions with slight side-effects (i.e memoization). Is that not named spark ? :)

alexchamberlain 6y ago

> One of the challenging parts about this today is that most software is designed to run on single machines, and parallelization may be limited to the number of machine cores or threads available locally. Depending how you look at it, I don't think most software is designed to take advantage of multiple cores, let alone multiple machines.

efnx 6y ago
Both points are true, but the author is talking about writing new software specifically for serverless compute.

lachlan-sneff 6y ago

There's no way to distribute really lightweight thunks of arbitrary code. Maybe WASM can work here, especially if you shape the standard APIs in the right way. You'd also need built-in support in tooling and compilers, where you can compile specific functions or modules into something that can run separately without actually doing that manually.

theamk 6y ago
This depends on how lightweight are those chunks. If you goal is <0.1 sec startup -- yeah, then you'll need WASM. If you are OK with 1-5 second of startup, you have a ton of options. Apache Spark uses JVM magic to send out the the raw bytecode. You can start up docker container. If you are willing to rewrite stdio, you can exec machine code under seccomp/pledge. There are even full-blown VM solutions -- Amazon Firecracker, which claims that: "Firecracker initiates user space or application code in as little as 125 ms and supports microVM creation rates of up to 150 microVMs per second per host."
- threeseed 6y ago
  And for people that don't know. AWS Lambda uses Firecracker under the hood.
j88439h84 6y ago
I dont really understand your claim. You can send code to another computer and it can execute it and send it back. What is the problem?

Uptrenda 6y ago

Except that none of these compute resources are free and the cost of cloud-based super computing is a complete rip-off. It's actually thousands of dollars cheaper to build your own super computer from used rack mount servers than it is to use AWS, Azure, or Google cloud for lengthy computations. As an example: you can buy a Dell PowerEdge M915 from ebay with 128 cores for ~$500 USD and a rack costs around the same. Five of them is 640 cores for a total cost of just 3k USD. That's 640 cores that you now own forever. Guess how much it would cost to use that many cores for a month on AWS? Well over 10k... and next month its another 10k... and so is next month... With this option you only pay for power and still have the resources for all future experiments. I think the m1000e rack can fit 8 blades in total so you could upgrade to a max 1024 cores in the future! The downside with this particular rack is that it's very, very loud. But I've run the numbers here and it's hard to beat high-core, used rack servers on a $/Ghz basis.

bpodgursky 6y ago
None of this math is correct. You absolutely cannot get 128 cores for $500. Please link to a single valid-looking eBay offer at that amount (you may have seen 128GB and thought it was 128 cores). I see 32 cores for $500, at best. Second, 640 cores on an m4.large for 3 years upfront (the equivalent purchase) is $8755/mo, not well over 10k. Third, you really underestimate electricity costs if you're running a server 24/7 at all cores. Last, and most importantly, most people simply don't run servers 24/7. They run batch computations for five hours a week, or a day, and then spend a week or two crunching the numbers. There's some case to make that for certain institutions with very particular compute needs, running on-premise might make short-term financial sense (let's not even get started on labor costs). But it's really inaccurate to call cloud computing a "rip-off" for anything but the niche-est customers.
- sigstoat 6y ago
  > Third, you really underestimate electricity costs if you're running a server 24/7 at all cores. not to mention the cooling, or the per square foot cost of renting the space for this hardware. anything close to what's being described here is going to need some sound deadening (be it distance or material) from humans.

vsskanth 6y ago

Yeah but how expensive will it be ? Some numbers would be nice for something like hyper parameter tuning

tlarkworthy 6y ago

Don't overcomplicate it. Xargs and curl is often enough to drive big, ad hoc jobs.

kristopolous 6y ago
These things only have a handful of customers in the world. Datasets that are tens of gigabytes, or maybe 100mil records or so...this really covers most things. And for every 1 thing it doesn't, there's 20 more claimed that a single machine using simple tools could handle just fine. Being able to detect when things have been processed, have a way to set dirty flags, prioritize things, have regions of interest, be able to have re-entrant processing, caching parts of the pipeline and having nuanced rules for invalidating it, these in my mind are kinda basic things here. When they aren't done, sure, someone will need giant resources because they're doing foolish things. But that's literally the only reason. Substituting money for sense is an old hack.
- logicchains 6y ago
  A nice paper about that idea: https://www.usenix.org/system/files/conference/hotos15/hotos15-paper-mcsherry.pdf https://www.usenix.org/system/files/conference/hotos15/hotos...
  - kristopolous 6y ago
    This is like an infoworld quality article in LaTeX. I don't know why this is beneficial. The LaTeX format scares me away most of the time. I think "oh god, I don't have all afternoon to study an academic document!" This thing is a 3 minute read
- necovek 6y ago
  Opening your mirrorless camera SD card with images and the image thumbnailer takes... Forever? Doing a facial search on it? Matching a rhythm picked up by the mic to your local music collection? Hashing and/or encryption of data. There's plenty of desktop-like use cases that would benefit from massively parallel computation, but network (or even IO) bandwidth is currently going to be the limiting factor. IOW, we are not there yet! Currently, we can parallelize tasks which are low on data and high on computation. So how can we expand the IO bandwidth for everyone, even desktop or mobile users?

ChuckMcM 6y ago

Hilariously, in 2008 Google was hugely resistant to this idea. DARPA had just put out an RFP for a new way of running computationally dependent tasks that currently ran on super computers on a 'shared nothing' architecture (which is what Google ran at the time (and I believe they still do). I had done some research in that space when I was at NetApp looking at decomposing network attached storage services into a shared nothing architecture so I had some idea of the kinds of problems that were the "hard bits" in getting it right. I recall pointing out to Platform's management that if Google could provide an infrastructure that solved these sorts of problems with massive parallelism that currently required specialized switching fabrics and massive memory sharing we would have something very special. But at the time it was a non-starter, way too much money to be made in search ads to bother with building a system for something like the 200 customers in the world total. I didn't care one way or the other if Google did it so after running at the wall of "under 2s" a couple of times I just said "fine, your loss."

dstaley 6y ago

I'm a huge fan of using Lambda to perform hundreds of thousands of discrete tasks in a fraction of the time it'd take to perform those same tasks locally. A while back I used Lambda and SQS to cross check 168,000 addresses with my ISP's gigabit availability tool.[1] If I recall correctly each check took about three seconds, but running all 168,000 checks on Lambda only took a handful of minutes. I believe the scraper was written in Python, so I shudder to think about how long it would have taken to run on a single machine. [1] https://dstaley.com/2017/04/30/gigabit-in-baton-rouge/ https://dstaley.com/2017/04/30/gigabit-in-baton-rouge/

kortilla 6y ago
> I believe the scraper was written in Python, so I shudder to think about how long it would have taken to run on a single machine. Scraping is an embarrassingly perfect scenario for coroutines. Most asynchronous frameworks even use scraping as one of the examples. In short, it would probably be done in 15 minutes, assuming you don’t get throttled quickly. If the tool wasn’t already async capable, another 15 minutes to wrap some scraping in gevent/eventlet.
- nijave 6y ago
  Even without async it's pretty easy to slap a concurrent.futures ThreadPool on something normally single threaded and get massive performance gains.

cranekam 6y ago

> Note: If you are comfortable with Kubernetes and scaling out clusters for big data jobs & the parallel workloads described below, godspeed! Probably my pedantic side showing through but I find reading text where ampersand is used in place of “and” really jarring (same for capitalised regular nouns). It seems somewhat common now so I guess I’ll have to get used to it.

loopback_device 6y ago
Kubernetes is a proper noun, and thus always to be capitalized in English, while the "&" is a ligature of the Latin "et", the word for "and". Not sure why that would be jarring, it's all fine and correct English.
- cranekam 6y ago
  I wasn’t talking about Kubernetes being capitalised. I know that’s a proper noun and deserves capitalisation. I didn’t intend to confuse on this point. Although & means “and” they are generally used differently. & is used in places like company names where it’s part of a noun (e.g. B&Q, Smith & Wesson). “And” joins parts of a sentence. I find it jarring to use & because a) it looks like a punctuation mark and I naturally pause when reading, b) I expect to have read a noun, not a join in a sentence, and it takes some cognitive effort to re-parse the sentence using & in a way I didn’t first expect. Reading, especially quickly, relies a lot on expectation and pattern matching and I find this disrupts it. If you don’t, good for you. Obviously in informal speech people write whatever they want and it’s true that language evolves over time. But I’d argue that using & instead of and isn’t “correct”, at least by current standards — if it was we’d see this used in newspapers, books, and so on.

fxtentacle 6y ago

I strongly believe that the author never tried out his examples. One time, I wanted to process a lot of images stored on Amazon S3. So I used 3 Xeon quad-core nodes from my render farm together with a highly optimized C++ software. We peaked at 800mbit/s downstream before the entire S3 bucket went offline. Similarly, when I was doing molecular dynamics simulations, the initial state was 20 GB large, and so were the results. The issue with these computing workloads is usually IO, not the raw compute power. That's why Hadoop, for example, moves the calculations onto the storage nodes, if possible.

papaf 6y ago
That's why Hadoop, for example, moves the calculations onto the storage nodes, if possible You make a good point about I/O and I actually wanted to comment something along the lines of "why not Hadoop?" since the programming model looks very similar but with less mature tooling. However, now I think about it, the big win of serverless is that it is not always on. With Hadoop, you build and administer a cluster which will only be efficient if you constantly use it. This Serverless setup would suit jobs that only run occasionally.
- fxtentacle 6y ago
  In my experience, the cloud is so slow and expensive for these tasks that even if your job only runs once per day, you're better off getting a few affordable bare metal servers. Plus, most tasks that only run occasionally tend to be not urgent, so instead of parallizing to 3000 concurrent executions, like the article suggests, you could just wait an hour instead. Serverless is only useful if you have high load spikes that are rare but super urgent. In my opinion, that combination almost never happens.
  - qeternity 6y ago
    This is us exactly. We pay around $2k for one of our analytic clusters. Hundreds of cores, over 1tb of ram, many tb of nvme. Some days when the data science team are letting a model train (on a different gpu cluster) or doing something else, the cluster sits there with a load of zero. But it’s still an order of magnitude cheaper than anything else we’ve spec’d out.

jonnypotty 6y ago

We can't even design systems to properly take advantage of multiple cpu cores yet.

rahimnathwani 6y ago

The scraping example seems poorly chosen. The original blog post describing that example is no longer online, but archive.org has a copy: https://web.archive.org/web/20180822034920/https://blog.seanssmith.com/posts/pywren-web-scraping.html https://web.archive.org/web/20180822034920/https://blog.sean... If the author just wanted to fetch pages in parallel, they could have done better than 8 hours even on their own laptop (you can run more than one chromium process at a time). The real benefit they got from using AWS Lambda is that the requests weren't throttled or ghosted by Redfin, probably because the processes were running on enough different machines, with different IP addresses.

choeger 6y ago

> Software compilation Well, no. Software compilation does not work massively parallel. Maybe parts of the optimization pipeline, but compilation of 1000 unit program (assuming your language of choice even has separate compilation) does normally require to put the units into a dependency graph (see OCaml, for instance), or puts most of the effort into the inherently serial tasks like preprocessing and linking (C++).

qeternity 6y ago

Lambda compute only seems to have been rising lately to those who never used it before. We’ve been running huge amounts of “serverless” style workloads in a Celery + RabbitMQ setup. Our workloads are fairly stable for that so bursting in a public cloud has no real value, but we regularly do batch jobs that burst capacity. And we spun up more workers as such. The author seems to think the paradigm is new (it isn’t) and claims that it hasn’t taken off massively (it has) because he incorrectly points to a number of workloads that aren’t embarrassingly parallel. On the other hand, in theory having a common runtime for these operations from a public cloud provider should enable them to keep their utilization of resources extremely high, such that it would be cheaper for us to use AWS/GCP/etc instead of rolling our own on OVH/Hetzner. But if anything, the per compute cost of FaaS is higher than it is for other compute models, which means the economics really only work for small workloads where the fixed overhead of EC2 is larger than the variable overhead of Lambda.

superjan 6y ago

It reminds me of this quote “a supercompter is a device that turns a compute bound problem into an IO bound problem” (Ken Batcher).

bob1029 6y ago
This is a very apt quote here. "Serverless" is basically equivalent to a supercomputer in that context, but then it goes on to exhibit latency characteristics that would be considered a non-starter for a supercomputer. Latency is one of the most important aspects of IO and is the ultimate resource underlying all of this. The lower your latency, the faster you can get the work done. When you shard your work in a latency domain measured in milliseconds-to-seconds, you have to operate with far different semantics than when you are working in a domain where a direct method call can be expected to return within nanoseconds-to-microseconds. We are talking 6 orders of magnitude or more difference in latency between local execution and an AWS Lambda. It is literally more than a million times faster to run a method that lives in warm L1 than it is to politely ask Amazon's computer to run the same method over the internet. This stuff really matters and I feel like no one is paying attention to it anymore. Your CPU can do an incredible amount of work if you stop abusing it and treating it like some worthless thing that is incapable of handling any sizeable work effort. Pay attention to the NUMA model and how cache works. Even high level languages can leverage these aspects if you focus on them. You can process tens of millions of client transactions per second on a single x86 thread if you are careful. Furthermore, the various cloud vendors have done an exceptional job at making their vanilla compute facility seem like a piece of shit too. These days, a $200/m EC2 instance feels like a bag of sand compared to a very low-end Ryzen 3300G desktop I recently built for basic lab duty. I'm not quite sure how they accomplished this, but something about cloud instances has always felt off to me. I can see how others would develop a perception that simply hosting things on one big EC2 instance would mean their application runs like shit. I am unsurprised that everyone is reaching for other options now. On-prem might be the best option if you have already optimized your stack and are now struggling with the cloud vendors' various layers of hardware indirection. Simply going from EC2 to on-prem could buy you an order of magnitude or more in speedup just by virtue of having current gen bare metal 100% dedicated to the task at hand. Obviously, this brings with it other operational and capital costs which must be justified by the business.

aquamesnil 6y ago

Stateless serverless platforms like Lambda force a data shipping architecture which hinders any workflows that require state bigger than a webpage like in the included tweet or coordination between function. The functions are short-lived and not reusable, have a limited network bandwidth to S3, and lack P2P communication which does not fit the efficient distributed programming models we know of today. Emerging stateful serverless runtimes[1] have been shown to support even big data applications whilst keeping the scalability and multi-tenancy benefits of FaaS. Combined with scalable fast stores[2][3], I believe we have here the stateful serverless platforms of tomorrow. [1] https://github.com/lsds/faasm https://github.com/lsds/faasm (can run on KNative, includes demos) [2] https://github.com/hydro-project/anna https://github.com/hydro-project/anna (KVS) [3] https://github.com/stanford-mast/pocket https://github.com/stanford-mast/pocket (Multi-tiered storage DRAM+Flash)

Lio 6y ago

This seems to come up a lot in these discussions but Moore’s Law (actually just an observation) says nothing about single thread performance. It only tells you that the number of transistors on on a silicon die will double every 18 months. If we’re still able to add parallel threads of execution at the same rate then Moore’s Law still holds.

raverbashing 6y ago

But how much is serverless efficient? Has anyone benchmarked the speed of running (let's say, on AWS) 1000x a lambda function vs. running the same function on a regular AWS instances? What about all the overhead (for example, k8s overhead, both in CPU and disk, etc) I'm afraid it would be very easy to get a repeat of this https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html https://adamdrake.com/command-line-tools-can-be-235x-faster-...

yangl1996 6y ago

Such a scheme depends heavily on whether the cloud providers can efficiently multiplex their bare-metal machines to run these jobs concurrently. Ultimately, a particular computing job takes a fixed amount of CPU-hours, so there's definitely no savings in such a scheme in terms of energy consumption or CPU-hours. At the same time, overhead comes when a job can't be perfectly parallelized: e.g. the same memory content being replicated across all executing machines, synchronization, the cost of starting a ton of short-lived processes, etc. These overhead all add to the CPU-hour and energy consumption. So, does serverless computing reduce the job completion time? Yes if the job is somewhat parallelizable. Does it save energy, money, etc.? Definitely no. The question is whether you want to make the tradeoff here: how much more energy would you afford to pay for, if you want to reduce the job completion time by half? It like batch processing vs. realtime operation. The former provides higher throughput, while the latter gives user a shorter latency. Having better cloud infrastructure (VM, scheduler, etc.) helps make this tradeoff more favorable, but the research community have just started looking at this problem.

mD5pPxMcS6fVWKE 6y ago

But Azure Functions existed for years, AWS only now got to that?