4 ms·

Laptop to Lambda: Outsourcing Everyday Jobs to Thousands of Transient Containers

147 points by mlerner 5y ago

haolez 5y ago
This could be very useful for quantum chemistry simulations, which are generally parallelizable and very CPU intensive. If gg gets tweaked to support MPI, this niche could have a breakthrough!
deleted 5y ago
[deleted]
z3ncyberpunk 5y ago
Disgustingly wasteful
neolog 5y ago
Hi Micah, I'd like to follow these posts but I don't like signing up. Would you mind adding an Atom/RSS feed?
- MarkSweep 5y ago
  There is a feed. It is title “untitled” though so it may be hard to find in your feed reader after you add it: https://www.micahlerner.com/feed.xml https://www.micahlerner.com/feed.xml
  - gumby 5y ago
    Good find -- I searched in the page source and there was no reference to it.
- gumby 5y ago
  I filed a github issue asking the author to enable the RSS feed which their web tool (Hugo) has built in.
  - mlerner 5y ago
    Thanks!
- mlerner 5y ago
  Thanks for the feedback! Does this Atom feed work for you? https://www.micahlerner.com/feed.xml https://www.micahlerner.com/feed.xml
  - neolog 5y ago
    Yep, thanks.
chubot 5y ago
2019 thread on the paper: https://news.ycombinator.com/item?id=20433315 https://news.ycombinator.com/item?id=20433315 Another review: https://buttondown.email/nelhage/archive/papers-i-love-gg/ https://buttondown.email/nelhage/archive/papers-i-love-gg/
keithwinstein 5y ago
Thank you for this nice writeup! This paper was led by my student Sadjad Fouladi (https://sadjad.org/ https://sadjad.org/), part of a broader theme of coercing a "purely functional-ish" design onto everyday applications. There's a less academic-ese version with a few extended results that was published in ;login: magazine (https://www.usenix.org/system/files/login/articles/login_fall19_02_fouladi.pdf https://www.usenix.org/system/files/login/articles/login_fal...). There was also a good analysis here (https://buttondown.email/nelhage/archive/papers-i-love-gg/ https://buttondown.email/nelhage/archive/papers-i-love-gg/) and don't miss https://buttondown.email/nelhage/archive/http-pipelining-s3-and-gg/ https://buttondown.email/nelhage/archive/http-pipelining-s3-... . Some of Sadjad's other work has included: - ExCamera, which somewhat kicked off the trend of "fire up 4,000 lambda workers in a burst, all working on one job" -- for things like making a neural network search a video frame-by-frame, video compression in parallel at sub-GOP granularity, etc. (https://news.ycombinator.com/item?id=16197253 https://news.ycombinator.com/item?id=16197253) - Salsify, which reused the "purely functional" video codec from ExCamera to improve WebRTC/Zoom-style live video (https://news.ycombinator.com/item?id=16964112 https://news.ycombinator.com/item?id=16964112 , https://news.ycombinator.com/item?id=20794541 https://news.ycombinator.com/item?id=20794541). Sadjad is giving an Applied Networking Research Prize talk about this work at IETF tomorrow. - 3D ray-tracing (running PBRT on thousands of Lambdas, sending rays across the network), SMT/SAT solving, etc. We're working to extend this line of work towards a more general, Wasm-based, "purely functional" operating system where most computations operate on content-addressed data and are content-addressed themselves, and determinism and reproducibility is properties guaranteed by the OS. Sort of analogous to how the operating systems of today (try to) guarantee memory isolation between processes. Imagine, e.g., a Git repository where you could represent the fact that "blob <x> is the result of running computation <y> given tree <z> as input," and anybody can verify that result, or rebase the computation to run on top of their own input. If you're interested in this general area, please consider doing a PhD at Stanford and/or get in touch -- I'm hiring.
- boulos 5y ago
  Hi, Keith! Glad to see you're still enjoying hipster compute :). How/what do you think about Cloudflare workers, fly.io, and similar "run pure-ish functions anywhere"? I no longer have any skin in the game, but it seems to me that "ignoring locality" just means having to reinvent locality later on.
JZL003 5y ago
My work uses GCP not AWS so I've been experimenting with google cloud run (it's actually parallelizing R code so need the docker container infra). My only problem is that I have very bursty useage and the auto-scaling is too slow. I made one attempt [1] to encourage larger allocation but don't know another way. Do people have experience with this [1] Slightly costly but ~ 5minutes before I need it, I set the minimum instance size to a larger number so it starts ramping up, then when I'm done I lower it
- nuclearnice1 5y ago
  > google cloud run (it's actually parallelizing R code so need the docker container infra). I'm not sure how the docker container support of Google Cloud differs from AWS. But FYI AWS Lambda supports containers as of Dec 2020.[1] The scaling on AWS [2] might be more to your liking. [1] https://aws.amazon.com/blogs/aws/new-for-aws-lambda-container-image-support/ https://aws.amazon.com/blogs/aws/new-for-aws-lambda-containe... [2] https://docs.aws.amazon.com/lambda/latest/dg/invocation-scaling.html https://docs.aws.amazon.com/lambda/latest/dg/invocation-scal...

147 points by mlerner 5y ago

This could be very useful for quantum chemistry simulations, which are generally parallelizable and very CPU intensive. If gg gets tweaked to support MPI, this niche could have a breakthrough!

deleted 5y ago

[deleted]

z3ncyberpunk 5y ago

Disgustingly wasteful

neolog 5y ago

Hi Micah, I'd like to follow these posts but I don't like signing up. Would you mind adding an Atom/RSS feed?

MarkSweep 5y ago
There is a feed. It is title “untitled” though so it may be hard to find in your feed reader after you add it: https://www.micahlerner.com/feed.xml https://www.micahlerner.com/feed.xml
- gumby 5y ago
  Good find -- I searched in the page source and there was no reference to it.
gumby 5y ago
I filed a github issue asking the author to enable the RSS feed which their web tool (Hugo) has built in.
- mlerner 5y ago
  Thanks!
mlerner 5y ago
Thanks for the feedback! Does this Atom feed work for you? https://www.micahlerner.com/feed.xml https://www.micahlerner.com/feed.xml
- neolog 5y ago
  Yep, thanks.

chubot 5y ago

2019 thread on the paper: https://news.ycombinator.com/item?id=20433315 https://news.ycombinator.com/item?id=20433315 Another review: https://buttondown.email/nelhage/archive/papers-i-love-gg/ https://buttondown.email/nelhage/archive/papers-i-love-gg/

keithwinstein 5y ago

Thank you for this nice writeup! This paper was led by my student Sadjad Fouladi (https://sadjad.org/ https://sadjad.org/), part of a broader theme of coercing a "purely functional-ish" design onto everyday applications. There's a less academic-ese version with a few extended results that was published in ;login: magazine (https://www.usenix.org/system/files/login/articles/login_fall19_02_fouladi.pdf https://www.usenix.org/system/files/login/articles/login_fal...). There was also a good analysis here (https://buttondown.email/nelhage/archive/papers-i-love-gg/ https://buttondown.email/nelhage/archive/papers-i-love-gg/) and don't miss https://buttondown.email/nelhage/archive/http-pipelining-s3-and-gg/ https://buttondown.email/nelhage/archive/http-pipelining-s3-... . Some of Sadjad's other work has included: - ExCamera, which somewhat kicked off the trend of "fire up 4,000 lambda workers in a burst, all working on one job" -- for things like making a neural network search a video frame-by-frame, video compression in parallel at sub-GOP granularity, etc. (https://news.ycombinator.com/item?id=16197253 https://news.ycombinator.com/item?id=16197253) - Salsify, which reused the "purely functional" video codec from ExCamera to improve WebRTC/Zoom-style live video (https://news.ycombinator.com/item?id=16964112 https://news.ycombinator.com/item?id=16964112 , https://news.ycombinator.com/item?id=20794541 https://news.ycombinator.com/item?id=20794541). Sadjad is giving an Applied Networking Research Prize talk about this work at IETF tomorrow. - 3D ray-tracing (running PBRT on thousands of Lambdas, sending rays across the network), SMT/SAT solving, etc. We're working to extend this line of work towards a more general, Wasm-based, "purely functional" operating system where most computations operate on content-addressed data and are content-addressed themselves, and determinism and reproducibility is properties guaranteed by the OS. Sort of analogous to how the operating systems of today (try to) guarantee memory isolation between processes. Imagine, e.g., a Git repository where you could represent the fact that "blob <x> is the result of running computation <y> given tree <z> as input," and anybody can verify that result, or rebase the computation to run on top of their own input. If you're interested in this general area, please consider doing a PhD at Stanford and/or get in touch -- I'm hiring.

boulos 5y ago
Hi, Keith! Glad to see you're still enjoying hipster compute :). How/what do you think about Cloudflare workers, fly.io, and similar "run pure-ish functions anywhere"? I no longer have any skin in the game, but it seems to me that "ignoring locality" just means having to reinvent locality later on.

JZL003 5y ago

My work uses GCP not AWS so I've been experimenting with google cloud run (it's actually parallelizing R code so need the docker container infra). My only problem is that I have very bursty useage and the auto-scaling is too slow. I made one attempt [1] to encourage larger allocation but don't know another way. Do people have experience with this [1] Slightly costly but ~ 5minutes before I need it, I set the minimum instance size to a larger number so it starts ramping up, then when I'm done I lower it

nuclearnice1 5y ago
> google cloud run (it's actually parallelizing R code so need the docker container infra). I'm not sure how the docker container support of Google Cloud differs from AWS. But FYI AWS Lambda supports containers as of Dec 2020.[1] The scaling on AWS [2] might be more to your liking. [1] https://aws.amazon.com/blogs/aws/new-for-aws-lambda-container-image-support/ https://aws.amazon.com/blogs/aws/new-for-aws-lambda-containe... [2] https://docs.aws.amazon.com/lambda/latest/dg/invocation-scaling.html https://docs.aws.amazon.com/lambda/latest/dg/invocation-scal...