15 ms·
Serverless Map/Reduce
- stcredzero 10y agoI wonder if something like AWS Lambda could be applied to multiplayer games? It seems like game-loop based games would be a good domain for such a programming model. The entire game could be expressed as a function that turns tick N into tick N+1. Such a function would be composed of many other functions, of course. So for example, there would also be a function that took as an argument the player at time N and gave the player at time N+1. Such a model would allow infrastructure developers to abstract away most of the concerns around networking, collisions, security, etc., and let game developers concentrate their efforts on simply making the game. I currently have a game server cluster written in Golang, where the locations are instantiated with an idempotent request operation. It doesn't matter if a particular location-instance exists at a particular moment. It's sufficient for the "master control" server to only approximately know the loads of the different cluster server processes. My experience leads me to believe that something like AWS Lambda, but optimized for implementing game loops would work well, so long as game developers could get their heads around pure functional programming and implement with soft real-time requirements in mind. (John Carmack already advocates the use of pure functions, and game devs in general already do the latter.) http://www.emergencevector.com http://www.emergencevector.com
- jedberg 10y agoThis was exactly the demo they gave and their AWS Dev Day in SF in July. Using Lambda and the IOT gateway to run a massively multiplayer game (they had everyone in the room play from the web via their phone).
- stcredzero 10y agoBut was that just a demo, or are they going to make a product out of that?
- jedberg 10y agoI mean it was a demo of a product they created using Lambda and the IOT framework. It was a working game, it just wasn't very fun. :) The code was on GitHub but I can't seem to find it.
- z3t4 10y agoWhat is the latency overhead in AWS Lambda ?
- TheHydroImpulse 10y agoLatency is around 80ms-200ms per invocation from what I've seen. Definitely not suitable for any real-time game.
- stcredzero 10y agoToo large for me to want to make a game in it. But if you made a specialized version that had an actual game loop underneath it, there's some potential there.
- amelius 10y agoIf it doesn't run on a server, then what does this plumbing-work run on? Clickbait name?
- deleted 10y ago[deleted]
- vog 10y agoThe term "Serverless" means that you don't manage the servers, neither hardware nor virtual machines. Instead, you use exclusively AWS lambda and/or similar services. I find that term confusing, too, but it seems to be well-established by now, so we have to live with that. See also: http://martinfowler.com/articles/serverless.html http://martinfowler.com/articles/serverless.html
- g_delgado14 10y agoDoesn't necessarily have to be through AWS. In fact, the Serverless Framework folks are working hard to make their framework suitable for other serverless service providers.
- vog 10y ago> Doesn't necessarily have to be through AWS Please refrain from strawman arguments. I wrote "AWS lambda and/or similar services". In the Fowler article, you find a nice classification of these services: - BaaS (Backend as a Service) - FaaS (Funciton as a Service)
- g_delgado14 10y agoSorry :P read "exclusively AWS", and stopped reading!
- luhn 10y agoOh come on. Is it still not accepted that "serverless" is the colloquial name for AWS Lambda and comparable services? Stop trying to make "FaaS" happen. It's not going to happen.
- eistrati 10y agoSaw the presentation last week at ServerlessConf in London and it really looks very promising. The cost behind this solution is what will really make me check this out :) P.S. Quoting the author: "As you can see for these queries, the reference implementation performs reasonably well; it's nowhere near Redshift performance for the same queries, but for the price it really can't be beat today"
- lucd 10y agoHow does it compare to 3 years old Joyent's Manta ? AFAIK it was especially designed for this kind of purposes. The processing is made directly on the servers storing the data..
- jahewson 10y agoManta is pretty similar to Elastic MapReduce, which also runs the computation on the same node as the data. So it compares pretty much the same as EMR.
- ralusek 10y agoThis is a screenshot of my google search from 2 days ago: http://i.imgur.com/BNAcSsn.png http://i.imgur.com/BNAcSsn.png I've been using Lambda quite a bit, I think it's SO amazingly useful. Tasks that are highly parallelized and CPU intensive can literally be infinitely scaled out. I find it weird that their poster child use case is still always a reactive event like watching S3 and formatting images. There are so many use cases for directly invoking a lambda directly from your code. Imagine a case where you had to parse a million documents with a relatively expensive computation, let's say 250MS per document. Maybe you have a solid machine with a few cores that's running your server, but even then you can't have the server cpu locked for so long, so naturally you'd need some sort of worker server set up. With a good machine and multiple cores, maybe you get 8 running at once. With a lambda, you can forego the worker server altogether. Just invoke a million lambdas directly from your application server, completely parallelized. Theoretically, you've taken something that would take 70 hours and had it run in 250ms without having to set up any additional infrastructure.
- teej 10y agoI just restructured a job to be parallel and run on Lambda and I couldn't be happier. Glad to see this idea is gaining mindshare. I found that it was easy to test, easy to debug, easy to maintain, not very expensive, and just kinda worked. The only caveat for those considering taking this path - AWS has an account-wide limit of 100 concurrent lambdas running at once. There's no way to know how many are currently running unless you keep track of it yourself. You'll only find out when you try to kick off a job and it get's throttled. I haven't contacted support yet to find out how hard it is for the limit to get raised or what they'll raise it to.
- ronack 10y agoWe had it raised to several thousand concurrent requests with no questions asked. YMMV based on age of account, amount of spend, etc.
- ceejayoz 10y agoYeah, most of the limits are there just to prevent "whoops, we fired up 500 $2/hour servers and left them running all month, can you refund us?" sort of newbie situations. Very few are technological and AWS has been happy to bump them up for me in each occasion I've had a need.
- c-smile 10y agoAbout the site: quite hard to read - almost white text on white background.
- pjc50 10y agoWorse than that, doesn't appear for me at all until I use Readability Mode.
- elcct 10y agoIs there any AWS Lambda equivalent that could be deployed on bare metal?
- cpollard0 10y agoDocker
- eudoxus 10y agoThat's a horrible and flat out wrong answer. Docker simply provides a transferable containerized execution environment for processes with a simple push pull workflow. Lambda is a fully elastic infrastructure you can build, deploy, and execute functions. Simply having docker-engine and docker-cli installed on my machine doesn't give me anything close to lambda.
- deleted 10y ago[deleted]
- deleted 10y ago[deleted]
- eudoxus 10y agoI've recently come across funktion[1] which is designed to run on kubernetes (which can run on baremetal). Also there is OpenWhisk[2] which is open sourced by IBM. Probably alot more but those are the ones I know off the top of my head. [1] - https://github.com/fabric8io/funktion https://github.com/fabric8io/funktion. [2] - https://developer.ibm.com/openwhisk/ https://developer.ibm.com/openwhisk/
- radiowave 10y agoNo first hand experience with it, but I believe Joyent's Manta system can be, across a cluster of SmartOS machines.
- stochastician 10y agoAuthor of the figures used in the blog post here. We wrote https://github.com/ericmjonas/pywren https://github.com/ericmjonas/pywren somewhat on a lark, because it seemed to fit well with our research goals and it's fun to push systems to their limit. I'm now a total serverless convert! I'd love more collaborators and feedback, the goal is to make these sorts of computations as easy as possible for python developers, especially on the scientific computing side of things.
- dnackoul 10y agoDoes anyone have experience building mobile backends in Lambda? I was looking at an API Gateway / Lambda / Amazon RDS stack for building a central data store and was wondering what people's experience with that setup is?
- danso 10y agoOT: I teach computational methods and even as much as I dislike teaching/conflating it with web dev, I have included "let's build a web app" because students like building and deploying a thing, and because Heroku has a free tier. I've considered the possibility of having students do things on AWS (beyond web dev), including Lambda, and just expensing the costs. It seems feasible to quickly set up every student with controlled access via IAM...but is there a way to set up rate-limiting, ideally through a policy? That is, shut an IAM down if a student accidentally invokes a million processes? Or, for that matter, limiting the storage capacity of a S3 bucket?
- tokenizerrr 10y agoAWS has a free tier for a year
- rspeer 10y agoIt becomes a problem when you're a student and people expect you to use AWS's free tier for different projects in two different years.
- sorenbs 10y agoYou can easily set up multiple accounts with same credit card and they will all be eligible for free tier. I use gmails email+alias@gmail.com feature for this.
- autotune 10y agoI would just set up a new account for each student, have them use their own billing info, have them use the free tier, teach them how to set up billing alerts, and let them go to town. They're going to need to learn to take cost into account when working at a real job with AWS so this is the best way to teach them to take accountability.
- PetahNZ 10y agoI'm not sure its reasonable to expect your students to have a credit card.
- partycoder 10y agoI do not agree with term serverless. Amazon Lambda is a service, therefore there is a server involved. It's like saying deathless meat, because someone else killed the animal you are consuming.
- deleted 10y ago[deleted]
- kpil 10y agoExactly. The cloud is just someone else's computer. It's a genuine business case, as running a lean and secure server farm is no child's play, but it's also oversold so much that my eyes hurts. Actual serverless computing would be implemented as p2p. Possibly interesting in a typical EvilCorpesque large corporation. Besides, Seti@home (Boinc) have been doing serverless computing for almost 20 years and it wasn't the first implementations.
- inopinatus 10y agoAchieving real-world service objectives via cloud infrastructure requires a different approach to architecture, data management, process design, service ownership, availability, cost management, the list goes on. Some uses of cloud services are superficially comparable to old-school bureau computing. But to say it is "just someone else's computer" is trite, grossly misleading and downright bad advice, because treating it as such will get you burned.
- kpil 10y agoI think we are on the same page here. How about - it's nothing more to it than someone else computer. With all the risks and benefits. If you have the right workload, it is super awesome. If you have a reasonably matching workload and you can spend spend time on all the things you counted, it's also probably good but it's far from a clear-cut certainty. If you have heap of legacy systems you will basically need to rewrite a lot it to make it work, and then it might not be worth it at all. This is the case that is oversold.
- 10y ago
- plandis 10y agoI've always had one big question about Lambda. Is it really worth the cost you get for the convienience of it? Is anyone using it in production that can comment?
- coredog64 10y agoI'm not using it for anything like this, but I am using it for production. I scrape CloudWatch metrics into our standard time series metrics repo. Total cost for every service I care about (ELB, ASG, RDS, ElasticSearch, SNS, and SQS) has been about a dollar for the last 3 months.
- edblarney 10y agoI think you can do the math yourself - the costs are published. FYI - we did some experiments and the limiting factor was latency. 250-300ms on average, you have to go through their API feature as well, and that's part of the delay. But worse - Lamda's that have not been called for several minutes (I'm assuming they are not 'hot') often take several seconds, up to 5s to be called. So it creates a problem for intermittent traffic. If that kind of latency is acceptable to you, it might work for you so long as the cost equation is right. I think some other people had issues with versioning, it's a problem we didn't go far enough to observe.
- twagner 10y agoQuick clarification: No need to go through API Gateway if you don't actually need the https endpoint - all AWS SDKs can hit Lambda's REST APIs directly, which also reduces p50 latency.
- willcodeforfoo 10y agoI wonder if Amazon will ever open Lambda up to any Docker image? (I know it's possible to run binaries, but its a bit of a pain to compile with the Amazon AMI, etc.) Being able to have a bunch of `docker run` with any image would be pretty powerful.
- irq-1 10y agoCheck out Hyper. https://www.hyper.sh/ https://www.hyper.sh/ "With HyperContainer, we make the performance/overhead of virtualized container similar to linux container --- 130ms launch time, and sharing the read-only part of memory (kernel and init)." -gnawux https://news.ycombinator.com/item?id=12873089 https://news.ycombinator.com/item?id=12873089
- initdaemon 10y agoYep, and "hyper func" in the roadmap
- IanCal 10y agoThis has me exceptionally excited!
- twagner 10y agoYes. First step was https://aws.amazon.com/blogs/aws/new-amazon-linux-container-image-for-cloud-and-on-premises-workloads/ https://aws.amazon.com/blogs/aws/new-amazon-linux-container-.... Layering Lambda's image on top of that to assist people building and testing is definitely on our roadmap.
- thinkloop 10y agoThe article counts characters in documents stored on S3 - which makes sense since S3 is great for storing documents and can handle unlimited concurrency, priced per usage. But what's the solution for structured data? DynamoDB is the obvious main candidate, but it's billed by hour and high concurrency is very expensive, requiring complicated temporary increases and decreases of concurrency that are hard to predict. Is there a good solution for running massively parallel lamdas on stuctured data?
- deleted 10y ago[deleted]
- ranman 10y agoIf you're doing any sort of table scan op then DDB perf/cost will be less than stellar. If you have an index / range key it works really (like really) well -- even in massively parallel situations. If you're dealing with a TON (5+ TB) of data I recommend heading in to RDS, BQ, or redshift.
- thinkloop 10y agoIt's less the total size of the data I'm worried about and more the concurrency. For example, say I had a process that retrieved 1000 tiny records (using index query) and ran some cpu-intensive calculation on them, and I wanted to run 1000 of those processes simultaneously to reduce into a final result. This would require tuning dynamo to thousands of concurrent reads (and maybe writes, depending on the process), then scaling it back down after the operation because it is very costly and priced by hour. This makes it complicated and expensive on dynamo. It seems the only storage services compatible with variable unlimited bursts of concurrency are S3 and SimpleDB. S3 comes with many problems for handling structured data (no update of records only replace, locking, listing items is slow/costly, etc.). SimpleDB is no longer being iterated, is limited to 10gb per domain, and looks like it's being slowly phased out. It seems like massively parallel lambdas depend on few fetches of large blobs of data - which is basically batch-processing EMR-style, or better suited to redshift. Not something that opens the door for novel use-cases. I would have really liked for dynamodb to be more of a service than a vm. I wish its concurrency was unlimited and you paid for usage rather than time. Basically DynamoDB with SimpleDB pricing.
- la6470 10y agoWow serverless ..:
- boulos 10y agoNote: the underlying comparison to other systems is from a 2014 blogpost [1] which suggest they used the m2.4xlarge series of EC2 VMs (which were Nehalem class parts from 2010). Nehalem vs Haswell or Broadwell (the likely parts underlying Lambda) is a pretty big jump. Disclosure: I work on Google Cloud, but I'm just pointing out a fact ;). [1] https://amplab.cs.berkeley.edu/benchmark/ https://amplab.cs.berkeley.edu/benchmark/
- deleted 10y ago[deleted]
- mallya16 10y agoImplementation guide for Serverless MapReduce: https://aws.amazon.com/blogs/compute/ad-hoc-big-data-processing-made-simple-with-serverless-mapreduce/ https://aws.amazon.com/blogs/compute/ad-hoc-big-data-process...
- frenchhacker 10y agoI guess the example assumes the data is already somehow in AWS. How is the total cost affected if I wanted to run this setup on a 10TB dataset?