13 ms·
Three Years on Google App Engine
- e2e4 9y agoFor me AWS is still a king of serverless. In particular, recent AWS step functions really help with connecting microservices (and other tasks) for more complex flows, and does so in a nice visual manner: https://aws.amazon.com/step-functions/ https://aws.amazon.com/step-functions/ p.s. it is pricey, but probably this will improve soon.
- jadeklerk 9y agoThe topic of this article is Google App Engine (GAE), which is a PaaS. Comparing GAE to AWS step functions (which is some wrapping around AWS Lambda) is moot since they are different categories of products solving different problems. For PaaS products, GAE is comparable to AWS Beanstalk (or Heroku, or CloudFoundry, etc). For "serverless" products * , AWS Lambda/Step Functions are comparable to Google Cloud Functions. * Serverless in the product sense rather than the literal sense.
- e2e4 9y ago> The topic of this article is Google App Engine (GAE), which is a PaaS. PaaS & FaaS are quite related; so it is not unreasonable to discuss in this threat. e.g. see informative comments by @indigodaddy on the same threat. p.s. article also talks about other non-PaaS topics e.g. Data Store, Memchase, BigQuery, etc. > Comparing GAE to AWS step functions (which is some wrapping around AWS Lambda) is moot since they are different categories of products solving different problems. Many of the problems that they solve are the same; but they do so in a different manner. AWS Step Functions is a lot more than "wrapping around Lambda" > For "serverless" products * , AWS Lambda/Step Functions are comparable to Google Cloud Functions. AWS Step Functions has no comparable service on Google Cloud
- nimchimpsky 9y agoYou can use Java 8 now
- deleted 9y ago[deleted]
- ryanianian 9y agoGood, balanced article. I'd love to see something similar for the competitors like AWS Elastic Beanstalk or the upcoming "App Engine Flexible Environment" that the author mentions (of course it'd be hard to get a "3 years in" perspective for such a new product).
- spyspy 9y agoApp Engine Flex is newer but I wouldn't describe it as "upcoming". My company's been using it in production for a while now.
- elithrar 9y agoAgree. In fact, I strongly prefer the Flex environment as it doesn't require you to use any proprietary libs or services: your applications get deployed as containers, can auto-scale, and are health checked. This has always been one of the benefits of Heroku to me: I don't have to architect my application for a single vendor's environment. If I use a hosted Postgres backend, then even more so. What Google should consider doing is making App Engine "classic" services like Memcached, etc. available to any service running across GCP. Flex is fairly different from classic, but I think there's some confusion generated from the "App Engine" name and the baggage (good & bad) it has gathered over the years.
- thesandlord 9y ago> What Google should consider doing is making App Engine "classic" services like Memcached, etc. available to any service running across GCP. Already being done. First was Datastore, now Cloud Tasks is in Alpha. Expect to see others (Cron, Cache, etc) come as well.
- karl42 9y agoBut there is still no way to use ndb from outside App Engine standard, right?
- ddorian43 9y agoStill weird that google doesn't offer a search service.. (there is one but doesnt look good)
- djtriptych 9y agoGAE certainly does have a search service: https://cloud.google.com/appengine/docs/standard/python/search/ https://cloud.google.com/appengine/docs/standard/python/sear...
- ddorian43 9y agoVery limited, like no cross-index-search etc...
- djtriptych 9y agoWhat do you mean by cross index search?
- ddorian43 9y agoSearch on 2 indexes and get back the merged results. Like solr/elasticsearch do search on index with 1+ shards.
- nawtacawp 9y ago- Doesn’t support Python 3
- sologoub 9y agoStandard environment does not and it's a damn shame. Flexible environment does not have that restriction.
- thanksgiving 9y agoBut there is no free tier in flexible as far as I know.
- sologoub 9y agoDon't think there is, but for initial trial there is $300 credit for a year.
- hdieiidjjd 9y agoFree tiers are probably the least economically interesting aspect of cloud services. If you're using gae to do valuable work, you shouldn't mind paying a little bit for the service.
- asamarin 9y agoIt's currently sitting at #1 feature request by vote count on GAE issue tracker: https://issuetracker.google.com/savedsearches/559750 https://issuetracker.google.com/savedsearches/559750
- CydeWeys 9y agoThis is embarrassingly a Google-wide problem, not just a GAE problem. Stick with Java or Go, I'd say.
- sologoub 9y agoI recently restarted fiddling with GAE after stopping in 2012 and moving everything to heroku/aws. To say I'm blown away by how far it has come is an understatement. Within less than a day I had a prototype up and running that handled inbound requests, did some processing, offloaded the rest to background workers and returned the request. The workers asynchronously stream data to BigQuery for real-time reporting. The only thing I found baffling is that I could not load Googles' own cloud sdk from within the GAE environment and had to "vendor" it in, doing a bunch of workarounds/patching to get it to run. That probably consumed 3 of 4 hours it took to get this running. Load testing this thing, GAE automatically managed everything and I never saw any error responses throughout the load test with. It's amazing how little ops work is needed. AWS lambda and kinesis firehose/s3/Athena could provide part of this, but it's far from turnkey/conclusive. GAE/GC are just so much more complete.
- sprt 9y ago> [...] doing a bunch of workarounds/patching to get it to run. That probably consumed 3 of 4 hours it took to get this running. This always happens when I try to use one of Google's APIs/SDKs. I dread using them, I feel like they're overengineered most of the time.
- sologoub 9y agoUnfortunately, that's not one of those over-engineering times. GAE standard environment simply does not include the needed SDK. I'm guessing they just haven't gotten to it. The SDK works as expected once imported as the outside library and various outbound requests things are configured and patched. I think I overstated the complexity. All I needed to do was enable SSL for outbound connections (one-liner in the yaml file) and add the requests library patch to use URLFetch service (python/GAE specific step). Searching SO and Google documentation is what took up 95% of the 3 hours I spent on this.
- sprt 9y agoYeah, poor choice of word on my part, but I feel you. This happens to me a lot with their stuff: the solution is trivial yet not obvious and it's not documented properly so you end losing a lot of time.
- marktangotango 9y agoIt is a fully-managed application platform. So far, I do not know a platform which comes close to GAE's full package: log management, mail delivery, scaling, memcache, image manipulation, distributed Cron jobs, load balancing, version management, task queue, search, performance analysis, cloud debugging, content delivery network - and that is not even mentioning auxiliary services that have popped up on Google's cloud in the meantime like SQL, BigQuery, file storage... the list goes on. I think the promise was never actually realized. When first released, you didn't have to worry about load balancing instances or scaling, it was "internet scale", out of the box, dependent only on your traffic. Then they went to the per instance pricing and raised prices. I was really disappointed no service ever arose to fill this gap; internet scale as service you could say.
- rak00n 9y agoThey still have the autoscaling. "Internet scale" doesn't mean we can ask for more resources without paying more.
- throwaway713824 9y agoI remember reading at least one case study of someone standing up an online poll related to a sporting event that got 100k's users over a few days on GAE, really impressive at the time. You can still do that of course, but now you're dealing with load balancers and instances, a lot more admin overhead. It is/was a total walled garden, but for scale and ease of use like that, it was really worth the sacrifice. I really don't understand why everyone wants to be wrangling containers/instances nowadays.
- sudhirj 9y agoMoving to the charge by time instead of by CPU doesn’t change the scale or the functioning of the system. They just realised that inactive apps also consumed resources. And AWS Lambda has sort of picked up on this. Not exactly the same but very close.
- whoisjuan 9y agoIt's also expensive as fuck. Good for prototyping, for when you're starting out and maybe for when you are creating an app that serves very specific uses cases that are easier to address with a fully managed architecture. But for everything else just start adding APIs and Services and see how your costs balloon out of control.
- segmondy 9y agoI never get this, why would you use it for prototype if you will not run on it? Why are we prototyping in the cloud? What happened to localhost?
- codingdave 9y agoOne use case would be for getting the product/market fit. Launch fast, then either fail fast or succeed and port to a cheaper service.
- 549362-30499 9y agoI think the GP comment means prototyping a product idea to a real market. Obviously individual developers will run a local version of the production systems.
- yeukhon 9y agoBecause your local computer is not where you will run your production. You absolutely are not required to develop on the cloud, but you should at least consider (and in fact this should be “must”) test you code in thr cloud before you enable the knot to ship it in production mode. I’d argue if cloud is convenient for you and cheap as hell, no reason not to try. However, it is not cheap because you would need to setup your own cloudfoundry or kubernetes in AWS and whatnot (or the cloud provider’s native container solution), which often means renting a machine. So that’s why most people don’t do development in the cloud. They test their “just committed to my feature git branch” latest version in the cloud by means of automatic (Jenkins etc) or by pushing to the cloud manually.
- sologoub 9y ago
- asamarin 9y agoBear in mind this article was written back in March 2017; quite a few things have changed/improved in GAE-land since. To name a few, off the top of my head: - Java 8 is already generally available on GAE standard and, on top of that, previous limitations wr/t class whitelisting, threading capabilities and others have been lifted - appcfg is pretty much outdated these days; all GAE management operations have been progressively integrated into gcloud app subcommand, which also fully supports newer GAE features like Flex, firewall rules, etc. - GAE standard instances are indeed limited in computing power, but it should be enough for most web-based applications. Again, GAE Flex fills in the gap here in case more muscle is needed, since it leverages GCE-class instances - Besides the already mentioned background work options (background threads, task queues, cron jobs), there's also Pub/Sub which can be used to inter-communicate GAE services or even other GCP services like GCE or GKE
- twblalock 9y agoJava 8 became available on GAE about 3 years after it was released. If Google continues that trend, Java 9 won't be available until 2020. Slow adoption of new Java versions is a pretty big downside. Even if you are satisfied using Java 8 on GAE today, you may want to use Java 9 in the near future, and you won't be able to. By choosing GAE you are committing yourself to being stuck with Java 8 for a long time.
- yegle 9y agoIf you sign an NDA with Google, you'll get more up to date info about future features/roadmap etc. I think you'll be happy to know those plans. Disclaimer I work on GAE.
- PeCaN 9y agoYou have to sign an NDA just to get the roadmap‽ Is it that volatile that they don't want to go public with their intentions? That's not really a good sign, IMO.
- BareNakedCoder 9y ago
- jmugan 9y agoI've generally had a lot of trouble with GAE. Debugging is really hard and things seem to fail for weird reasons. The use case I have, with a front end and a static backend, has been surprisingly complicated. I recently shut my app down because it wasn't worth the time to figure out how to change my app for the Dec 15th upgrade.
- NearAP 9y agoI've been using GAE for a while now starting from when they only supported Java and Python (I think). It's my go to source when I want to create an MVP or proof of concept. I've also used it to run some sites with medium traffic. Over the years they've introduced a whole bunch of stuff/services, simplified a bunch of things (e.g. datastore pricing is now simpler). They've also made things more complicated (in my opinion). Years ago, downloading and using GAE was straight forward. Now they have all sorts of versions (standard, flexible, etc) and running a program is no longer as simple as before. Their documentation is also extremely poor and they sometimes change/remove services without enough notification (switching from modules to services, changing support for openID/Federated login even though it still showed up in GAE). At the end of the day, I would still recommend it. It just takes more effort for me to explain to new folks how to use it.
- indigodaddy 9y agoBelieve this article was posted about 3-5 times previously on HN in 2017, and never gained traction. I think perhaps that "cloud" and "serverless" have become even more of a thing throughout 2017. Even since March (was in the job market in right after March), I noticed how positions in tech really seemed to be coalescing around "cloud engineers," and "SRE's/Devops" and the like... people on LinkedIn getting 5 AWS certs in a week (lots of this), stuff like that; I really don't even think we are at peak cloud/serverless usage/hype right now. Believe it will continue to grow in 2018, as that other article on HN right now (about serverless in 2018) also proposes.
- spyspy 9y agoI've seen that most companies are still coming around to containerization as the future of their infrastructure. Serverless isn't even on the roadmap, so we're definitely far from peak hype.
- yeukhon 9y agoI think there is much to show how to leverage container in development. I personally still struggling to get my mindset fix on using containers than running locally. It gets me every time and this makes me feel sad as if I am not in touch with the trend. If people were so stereotype about age and workers, which is not really true as my older coworkers are just as competent and as curious and as willing to adopt new technologies, now I find myself in that stereotype fallacy. But I feel helpless. Anyone?
- ne01 9y agoFor those who are interested to know how much App Engine Standard Environment roughly will cost in real world applications here is my simple formula for it: Average RPS * $15 So for 100 RPS for one month it costs around $1,500. Also, you can optimize for cost by adding the following to your app.yaml: threadsafe: true Also Golang and Node costs less than other environments. My two cents. Update: Formatting.
- user5994461 9y ago100 average => the service has to do 1000 req/s during active business hours. It's not much for anyone who is used to high scale system but it is a lot more than the average application can take without effort. It's gonna require quite a few cores, disks and bandwidth to achieve. Definitely not gonna run on a single cheap instance.
- duiker101 9y agoI'm sorry, I'm lost. What is RPS?
- ne01 9y agoRequests per second
- thefounder 9y agoGAE -> easy to get in...very hard to get out! and expensive...and risky...it's not worth it!
- erikb 9y agohow about other PaaS?
- foxylad 9y agoIt's actually not that hard to get out of, unless you rely on their scaling - in which case any system is hard to get out of. We did some contingency planning a while back, which included moving a low volume service to a LAPP (Linux/Apache/Python/PostgreSQL) stack to see how difficult is was. The biggest job was re-writing the NDB API calls, but it was not difficult. For me, knowing the world's best sysadmins are looking after our services is worth a lot. I've heard a couple of talks by Google's system relability engineers, and they are fanatical. Visible evidence of this is the brutally honest post mortem they produce every time there is a significant outage, including the vitally important "what we're going to do to stop this ever happening again" section. This focus works - GAE is way more reliable than AWS or Azure. So YMMV, but GAE is well worth it for us.
- duiker101 9y agoI have a Java Application on it but I have never found a straight forward way of exporting the data from the Datastore. Any ideas?
- foxylad 9y agoYou are being down-voted because the whole point of almost all GAE apps is to return data from the datastore, albeit usually in nicely formatted HTML for browsers. You simply need to provide a handler that returns your data in whatever chunks or format you want.
- thefounder 9y agoIt's not the scalling that locks you on GAE. It's about the proprietary services. I have 32 micro services, all depending by datastore, task queues etc. The only contigency plan that someone could implement is to do not use GAE in the first place. Almost 70% of the app code is really just about different ways to index/store and return data. Rewriting all the datastore ops is just like rewriting the whole application. With GAE you get stuck and there is little you can do! I had to choose between moving to their "flex" version or just dumping all the code and start over. Starting over is not an option so I'm still on GAE paying the flex prices. Now I pay regardless if the microservices are running or not. If it would be that simple to just move it all to an open source stack you can be sure I would do that.
- ad_hominem 9y agoI tried to use App Engine flex for my application, but it seems to only support the simplest of apps. `gcloud app deploy` would always time out after about 10 mins with DEADLINE_EXCEEDED due to the Docker build taking too long. It was not well documented, but I figured out I could work around that by defining a custom cloudbuild.yaml with my own "timeout" parameter and submit remote builds using that (`gcloud container builds submit --config cloudbuild.yaml .`), then deploy that specific image (`gcloud app deploy --image`). Then I found out that I couldn't run my own sidecar container (nginx reverse proxy), and also that Cloud CDN doesn't support App Engine Flex yet. Those were dealbreakers, so I switched to GKE. So far things are working well except a Kubernetes issue not terminating Job/CronJob sidecars correctly which gets tangled up with Google forcing you to run a "magic" proxy sidecar container to access your Cloud SQL instance(s).
- grogenaut 9y agowhat are you doing that takes over 10 minutes that can't be broken up into a series of retryable (and likely parallelizable) steps that a worksflow or queue system wouldn't work, be more resilient, scalable?
- ad_hominem 9y ago`docker build` is not parallelizable by its very nature of each layer depending on the previous layer
- grogenaut 9y agoso you're running a build server?
- ad_hominem 9y agoNo. App Engine Flex uses containers internally (the "flex" meaning supporting any type of app rather than the traditional built-in App Engine environments of Ruby, Python, etc.). The appeal of App Engine is simply being able to type `gcloud app deploy`, which tars up your local source code, builds a Docker image from it, then does a rolling deploy of the new image to replace your old app containers - all from that one command, no complicated build system needed on your part (i.e. Google Cloud does it all for you). My point was the moment you stray from the App Engine happy path (`gcloud app deploy` fails for obtuse reasons), you're out in the weeds. That plus the fact that App Engine has some weird missing integrations (e.g. Cloud CDN) often makes it easier to just jump to GKE in my opinion.
- jorgec 9y agoGoogle App Engine (GAE) was cool because it was for free. Then, Google started charging (really expensive), i got enough with it and i quit.
- CydeWeys 9y agoGAE does still have a free tier. Also, being able to run as much stuff as you want in the cloud for free does not seem like a reasonable expectation.
- jpatokal 9y agoThe previous poster may be confused by the fact that App Engine Flex does not have a free tier. For the record, here are the Standard Edition free tier quotas: https://cloud.google.com/free/docs/always-free-usage-limits#gae_name https://cloud.google.com/free/docs/always-free-usage-limits#... And you can trivially ensure you never exceed them simply by setting a $0 daily budget. (Although I recommend setting a low but non-zero cap instead, so your app doesn't go down if there's a legit but unexpected traffic surge.) Disclaimer: I work at GCP.
- user5994461 9y agoApp Engine was free before it forced you to enter a valid card number to be able to use the service.
- kuschku 9y agoIt did? I remember using it as a cheap solution for serving static files, without ever entering a CC number (as I have no CC anyway)
- user5994461 9y agoYes, the change was made in 2017.
- indescions_2018 9y agoStill in beta, but also consider Firebase + Cloud Functions on GCP. Very fast to build with. And easy to upgrade to GAE or K8 containers if need be. Cloud Functions for Firebase https://firebase.google.com/docs/functions/ https://firebase.google.com/docs/functions/
- asah 9y agoI've been on GAE continuously since 2008. Can confirm virtually all comments (pro/con) in this HN thread.
- bananarepdev 9y agoI understand the the benefits of using a fully managed platform like GAE, but wonder if it would be the right choice for most scenarios. Personally, I would prototype with GAE, but go into production with GKE, under the impression that the application would be a little more decoupled from the platform.
- Jyaif 9y agoI've been hosting a service on GAE since the beginning. Pretty much 100% uptime, just had to update a .yaml file twice in... a decade? Amazing service.
- sitkack 9y agoThe future of GAE is K8S. GAE should be an app on K8S.
- kccqzy 9y agoThis largely matches my own experience in using GAE. I might just add that since I was using GAE for personal projects, it is very easy to accidentally use a lot of money with the ease to scale up and handle more traffic. There is a way to tell GAE to have a budget limit though. And yes, although this is essentially vendor lock-in, the access to the many services provided is really really a timesaver that boosts productivity tremendously. I just wonder whether Google is spending much time on it, given how slow they support newer versions of languages.
- arunaugustine 9y agoDoes GAE support websockets?
- ivanb 9y agoNo. You would need to use a third-party service or build one on a VM in GCE. You can deploy a queue with websocket support to save some effort.
- tapirl 9y agoI agree with the author that GAE is fascinating, for hosting large scale web apps. But I strongly recommend to avoid using the Java runtime on GAE. Each GAE instance has very limited memory, 128M (the prices of high memory instances are much higher). 128M is not enough for most Java web apps! Another big problem of Java apps on GAE is a general Java app needs 20 seconds to fully warm-up (considering that the 128M instance has a 600MHz CPU). GAE instances stop and start frequently. If a user visits your website and the visit is served by an instance which has not warmed-up yet, the unlucky user will wait 10s+ to get responded. Then the user experience is very bad. (GAE really provides a way to avoid cold instances serving requests, but the way is tedious to setup and often needs more instances to run. More instances mean higher cost.) So if you would like to use GAE to host your web app, Go is most preferred. Go instances consume much less memory than Java instances (less than 1/5) and need much less time to fully warm-up (less than one second, so you don't need to handle the cold instance problems at all). Just my personal experiences.
- stickfigure 9y agoAs someone that's been building GAE apps on Java since the runtime first launched... well, it's a mixed bag, mostly good. It works fine for basic web apps. If you have any significant traffic it's a good idea to upgrade to F2 or F4 instances; you get more throughput per-instance so it's break-even costwise. The instance warmup time tends not to be an issue at scale, but it does mean you end up paying for more idle instances than "optimum". It's still cheaper than hiring a devops team. The cool thing about GAE is that you can build a multi-million-dollar business with a couple engineers and they can both go camping for a weekend at the same time. Go is nice on GAE because you have near-zero instance start time. But then you must work in Go, which is a poor language for general business processing (despite its other merits). The Go runtime also lags behind the Java and Python runtimes on GAE Standard. The GAE community does not think "Go is most preferred"; actually the GAE/Go community tends to whine a bit about support. It's not perfect, but then again nothing is.
- tapirl 9y agoYes, if the traffic of your web app is large and need tons of GAE instances, then the memory and warm-up problems will become much less severe. But if the traffic is not large, then the two problems are the big obstacles to save spending and happy roll deployments. One the other hand, Go is good for both low traffic and high traffic websites. I disagree on "Go is a poor language for general business processing". The current Go runtime on GAE is very mature to build all kinds of websites.
- nhooyr 9y agoWhat are does he mean by using mnemonic as key?
- pzk1 9y agoWhen is GAE & Cloud SQL coming to Southeast region?
- gsvclass 9y agoI built my first startup (Socialwok) in 2008 on GAE Java it had just launch then. Since then I've totally been sold on the Google Cloud and building products with serverless tech like GAE. Even today surprisingly few people know about it. So I finally authored and published a book on building products using the Google Cloud this year. (Free to read on the site) http://www.howtobuildthefuture.com/ http://www.howtobuildthefuture.com/
- bshanks 9y agoAt the end of the article (and in the linked article), the author says that Flex is too immature for production use, but also says that GAE Standard is practically on autopilot and not recommended for new projects. So it seems that the author doesn't recommend starting new production projects on GAE at all. Has this been others' experience? Given Google's record with other products, I'm particularly worried Google might discontinue or stop improving GAE. Does Google provide any offical reassurance that GAE is a high-priority product that will be quickly improved and not put on autopilot? If so, do their promises seem to match the reality observed by the author for GAE standard? quote from the end of the article: " It feels like they only invest the bare minimum anymore. Actually, I have had this feeling for the last two years...It is infuriating if there are known issues but they are not fixed. It is depressing to receive so little information on where the platform is heading. You feel trapped. ... the 'old' App Engine is on its way out. I do not think it is a good idea to start new projects on it anymore. If App Engine Flexible Environment on the other hand can actually fix its predecessor's major issues, it might become a very intriguing platform to develop apps on. "