4 ms·
A lot of these AppEngine costs too much notices have been coming up recently, but upon further inspection, they all tend to boil down to operator error. Unfort
by latchkey 15y ago
A lot of these AppEngine costs too much notices have been coming up recently, but upon further inspection, they all tend to boil down to operator error.
Unfortunately, AppEngine isn't forgiving of that and there is a real monetary value associated with questionable engineering design. Or, design that wasn't thought through enough in the context of a service like AppEngine.
This leads to a few people getting upset and making a lot of noise when the reality is that AppEngine is actually an amazing service.
So, to boil down the operator error from a quote in the thread:
"We're running a mapreduce to change the geobox sizes/precision for a large number of entities."
That is the real source of the problem. Instead of using geoboxes, they should be using geohashes, which allow arbitrary precision.
http://code.google.com/apis/maps/articles/geospatial.html http://code.google.com/apis/maps/articles/geospatial.html
http://en.wikipedia.org/wiki/Geohash http://en.wikipedia.org/wiki/Geohash
Instead of an indexed property that looks like this (what they currently have):
[u'37.3411|-121.8940|37.3395|-121.8926', u'37.3411|-121.8929|37.3395|-121.8916', ...]
They would have an indexed List<String> property that looks like this:
[8, 8f, 8f1, 8f12, 8f12a, 8f12ac, 8f12ac6, 8f12ac60, 8f12ac605, 8f12ac605f, 8f12ac605fb, 8f12ac605fb3, 8f12ac605fb34]
Finding if the location is in a box would be computing the hash from the lat/lng (there is free code out there to do that) and then doing an indexed 'in' query. The indexes would only need to be updated if the location of the entity changes, not when they want varying levels of precision.
- erichocean 15y agoWe wrote some App Engine code to do geospatial lookups. Before anyone gets the idea that the links in the parent are worth trying, they're not: the performance is absolutely atrocious – on our data set, 13 seconds per lookup. We wrote our own approach on App Engine and now get stable performance on our datasets at ~300ms per lookup. (We're doing Foursquare-type lookups.)
- latchkey 15y agoI can't imagine how an indexed query on a List<String> of ~10 items takes 13 seconds. 'We wrote our own approach on App Engine' Hmm... details?
- erichocean 15y agoYou can find the code we wrote here, along with the testing infrastructure: https://github.com/erichocean/appengine-geospatial-lookups https://github.com/erichocean/appengine-geospatial-lookups Like Foursquare, mobile clients send their current location, plus a search radius, and App Engine code returns a result set ordered by distance. We were getting query times with realistic data set sizes in the ~13 second range from the code you linked to. With the code above, we're ~40x faster. YMMV.
- deleted 15y ago[deleted]
- nicksdjohnson 15y agoYou should write this up somewhere and post it to HN and the App Engine Reddit. Getting geospatial right is hard, and it sounds like you've done a good job.
- DennisP 15y agoThat's awesome. How does it work, basically?
- thefool 15y agoThis is incorrect for several reasons. First off, they mention that when the initial design decision was made, a similar operation cost ~$160, which is tenable for an operation that only happens once in a while. This is in fact a case of them getting bitten by the pricing structure changing after a reasonable design decision (at the time) was implemented. Secondly, they mention that this is part of a larger issue: "In our most common case we might have to add and delete a couple items to the list property every once in a while. That would still cost us well over $1,000 each time. Most of the reasons for this type of data in our product is to compensate for the fact that there isn't full text search yet. I know they are beta testing full text, but I'm still worried that that also might be too expensive per write." This is a real problem that GAE needs to solve. Finally, their problem doesn't seem to be that they need arbitrary precision, its that they seem to need fast location centric queries of a large database. Geoboxes allow you to solve this problem correctly (and quickly), returning the results in the database that are closest to you. Matching on a geohash can end up serving the incorrect data unless you resort to hacks involving a number of queries.
- latchkey 15y ago1) You are making an assumption that the original design decision was made around cost. I bet the more likely fact is that they made the change and found out that it cost them $160. Remember, that was when AppEngine was a _beta_ product and it seems they got lucky the first time. 2) They seem to have an extreme use case. No one is going to argue that maybe AppEngine doesn't fit the bill for them. Or, one could argue that doing 6.5 billion writes times a large number of customers, across multiple datacenters is something that a lot of databases would choke on. 3) Running more queries, while admittedly hacky is less expensive than doing more writes.
- ww520 15y agoYes, blame the operators. The operators designed their apps based on the original pricing incentive published by Google but got the rug pulled under them when Google hiked the price to completely change the pricing incentive and invalidate their design. I got burnt by AppEng, too. Picking AppEng as a platform is one of my worst technical decisions.
- latchkey 15y agohttp://googleblog.blogspot.com/2008/04/developers-start-your-engines.html http://googleblog.blogspot.com/2008/04/developers-start-your... "Google App Engine is free to use during the preview release, but the amount of computing resources any app can use is limited. In the future, developers will be able to purchase additional computing resources as needed, but Google App Engine will always be free to get started." You got 2+ years of use for 'free' and now they decided to turn it into a supported business model and are asking you to pay for what you use. Seems reasonable to me. I don't disagree that they fubar'd their original pricing release announcement and should have had multithreading Python 2.7 for those folks. But they did listen to the (loud) feedback, made adjustments and even apologized (were you there at the ThirstyBear meetup where they bought us all beers?). Not having to hire an IT staff or be woken up in the middle of the night when AWS decides to reboot the host and your servers go down is worth its weight in gold.
- ww520 15y agoYes, stupid of me to trust the marketing speak in that blog post. I've already said it was one of my worst technical decisions. I've advocated to clients to use AppEng and I looked real stupid when the projects failed because of operation cost hike. Software are architected and designed with constraints in mind. Features are feasible or unfeasible because of these constraints. Cost is one of the big constraints. With the new cost structure, the apps have to be re-architected and redesigned, lots of things aren't possible. I didn't go to the ThirstyBear meetup because I have given up on AppEng and moved on. I simply do not have trust for them to be a platform vendor.
- 15y ago