4 ms·

Partychat — migrating from Google App Engine to EC2

57 points by BarkMore 15y ago

hello_moto 15y ago
Seems like the world is full of super-talented people these days...
mike-cardwell 15y ago
Can someone TLDR the TLDR for me please
jrockway 15y ago
Why do you think that Google's costs are less when sending messages inside their "intranet" versus sending messages to and from the outside world? Their network is huge and it's unlikely that your users XMPP accounts are managed by a server in the same data centers as your App Engine instances. So it's not costing Google more, it probably costs them the same. Also keep in mind that if your time is not free, App Engine may be cheaper than EC2. At some point, renting out a colo and buying your own hardware is cheaper than EC2. And then at another point, it's cheaper to build and manage your own data center. It's all a continuum of effort and upfront costs vs. monthly costs.
vinhboy 15y ago
I build an image hosting site, similar to imgur, on GAE. Some people in brazil are using it. Last month, the bill went from $4 a month, to $4 a week. As far as I can tell, the site did not get any more popular than it was a month ago. That is a ridiculous increase! Needless to say, I took down the site.
latchkey 15y ago
Google hasn't made 'random' pricing changes. Sure, they didn't do the best job of handling the switch from beta to a real live service, but they did listen to the community and they learned a lot of lessons. I really don't think they will make the same mistakes twice. Maybe you missed the meetup at Thirsty Bear in SF where they bought us all drinks and apologized? Even Guido was there. I think you are wrong about the costs of EC2. First off, you have to run your own servers on EC2. When one of those servers goes down in the middle of the night for some random unexplained reason, who are you going to call? Yourself? If you don't think EC2 servers go down, you are kidding yourself. With GAE, I'll never have to hire IT staff to manage servers or networks. The high replication datastore on GAE is amazing. I'll never have to worry about running out of capacity, backups, or downtime for that matter. I've been told that entire datacenters have gone offline and nobody even noticed. Now, that is reliability. It sounds like a lot of your issues surrounded using XMPP on GAE. I think that is a perfectly valid thing to complain about. That said, just cause XMPP is expensive doesn't mean that all of GAE is expensive (or problematic). As you found out by writing your own C++ servers and what not, XMPP is a difficult service to provide reliably. Maybe that is worth that cost? Note: I'm building my sporting event ticketing website entirely on GAE. People will be able to register for marathons, bike races, etc. It may cost a bit more for their service, but given that I won't have to hire any staff to manage servers or worry about adding capacity when thousands of people all want register for an event at the same time, I think that is worth it. Note2: I'm a user of PartyChatApp. Thanks for providing it.
- cstejerean 15y ago
  Careful with the "or worry about adding capacity" part. I've talked to several people that built products on top of GAE because it would just scale, and all were surprised to discover the opposite. Certain queries would start to take inexplicably long and there was no way to figure out why or how to fix it, etc. Things might have gotten better recently, but about a year ago everyone I talked to that tried to scale on GAE gave up and moved elsewhere.
  - latchkey 15y ago
    The datastore is complicated and not what you'd normally expect from a SQL type system. It is a real mental shift when designing your schema to be efficient. That said, using it is well understood at this point and my co-founder is Jeff Schnitzer who wrote Objectify [1] which is the best way to tie Java to the datastore. Outside of Google, I think he understands the datastore more than anyone else on this planet. Thanks for the warning, but I think we will be ok. =) Also note, I'm speaking from a Java perspective. Using Python has definitely had MT issues which are being resolved with the new version of Python 2.7 available there now. I'd still recommend using either Go or Java for your app. [1] http://objectify-appengine.googlecode.com/ http://objectify-appengine.googlecode.com/
    - cstejerean 15y ago
      I understand that GAE's data store is a different kind of beast, and that's actually one of the things that worries me about investing in GAE. What's the migration strategy for moving away from GAE if things don't work out? Or if Google decides to abandon it at some point?
      - latchkey 15y ago
        Sure, the datastore makes you think a bit harder when designing your application. But, I'd rather do that thought upfront than down the line when my MongoDB (or insert your favorite nosql solution here) instance is falling over under the load. In other words, you should probably be doing that anyway regardless of using GAE or not. ;-) The rest of my app is just a regular .war file, so really it is just the data layer that would need to be ported. Given that the data store is pretty much just a big hashmap, it really won't be that difficult to rewrite the underlying code for another nosql store like MongoDB. The hard part of the original 'schema' design is already done. As part of GAE leaving beta, Google has given a 3 year EOL commitment. I'm actually glad it has left beta because that means we get something solid from them now. What I mean is that once they declare that they are killing GAE (if they ever announce that), you have 3 years to migrate. I'm ok with that term. http://code.google.com/appengine/terms.html http://code.google.com/appengine/terms.html 7.2.a: "For a period of 3 years after an announcement (the “Deprecation Period”), Google will use commercially reasonable efforts to continue to operate the Deprecated Version of the Service and to respond to problems with the Deprecated Version of the Service deemed by Google in its discretion to be critical." Seriously, I've done the research and I've used the different solutions that are available today. GAE is one of the best solutions out there at this time for my type of application, which is pretty much just a straight up webapp. The only difference I have is that I have really bursty traffic as a ton of people all try to register for an event at the same time and then go away. This means, I need something to truly auto scale up and down without requiring me to do work or know my load in advance. No other solution offers the level of PaaS that GAE offers and at the price.
randall 15y ago
PS- Partychat is amazing. If you use Google Apps for your biz, Partychat is like the missing chatroom feature.

57 points by BarkMore 15y ago

hello_moto 15y ago

Seems like the world is full of super-talented people these days...

mike-cardwell 15y ago

Can someone TLDR the TLDR for me please

jrockway 15y ago

Why do you think that Google's costs are less when sending messages inside their "intranet" versus sending messages to and from the outside world? Their network is huge and it's unlikely that your users XMPP accounts are managed by a server in the same data centers as your App Engine instances. So it's not costing Google more, it probably costs them the same. Also keep in mind that if your time is not free, App Engine may be cheaper than EC2. At some point, renting out a colo and buying your own hardware is cheaper than EC2. And then at another point, it's cheaper to build and manage your own data center. It's all a continuum of effort and upfront costs vs. monthly costs.

vinhboy 15y ago

I build an image hosting site, similar to imgur, on GAE. Some people in brazil are using it. Last month, the bill went from $4 a month, to $4 a week. As far as I can tell, the site did not get any more popular than it was a month ago. That is a ridiculous increase! Needless to say, I took down the site.

latchkey 15y ago

Google hasn't made 'random' pricing changes. Sure, they didn't do the best job of handling the switch from beta to a real live service, but they did listen to the community and they learned a lot of lessons. I really don't think they will make the same mistakes twice. Maybe you missed the meetup at Thirsty Bear in SF where they bought us all drinks and apologized? Even Guido was there. I think you are wrong about the costs of EC2. First off, you have to run your own servers on EC2. When one of those servers goes down in the middle of the night for some random unexplained reason, who are you going to call? Yourself? If you don't think EC2 servers go down, you are kidding yourself. With GAE, I'll never have to hire IT staff to manage servers or networks. The high replication datastore on GAE is amazing. I'll never have to worry about running out of capacity, backups, or downtime for that matter. I've been told that entire datacenters have gone offline and nobody even noticed. Now, that is reliability. It sounds like a lot of your issues surrounded using XMPP on GAE. I think that is a perfectly valid thing to complain about. That said, just cause XMPP is expensive doesn't mean that all of GAE is expensive (or problematic). As you found out by writing your own C++ servers and what not, XMPP is a difficult service to provide reliably. Maybe that is worth that cost? Note: I'm building my sporting event ticketing website entirely on GAE. People will be able to register for marathons, bike races, etc. It may cost a bit more for their service, but given that I won't have to hire any staff to manage servers or worry about adding capacity when thousands of people all want register for an event at the same time, I think that is worth it. Note2: I'm a user of PartyChatApp. Thanks for providing it.

cstejerean 15y ago
Careful with the "or worry about adding capacity" part. I've talked to several people that built products on top of GAE because it would just scale, and all were surprised to discover the opposite. Certain queries would start to take inexplicably long and there was no way to figure out why or how to fix it, etc. Things might have gotten better recently, but about a year ago everyone I talked to that tried to scale on GAE gave up and moved elsewhere.
- latchkey 15y ago
  The datastore is complicated and not what you'd normally expect from a SQL type system. It is a real mental shift when designing your schema to be efficient. That said, using it is well understood at this point and my co-founder is Jeff Schnitzer who wrote Objectify [1] which is the best way to tie Java to the datastore. Outside of Google, I think he understands the datastore more than anyone else on this planet. Thanks for the warning, but I think we will be ok. =) Also note, I'm speaking from a Java perspective. Using Python has definitely had MT issues which are being resolved with the new version of Python 2.7 available there now. I'd still recommend using either Go or Java for your app. [1] http://objectify-appengine.googlecode.com/ http://objectify-appengine.googlecode.com/
  - cstejerean 15y ago
    I understand that GAE's data store is a different kind of beast, and that's actually one of the things that worries me about investing in GAE. What's the migration strategy for moving away from GAE if things don't work out? Or if Google decides to abandon it at some point?
    - latchkey 15y ago
      Sure, the datastore makes you think a bit harder when designing your application. But, I'd rather do that thought upfront than down the line when my MongoDB (or insert your favorite nosql solution here) instance is falling over under the load. In other words, you should probably be doing that anyway regardless of using GAE or not. ;-) The rest of my app is just a regular .war file, so really it is just the data layer that would need to be ported. Given that the data store is pretty much just a big hashmap, it really won't be that difficult to rewrite the underlying code for another nosql store like MongoDB. The hard part of the original 'schema' design is already done. As part of GAE leaving beta, Google has given a 3 year EOL commitment. I'm actually glad it has left beta because that means we get something solid from them now. What I mean is that once they declare that they are killing GAE (if they ever announce that), you have 3 years to migrate. I'm ok with that term. http://code.google.com/appengine/terms.html http://code.google.com/appengine/terms.html 7.2.a: "For a period of 3 years after an announcement (the “Deprecation Period”), Google will use commercially reasonable efforts to continue to operate the Deprecated Version of the Service and to respond to problems with the Deprecated Version of the Service deemed by Google in its discretion to be critical." Seriously, I've done the research and I've used the different solutions that are available today. GAE is one of the best solutions out there at this time for my type of application, which is pretty much just a straight up webapp. The only difference I have is that I have really bursty traffic as a ton of people all try to register for an event at the same time and then go away. This means, I need something to truly auto scale up and down without requiring me to do work or know my load in advance. No other solution offers the level of PaaS that GAE offers and at the price.

randall 15y ago

PS- Partychat is amazing. If you use Google Apps for your biz, Partychat is like the missing chatroom feature.