12 ms·

Issue 8788 - Every day around 9 AM Brussels time, huge drop in GAE performance

258 points by thijser 14y ago

mos 14y ago
Customer support of Google really sucks! Currently the GAE cloud has a reliability problem (also for new customers). Instances are restarted like crazy. This leads to downtimes. But that's not enough. Customers have even to pay more(!) instance hours because of this. There is the running gag on the mailing-list: "Whenever GAE is unreliable for weeks Google needed to make revenue targets ;-)" References Current Issue: http://code.google.com/p/googleappengine/issues/detail?id=8844 http://code.google.com/p/googleappengine/issues/detail?id=88... Same issue from last year that took weeks to be resolved (check last comments!): http://code.google.com/p/googleappengine/issues/detail?id=8004 http://code.google.com/p/googleappengine/issues/detail?id=80... Some Pros and Cons of Google App Engine in this blog-post: http://www.mosbase.com/ http://www.mosbase.com/
zwischenzug 14y ago
I manage the 3rd line support of some of the busiest websites in the world (we provide back-end e-commerce software). I can't say I think much of google's response here. Nearly two weeks before the first comment, and then shut down after 2 days and a question directed at who knows who, and no explanation? The analysis elsewhere on here suggests they're violating SLA, so this should get more attention. I'm guessing support is under-resourced @ google, and the culture of support is a bit shabby (no acknowledgement of inconvenience or indication or evidence of work undertaken in the background) - hardly surprising for a large-scale software business based on free services.
brown9-2 14y ago
A bug tracker seems like a horrible way to report production (or non-production) support issues. This is the same bug tracker OSS projects on Google Code use. Is it really helpful for the public to comment on my support request? Seems like the signal to noise ratio would be quite low, and then you get inane comments like: I got here from HackerNews, but after seeing the original poster spam the forums in multiple places and have a bad attitude, I can't blame Google for not fixing what looks to me like a non-issue. Fuck 'em. You have to believe that the choice of tools has some bearing on the quality of the response from Google. Seems like there is very little incentive for any "Project members" to trawl through open bug reports when no one is ever responsible.
logn 14y ago
Ironically, the guy who closed this issue owns this project: https://code.google.com/p/sentimentally/ https://code.google.com/p/sentimentally/ "sentimentally is a tool that determines sentiment of your emails. Once determined, it helps you gauge your relationships with co-workers, customers, friends, or other individuals based on the tone of your conversations with these people."
hosay123 14y ago
> I'm going to assume everyone experiencing this issue is using M/S. Upgrading to HRD will solve your issue. This is the reason I abandoned AE and part of why adopting a platform that isn't standardized is incredibly dangerous. The problem is technical debt constantly accrues even when you aren't making changes. Even though the API was unchanged, HRD differs subtly enough that breakage can occur on any non-trivial project. Edge cases (how indices behave within transactions comes to mind, but there are plenty more examples) will see new semantics compared to M/S, and so this "upgrade" involves not only thorough testing and auditing, but likely also code changes and potentially significant engineering hours. http://goo.gl/HVuaC http://goo.gl/HVuaC: These techniques are not needed with the (now deprecated) Master/Slave Datastore, which always returns strongly consistent results for all queries. This means a project written and signed off circa 2011 requires mandatory engineering costs just to continue running in a functioning and supported fashion. An AE app will never quite resemble that ancient perl5 behemoth running uninterrupted since 1997, as the underlying implementation and recommended APIs are constantly modified and replaced (Datastore, NDB, Python major version). "A strong test suite will save your soul!" I hear you say, tests that a small project might have survived without if targeting any other platform, and testing on AppEngine is also yet another moving target (for example, testing nested subrequests was all but impossible using the SDK until relatively recently). The promise was a carefree life for a project willing to code against their proprietary APIs; the reality is a constantly moving target, "not quite free" autoscaling and the threat that while you're asleep an unannounced change will take down your app (I could name a few, but as many will attest this has happened regularly since launch).
- Benferhat 14y ago
  The Master/Slave datastore has been deprecated for almost a year. [0] Beyond the latency issues, it simply wasn't reliable; both reads and writes were failing way too often. I'm glad GAE is focusing its resources on the HRD. [0] http://googleappengine.blogspot.com/2012/04/masterslave-datastore-thanks-for-all.html http://googleappengine.blogspot.com/2012/04/masterslave-data...
  - muraiki 14y ago
    The problem is occurring during the same timeframe for someone else who is using the HRD: https://code.google.com/p/googleappengine/issues/detail?id=8788#c12 https://code.google.com/p/googleappengine/issues/detail?id=8...
saosebastiao 14y ago
Ahh the perils of being a customer of Google.
deleted 14y ago
[deleted]
sgift 14y ago
It is interesting that basically no one (including the news poster) noticed that there is an comment (#12) which states that this problem happens on HRD too. This statement may be false and/or a completely different issue, but at least it should be considered here for HN comments which state "M/S is deprecated, Google is right, just use HRD."
kelvin0 14y ago
BTW, this issue is not simply a due to MS, it also happens on HRD. So any google support apologists here, please read the BUG thread submitted by this poor customer before dismissing it simply as a 'migration issue'. I have had some issues with Google Docs (paid for premier commercial account). Some documents we had stored simply vanished from our account. After getting the run around for 3-4 days, finally a google engineer tolds us they can't help us recover the documents THEY 'lost' unless we have the URL to the document ... Thankfully someone on our team had kept the URL when I first shared that document with them (1+ year after the document had been created). Nightmare ...
xsace 14y ago
Jesus, so glad I switched to node when the hosting rates increased back in the sept 2011 with the "GAE out of preview" move
lnanek2 14y ago
I've heard a lot of people saying this is why Google can't get a lot of businesses to sign on. There's no one for the CEO to call and complain to directly when their stuff is down.
petersmagnusson 14y ago
Hi folks. We are fully aware of this issue. We've added it to external issue tracker (https://code.google.com/p/googleappengine/issues/detail?id=8901 https://code.google.com/p/googleappengine/issues/detail?id=8...), please follow up there. Response from us was initially muted because it looked like it only affected M/S apps, but it turns out (a) it can impact HRD as well, and (b) we're pretty unhappy about the level of impact for many M/S apps so we're looking at ways to resolve. It's a high priority and we're looking at a number of ways to address it. It's also a pretty interesting issue, because indirectly it's caused by (a) the large scale that App Engine is running, and (b) the large extent with which GAE is running free applications. Regardless, apologies to those who felt support was unresponsive. We are working very hard to improve support. For the sophisticated audience that comes to these pages, please link to me on Google+ to get my attention if we are failing you (https://plus.sandbox.google.com/110401818717224273095 https://plus.sandbox.google.com/110401818717224273095).
raverbashing 14y ago
" Upgrading to HRD will solve your issue. M/S is deprecated and there is a clear and straightforward path to migrating to HRD." Can anyone explain why this is not possible for them? Wonderful Google support apart, there are a lot of alternatives out there.
- log0 14y ago
  Multiple comments mentioned that this occurs in HRD as well.
  - afhof 14y ago
    Only one other comment mentioned the problem with HRD as the datastore.
    - justin66 14y ago
      And the guy having the problem with HRD doesn't count because ... ?
      - Kylekramer 14y ago
        N of 1?
      - badgar 14y ago
        Because we don't implicitly trust everything everybody says on the Internet just because they suggest Google is bad.
      - Ironlink 14y ago
        Because they did not post any kind of evidence (request logs, Pingdom report, etc.), not to mention the App ID in question (so that Google would know where to look). All too often, bug reports end up being some kind of misunderstanding.
afhof 14y ago
Comment from the WontFix mark: "M/S is deprecated and there is a clear and straightforward path to migrating to HRD." M/S was deprecated April 4, 2012, so it has been some time since the notice has been out there. High replication data store has been available for over 2 years now. Whether or not less than a year is too short a deprecation period is another issue.
- Udo 14y ago
  Seems to happen on HRD as well.
- nailer 14y ago
  Read the following message - other customers using Python on HRD are reporting the same issue as the parent, who is Java on M/S.
davedx 14y ago
Why would anyone put anything production critical on Google these days, knowing that they provide 0 support across most of their business?
- tbatterii 14y ago
  there is support if you pay for it. in general whenever I file a ticket I get a response within hours. that's at least as good(if not better) as working with any other vendor in my experience. So... where's this 0 support you are talking about?
- wiradikusuma 14y ago
  because we're paying customers? my bill is peanuts, but i know there are many big customers, e.g. Khan Academy. and also they have Premier Support which is $500/mo.
  - dylanvee 14y ago
    Yep. We (Khan Academy) have a lot riding on App Engine, and they are supportive when we run into issues with the platform.
  - raverbashing 14y ago
    One thing I've learned with Google is that they don't give a rat's ass if you're a paying customer or not. Khan Academy may have it easier, I'm sure Google won't let them down Really, go somewhere else, spend less money and have better support. (At the expense of, if you're lucky, Google will give you almost zero headaches)
    - dylanvee 14y ago
      There's nothing about Khan Academy's application (or any other customer's) that would somehow immunize it from platform-wide serving issues.
      - raverbashing 14y ago
bromley 14y ago
Quick tip for anyone making a system with high load and daily or hourly quotas: When an account is created, assign a random start time (e.g. 05:43 for daily quotas or minute 12 for hourly) to measure that account's quotas against. Then you can avoid this issue of the system getting a huge spike in load when everyone's quota refreshes at the same time.
deleted 14y ago
[deleted]
Confusion 14y ago
Well, the bug report doesn't really invite quick attention. Simply reporting your observations is not enough: you should position yourself as a competent customer, by explaining what you have done to ensure the problem isn't on your side. Mention the code hasn't changed, that you have no database cleanup cronjobs or similar running that could be interfering, etc. My first instinct when I see a report like this is: he probably has some cronjob running he forgot about; perhaps one whose performance decreased with O(n^2). By which I'm not saying that Google is right in not replying for days, but by which I am saying that as a customer, there are easy ways to get attention beyond shouting and threatening. Show it's an interesting problem and you're bound to get some techie's attention.
- spqr 14y ago
  Keep blaming that victim. As for me, I moved off GAE over a year ago and now I wear as many short skirts as I want without the fear of someone telling me I deserve to be raped by Google.
- Ensorceled 14y ago
  You pretty much only have bug reports and google groups. As other commenters note: they DID provide a minimal test case that showed the problem wasn't their side. Unfortunately with GAE support screaming and yelling is pretty much the only recourse.
- bodegajed 14y ago
  I consider him panicking more than shouting and threatening. I couldn't imagine having that kind of treatment as a vps customer else I'll be moving out asap.
  - meaty 14y ago
    If you were a VPS customer, you'd be less locked in and can just walk to another vendor so they actually are shit hot with support usually. It was obvious when I first tried it that GAE has crappy support.
    - 14y ago
neya 14y ago
I'm sorry, but this is the price you pay for running your business that is dependent TOTALLY on a 3rd party service. Forget Google, everyone out there is most likely the same, that's why it's important for you to run your 'apps' on something you have control over - Like Linode, AWS, Rackspace, Openshift, etc. and also have back-up nodes from other providers for redundancy, for emergency situations, incase of storms, etc. I would recommend trying your apps on OpenStack (Openshift in particular), which doesn't have the vendor lock-in, which you face right now.
- addlepate 14y ago
  God DAMN it, I'm sick of seeing this comment anytime a service goes down or basically anything anywhere has a problem of any sort. We depend on third parties, all of us, to some extent. Stop it!
- pestaa 14y ago
  Interesting comment. When AWS went down in the US, devops rightly said "we told you to run your apps on something you have control over."
  - jamespo 14y ago
    More relevant would be "where's your DR site?"
  - StavrosK 14y ago
    What's that? A server I own on a rack I own in a datacenter I own on a power factory I own and a telco I own? "Having control over" something is a scale, it's not binary.
    - pestaa 14y ago
      Indeed, I was trying to say it is all relative.
      - StavrosK 14y ago
nivla 14y ago
Having never used GAE, it would nice if someone could expand M/S and HRD for me. It looks like OP of the bug-report is using a depreciated feature/program which according to the Project Member is causing latency issues at a specific time daily. But that could not be the real issue since another commentator who is using the new HRD is also having the same problem. It is even frustrating for people who are reading this. All it implies is the lack of communication from Google when something goes awry. Come on Google, stop reinforcing my stereotypes about your customer support! Selling to a customer is different than selling to a business, you may have a great product at a great price but if you offer terrible CS, in the B2B world everyone is going to avoid you. It is a place where support is valued more than the product itself. Therefore, unless you start offering a decent CS, you can lower your price all you want, I will be sticking with AWS.
- deleted 14y ago
  [deleted]
- ronyeh 14y ago
  M/S is Master/Slave: https://developers.google.com/appengine/docs/python/datastore/usingmasterslave https://developers.google.com/appengine/docs/python/datastor... HRD is High Replication Datastore: https://developers.google.com/appengine/docs/adminconsole/migration https://developers.google.com/appengine/docs/adminconsole/mi... M/S is deprecated, and HRD is the new hotness (and it conveniently costs more).
  - moobirubi 14y ago
    I think it cost the same, when it was first out it cost more, but now it cost the same.
    - Ensorceled 14y ago
      For small applications it costs more because of thresholds for free services is lower. In our case it costs a lot more since some of things we were doing need another instance on HRD that we didn't need on M/S
edent 14y ago
Why would anyone expect customer support from Google? They have made it clear time and time again that they don't provide it. http://shkspr.mobi/blog/2013/02/googles-customer-contempt-conundrum/ http://shkspr.mobi/blog/2013/02/googles-customer-contempt-co...
- Ironlink 14y ago
  Support packages are available. http://googleenterprise.blogspot.com/2013/02/google-cloud-platform-introduces-new.html http://googleenterprise.blogspot.com/2013/02/google-cloud-pl...
  - EwanToo 14y ago
    The problem with saying things like "Support packages are available", is that time and again we see paying Google customers with support packages being treated awfully. For example, this guy was a paying Google customer but couldn't get help http://www.sultansolutions.com/google-voice-lost-number/ http://www.sultansolutions.com/google-voice-lost-number/ These are paying customers who are paying a non-trivial amount of money for support (though not the "Premium" support in this case, which is an extra $500 per month for GAE).
    - Ironlink 14y ago
      We're a paying customer of GAE. I think it's quite clear that paying for the basic service doesn't include support beyond the public issue tracker and the forums. Support packages start at $150 per month, and at that point you get a 4 hour response time. I think that's entirely reasonable. We have yet to sign up for a support level, but then again we're not really seeing any troubles with the service.
      - EwanToo 14y ago
        I guess the question is, if you're paying and you find what you believe is a system wide issue, do you expect a resolution? In this case, Google's response is seemingly "It might (or might not) be a system-wide issue, but we don't care - we won't fix it". There's no indication in this case that someone paying $500 a month for the premium support would get a better answer.
pyalot2 14y ago
Ok, so here's the deal. If your app runs exclusively on GAE you've essentially tied yourself to one cloud vendor. Now disregarding the respective benefits and drawbacks of google as a hosting company for your app (I would never do that), being dependent on one cloud provider is a very bad idea. No matter if you run on EC2, Azure or GAE, if you can't seamlessly switch to another provider, you're screwed. These all go down regularly and have issues. They're big companies, you're a small company, you have no such thing as "recourse". The court of public opinion will not save your company.
- rbanffy 14y ago
  Not really. You can host your own applications using AppScale or Typhoonae. Moving applications is easy. Moving data is much worse.
- kaolinite 14y ago
  Agree with this to an extent however the company I work for deploys on AWS and is far too cautious about vendor lock-in, to the point where we use AWS basically as a VPS, not a cloud service, and get none of the advantages (and all of the disadvantages, e.g. worse performance, higher price).
rachelbythebay 14y ago
Remember the old "thundering herd" problem with Apache children and things of that nature? You'd basically have a whole bunch of processes which had a listening fd from an earlier call to listen(). When a new connection would come in, the kernel would wake all of them, even though only one of them would actually have something to get. The others would go through the process for nothing. It caused a big performance hit back in the day. Well, imagine now that you have a directory or lock service where you can store things and perform atomic updates. When you do a write to something in it, it fans out to all of its clients, and they all wake up (nearly) simultaneously and receive the update. They then have to do whatever processing you do with new data of that type. If they all do this at the same time, then you have no processes left to service incoming requests. They're all identically busy with whatever mutexes held in order to apply those config changes safely, so no other work happens on those clients while they load in the new data. It's not so much that it's taking a mutex and is getting stuck for a little bit, since that's going to happen no matter what. It's that all of the children do it at the same time, so there's nobody to service your hit, and you're guaranteed to get stuck. If it was spread out, then only some percentage of incoming requests would get stuck behind this. The others would get lucky and would hit another instance which either had already run it or hadn't yet run it. I'm not saying this is what's going on here, but it sure sounds familiar.
- stingraycharles 14y ago
  On what basis do you think these issues are related? The bug report provides very little insight in what's going on, only that there's a severe performance degradation at 9AM. The thundering herd problem applied to waking up child processes is one possible explanation, but there are dozens of other explanations that are just as likely, based on the information we're provided with.
  - brown9-2 14y ago
    The commenter you are replying to is a former Google employee; the description of the lock service sounds like Chubby (http://research.google.com/archive/chubby.html http://research.google.com/archive/chubby.html), App Engine likely uses some sort of distributed directory service for keeping track of things like quotas.
mnml_ 14y ago
Gae is cool but its not worth the money. I shouldn't be out of beta.
afhof 14y ago
A GAE user sees a problem of his service being slow, writes a frantic bug report with caps and exclamation marks and threatens to leave GAE. As a GAE user myself, two questions come to mind: 1. Is GAE outside of their .9995 SLA* uptime? If they aren't, then it probably isn't important enough spend time looking into it. Customers cannot expect better than the agreed upon uptime percent, and hosting companies are obligated to reimburse customers if they go below SLA. Both of these are covered in the SLA doc. 2. Is it reproducible? So far, the bug report mentions 2 people out of GAE users. Is 2 people enough to say its a problem with GAE? One person is panicked, and the other provides few details for the bug report. *https://developers.google.com/appengine/sla https://developers.google.com/appengine/sla
- stickfigure 14y ago
  Not trying to apologize for G here, but just making sure the facts are straight: There is no SLA for M/S applications (which run on completely different infrastructure). GAE has only ever offered a SLA for HRD applications.
- efdee 14y ago
  1. 0.9995 SLA means about 6 minutes of downtime a month. Since it's a daily event, I'm guessing that yes, the SLA is violated. 2. It's a problem that is occurring daily, with a test case that has pretty much no code at all. That in itself does not prove anything, but it really makes me wonder how it could be a problem on the user's side.
  - mark-r 14y ago
    My math for downtime per month works out a little differently: (1-0.9995)x30x24x60 = 21.6 minutes. Still I mostly agree with you.
lucb1e 14y ago
To their credit, people are apparently using something that's been deprecated and should be changed regardless. At least, that was their conclusion when it was changed to wontfix. The replies are very rare and curt though, I can't really say it's quality service when you're paying for a product. Customer support from Google has always been like this as far as I've experienced and heard. There is no way to actually reach and converse with anyone, regardless whether you are paying them for the service or what kind of request it is. Once a Google employee randomly replied to a complaint of mine about Google+ (I didn't even +mention them). After a few comments and him confirming that it was added to the bugs list, I asked if it was okay to +mention him in the future with similar issues. It was okay. I did. He never showed his face again. (His profile still says "Works at Google+".) Another Google employee I know online also never replies to anything concerning Google. I know he works on the Google+ project, but I can only hope he passes on any bugs I +mentioned him in. For Youtube, you can post in their forums but merely hope for a reply. Copyright complaint disputes are no priority, either. I haven't used many paid products, but I have read about their customer support being one of the very worst and also have never been able to find a single e-mail address or phone number to get support at for any service. Edit: By the way, I would have moved away from the Google Apps Engine a long time ago if my app went down every morning during rush hour for 10 days straight.
timme 14y ago
The headline blows this out of proportion. Few people (who act obnoxious as hell) report a problem that can be solved by moving away from a deprecated system, yet they fail to even read the note because they're busy smashing exclamation marks into the issue tracker.
- Ironlink 14y ago
  I couldn't agree more. In fact, it was deprecated just about eleven months ago: http://googleappengine.blogspot.com/2012/04/masterslave-datastore-thanks-for-all.html http://googleappengine.blogspot.com/2012/04/masterslave-data... When your datastore gets deprecated, you act sooner rather than later.
  - jfoster 14y ago
    The two datastores even have the same API. As long as your app doesn't depend on the exact performance characteristics of the old one, the migration is very straightforward. I did it for one of my apps in a morning and was done well before lunch.
- efdee 14y ago
  The problem apparently also occurs on the non-deprecated system. I can understand their frustration after not getting a reply for X days on what seems to be a critical issue for them. That's not "obnoxious as hell", that's customers panicking. You really don't want your customers panicking about your service.
benjaminwootton 14y ago
Google support is an absolute disgrace. I had a Nexus 7 go AWOL at Christmas and I've never had such a shambolic customer service experience. They have absolutely no respect or customer service ethos when it comes to people who are actually paying them real cash money. Not in a million years would I sign off on hosting a production project on App Engine.
- PostOnce 14y ago
  Google support borders upon the farcical. It doesn't appear to be costing them too much money in the grand scheme of things, which is sort of surprising to me. I had a support guy tell me I had to get Apple's legal team to contact Google so I could use the "Mac" trademark, because I happened to be selling a piece of software that ran on OS X. Like that's ever going to happen. My ad simply said "Try ____, a better way to _____ on Windows and Mac.", linking them to http://www.apple.com/legal/trademark/guidelinesfor3rdparties.html http://www.apple.com/legal/trademark/guidelinesfor3rdparties... didn't quite cut it, apparently, even though it clearly states that such use is acceptable under "2. Compatibility" near the top of the page. i.e. I can say my product runs on Mac if in fact it runs on Mac. Approving a ten word ad takes Google over a week, in my experience. Baffling. All this with their adwords $100 free trial. All that trial did was convince me that I should never ever in the life of the universe commit any money to Google, because they made it starkly apparent that I would never get what I paid for... running honest ads for honest products in a reasonable timeframe. I went with other ad networks in the end and had zero trouble whatsoever, and infinitely faster approval times. I suppose I may have had a smaller audience, but the headaches Google causes aren't worth the extra money. Someone is going to come along and pull the rug out from under Google eventually. You can't rest on your laurels forever. /rant
- Ironlink 14y ago
  This customer is complaining about a service component which has been deprecated since almost 11 months ago. There is a tool which migrates application data from the old datastore to the new one. When you don't move off of deprecated infrastructure, I'd say you've set yourself up for problems.
  - 14y ago
okku 14y ago
I am also a GAE-user, I have had no problems like the OP. But I start to miss a fundamental feature, sockets. I have worked around it by using other services and polling. Maybe wrong forum, but is there any infrastructure templates for setting up a scalable web/db/loadbalancer/memcached for a simple tradional webservice, in my case a game? I want to be able to sleep at night, and easily scale up by adding some more machines in case of higher load. I could use denormalized myslq/postgre or mongodb for speed. Preferred language is Python (or maybe c# or java). Any ideas?
- ThingTwo 14y ago
  Channels.
  - okku 14y ago
    Channels are for communicating with javascript clients. I want to communicate with other servers and applications.
    - petersmagnusson 14y ago
      stay tuned
- EwanToo 14y ago
  Depending on your budget (isn't it always..?), speak to Rightscale - they provide a set of frameworks to deploy infrastructure to various cloud platforms, and can handle auto-scaling and all that stuff.
chris_wot 14y ago
Well, there's a good reason not to use this service.
killermonkeys 14y ago
Many on the thread say the reporters are over-reacting. They are not. What would amazon do? They would not consider this an issue, would respond in less than 24 hours, and would take complete responsibility. GAE is a pay service. I think this level of service is pathetic. As noted the only attempt at diagnosis is completely wrong (even the reporter is not on MS) and very late.
kushti 14y ago
Never pay Google. It has terrible support for all products
- rbanffy 14y ago
  Not my experience at all. While it's true I never had much trouble with App Engine, when I had problems, they were solved rather quickly.
Al-Khwarizmi 14y ago
Not surprising... the second most-voted bug in Google Code, reported exactly a year ago ( http://code.google.com/p/support/issues/detail?id=24324 http://code.google.com/p/support/issues/detail?id=24324 ) deplores the removal of a feature that was already there (the Updates page) and was the single most useful feature in Google Code for many of us. After one year and more than 800 people registering their interest on the issue, they haven't even explained why they removed it or whether there are any plans of brinding it back.
mrerrormessage 14y ago
It happens that 9 AM Brussels time is midnight pacific time. I'm sure Google is running some maintentance cron at midnight thinking "This is a low demand time," and it is, across the US, but not in Brussels. These are old instances, and Google probably doesn't want to re-time or rewrite the cron job to be more efficient.
- guard-of-terra 14y ago
  Do people go to bed that early in the US? Looked at my project's charts, the low demand time is definitely 4AM and midnight is 40% of peak.

258 points by thijser 14y ago

mos 14y ago

Customer support of Google really sucks! Currently the GAE cloud has a reliability problem (also for new customers). Instances are restarted like crazy. This leads to downtimes. But that's not enough. Customers have even to pay more(!) instance hours because of this. There is the running gag on the mailing-list: "Whenever GAE is unreliable for weeks Google needed to make revenue targets ;-)" References Current Issue: http://code.google.com/p/googleappengine/issues/detail?id=8844 http://code.google.com/p/googleappengine/issues/detail?id=88... Same issue from last year that took weeks to be resolved (check last comments!): http://code.google.com/p/googleappengine/issues/detail?id=8004 http://code.google.com/p/googleappengine/issues/detail?id=80... Some Pros and Cons of Google App Engine in this blog-post: http://www.mosbase.com/ http://www.mosbase.com/

zwischenzug 14y ago

I manage the 3rd line support of some of the busiest websites in the world (we provide back-end e-commerce software). I can't say I think much of google's response here. Nearly two weeks before the first comment, and then shut down after 2 days and a question directed at who knows who, and no explanation? The analysis elsewhere on here suggests they're violating SLA, so this should get more attention. I'm guessing support is under-resourced @ google, and the culture of support is a bit shabby (no acknowledgement of inconvenience or indication or evidence of work undertaken in the background) - hardly surprising for a large-scale software business based on free services.

brown9-2 14y ago

A bug tracker seems like a horrible way to report production (or non-production) support issues. This is the same bug tracker OSS projects on Google Code use. Is it really helpful for the public to comment on my support request? Seems like the signal to noise ratio would be quite low, and then you get inane comments like: I got here from HackerNews, but after seeing the original poster spam the forums in multiple places and have a bad attitude, I can't blame Google for not fixing what looks to me like a non-issue. Fuck 'em. You have to believe that the choice of tools has some bearing on the quality of the response from Google. Seems like there is very little incentive for any "Project members" to trawl through open bug reports when no one is ever responsible.

logn 14y ago

Ironically, the guy who closed this issue owns this project: https://code.google.com/p/sentimentally/ https://code.google.com/p/sentimentally/ "sentimentally is a tool that determines sentiment of your emails. Once determined, it helps you gauge your relationships with co-workers, customers, friends, or other individuals based on the tone of your conversations with these people."

hosay123 14y ago

> I'm going to assume everyone experiencing this issue is using M/S. Upgrading to HRD will solve your issue. This is the reason I abandoned AE and part of why adopting a platform that isn't standardized is incredibly dangerous. The problem is technical debt constantly accrues even when you aren't making changes. Even though the API was unchanged, HRD differs subtly enough that breakage can occur on any non-trivial project. Edge cases (how indices behave within transactions comes to mind, but there are plenty more examples) will see new semantics compared to M/S, and so this "upgrade" involves not only thorough testing and auditing, but likely also code changes and potentially significant engineering hours. http://goo.gl/HVuaC http://goo.gl/HVuaC: These techniques are not needed with the (now deprecated) Master/Slave Datastore, which always returns strongly consistent results for all queries. This means a project written and signed off circa 2011 requires mandatory engineering costs just to continue running in a functioning and supported fashion. An AE app will never quite resemble that ancient perl5 behemoth running uninterrupted since 1997, as the underlying implementation and recommended APIs are constantly modified and replaced (Datastore, NDB, Python major version). "A strong test suite will save your soul!" I hear you say, tests that a small project might have survived without if targeting any other platform, and testing on AppEngine is also yet another moving target (for example, testing nested subrequests was all but impossible using the SDK until relatively recently). The promise was a carefree life for a project willing to code against their proprietary APIs; the reality is a constantly moving target, "not quite free" autoscaling and the threat that while you're asleep an unannounced change will take down your app (I could name a few, but as many will attest this has happened regularly since launch).

Benferhat 14y ago
The Master/Slave datastore has been deprecated for almost a year. [0] Beyond the latency issues, it simply wasn't reliable; both reads and writes were failing way too often. I'm glad GAE is focusing its resources on the HRD. [0] http://googleappengine.blogspot.com/2012/04/masterslave-datastore-thanks-for-all.html http://googleappengine.blogspot.com/2012/04/masterslave-data...
- muraiki 14y ago
  The problem is occurring during the same timeframe for someone else who is using the HRD: https://code.google.com/p/googleappengine/issues/detail?id=8788#c12 https://code.google.com/p/googleappengine/issues/detail?id=8...

saosebastiao 14y ago

Ahh the perils of being a customer of Google.

deleted 14y ago

[deleted]

sgift 14y ago

It is interesting that basically no one (including the news poster) noticed that there is an comment (#12) which states that this problem happens on HRD too. This statement may be false and/or a completely different issue, but at least it should be considered here for HN comments which state "M/S is deprecated, Google is right, just use HRD."

kelvin0 14y ago

BTW, this issue is not simply a due to MS, it also happens on HRD. So any google support apologists here, please read the BUG thread submitted by this poor customer before dismissing it simply as a 'migration issue'. I have had some issues with Google Docs (paid for premier commercial account). Some documents we had stored simply vanished from our account. After getting the run around for 3-4 days, finally a google engineer tolds us they can't help us recover the documents THEY 'lost' unless we have the URL to the document ... Thankfully someone on our team had kept the URL when I first shared that document with them (1+ year after the document had been created). Nightmare ...

xsace 14y ago

Jesus, so glad I switched to node when the hosting rates increased back in the sept 2011 with the "GAE out of preview" move

lnanek2 14y ago

I've heard a lot of people saying this is why Google can't get a lot of businesses to sign on. There's no one for the CEO to call and complain to directly when their stuff is down.

petersmagnusson 14y ago

Hi folks. We are fully aware of this issue. We've added it to external issue tracker (https://code.google.com/p/googleappengine/issues/detail?id=8901 https://code.google.com/p/googleappengine/issues/detail?id=8...), please follow up there. Response from us was initially muted because it looked like it only affected M/S apps, but it turns out (a) it can impact HRD as well, and (b) we're pretty unhappy about the level of impact for many M/S apps so we're looking at ways to resolve. It's a high priority and we're looking at a number of ways to address it. It's also a pretty interesting issue, because indirectly it's caused by (a) the large scale that App Engine is running, and (b) the large extent with which GAE is running free applications. Regardless, apologies to those who felt support was unresponsive. We are working very hard to improve support. For the sophisticated audience that comes to these pages, please link to me on Google+ to get my attention if we are failing you (https://plus.sandbox.google.com/110401818717224273095 https://plus.sandbox.google.com/110401818717224273095).

raverbashing 14y ago

" Upgrading to HRD will solve your issue. M/S is deprecated and there is a clear and straightforward path to migrating to HRD." Can anyone explain why this is not possible for them? Wonderful Google support apart, there are a lot of alternatives out there.

log0 14y ago
Multiple comments mentioned that this occurs in HRD as well.
- afhof 14y ago
  Only one other comment mentioned the problem with HRD as the datastore.
  - justin66 14y ago
    And the guy having the problem with HRD doesn't count because ... ?
    - Kylekramer 14y ago
      N of 1?
    - badgar 14y ago
      Because we don't implicitly trust everything everybody says on the Internet just because they suggest Google is bad.
    - Ironlink 14y ago
      Because they did not post any kind of evidence (request logs, Pingdom report, etc.), not to mention the App ID in question (so that Google would know where to look). All too often, bug reports end up being some kind of misunderstanding.

afhof 14y ago

Comment from the WontFix mark: "M/S is deprecated and there is a clear and straightforward path to migrating to HRD." M/S was deprecated April 4, 2012, so it has been some time since the notice has been out there. High replication data store has been available for over 2 years now. Whether or not less than a year is too short a deprecation period is another issue.

Udo 14y ago
Seems to happen on HRD as well.
nailer 14y ago
Read the following message - other customers using Python on HRD are reporting the same issue as the parent, who is Java on M/S.

davedx 14y ago

Why would anyone put anything production critical on Google these days, knowing that they provide 0 support across most of their business?

tbatterii 14y ago
there is support if you pay for it. in general whenever I file a ticket I get a response within hours. that's at least as good(if not better) as working with any other vendor in my experience. So... where's this 0 support you are talking about?
wiradikusuma 14y ago
because we're paying customers? my bill is peanuts, but i know there are many big customers, e.g. Khan Academy. and also they have Premier Support which is $500/mo.
- dylanvee 14y ago
  Yep. We (Khan Academy) have a lot riding on App Engine, and they are supportive when we run into issues with the platform.
- raverbashing 14y ago
  One thing I've learned with Google is that they don't give a rat's ass if you're a paying customer or not. Khan Academy may have it easier, I'm sure Google won't let them down Really, go somewhere else, spend less money and have better support. (At the expense of, if you're lucky, Google will give you almost zero headaches)
  - dylanvee 14y ago
    There's nothing about Khan Academy's application (or any other customer's) that would somehow immunize it from platform-wide serving issues.
    - raverbashing 14y ago

bromley 14y ago

Quick tip for anyone making a system with high load and daily or hourly quotas: When an account is created, assign a random start time (e.g. 05:43 for daily quotas or minute 12 for hourly) to measure that account's quotas against. Then you can avoid this issue of the system getting a huge spike in load when everyone's quota refreshes at the same time.

deleted 14y ago

[deleted]

Confusion 14y ago

Well, the bug report doesn't really invite quick attention. Simply reporting your observations is not enough: you should position yourself as a competent customer, by explaining what you have done to ensure the problem isn't on your side. Mention the code hasn't changed, that you have no database cleanup cronjobs or similar running that could be interfering, etc. My first instinct when I see a report like this is: he probably has some cronjob running he forgot about; perhaps one whose performance decreased with O(n^2). By which I'm not saying that Google is right in not replying for days, but by which I am saying that as a customer, there are easy ways to get attention beyond shouting and threatening. Show it's an interesting problem and you're bound to get some techie's attention.

spqr 14y ago
Keep blaming that victim. As for me, I moved off GAE over a year ago and now I wear as many short skirts as I want without the fear of someone telling me I deserve to be raped by Google.
Ensorceled 14y ago
You pretty much only have bug reports and google groups. As other commenters note: they DID provide a minimal test case that showed the problem wasn't their side. Unfortunately with GAE support screaming and yelling is pretty much the only recourse.
bodegajed 14y ago
I consider him panicking more than shouting and threatening. I couldn't imagine having that kind of treatment as a vps customer else I'll be moving out asap.
- meaty 14y ago
  If you were a VPS customer, you'd be less locked in and can just walk to another vendor so they actually are shit hot with support usually. It was obvious when I first tried it that GAE has crappy support.
  - 14y ago

neya 14y ago

I'm sorry, but this is the price you pay for running your business that is dependent TOTALLY on a 3rd party service. Forget Google, everyone out there is most likely the same, that's why it's important for you to run your 'apps' on something you have control over - Like Linode, AWS, Rackspace, Openshift, etc. and also have back-up nodes from other providers for redundancy, for emergency situations, incase of storms, etc. I would recommend trying your apps on OpenStack (Openshift in particular), which doesn't have the vendor lock-in, which you face right now.

addlepate 14y ago
God DAMN it, I'm sick of seeing this comment anytime a service goes down or basically anything anywhere has a problem of any sort. We depend on third parties, all of us, to some extent. Stop it!
pestaa 14y ago
Interesting comment. When AWS went down in the US, devops rightly said "we told you to run your apps on something you have control over."
- jamespo 14y ago
  More relevant would be "where's your DR site?"
- StavrosK 14y ago
  What's that? A server I own on a rack I own in a datacenter I own on a power factory I own and a telco I own? "Having control over" something is a scale, it's not binary.
  - pestaa 14y ago
    Indeed, I was trying to say it is all relative.
    - StavrosK 14y ago

nivla 14y ago

Having never used GAE, it would nice if someone could expand M/S and HRD for me. It looks like OP of the bug-report is using a depreciated feature/program which according to the Project Member is causing latency issues at a specific time daily. But that could not be the real issue since another commentator who is using the new HRD is also having the same problem. It is even frustrating for people who are reading this. All it implies is the lack of communication from Google when something goes awry. Come on Google, stop reinforcing my stereotypes about your customer support! Selling to a customer is different than selling to a business, you may have a great product at a great price but if you offer terrible CS, in the B2B world everyone is going to avoid you. It is a place where support is valued more than the product itself. Therefore, unless you start offering a decent CS, you can lower your price all you want, I will be sticking with AWS.

deleted 14y ago
[deleted]
ronyeh 14y ago
M/S is Master/Slave: https://developers.google.com/appengine/docs/python/datastore/usingmasterslave https://developers.google.com/appengine/docs/python/datastor... HRD is High Replication Datastore: https://developers.google.com/appengine/docs/adminconsole/migration https://developers.google.com/appengine/docs/adminconsole/mi... M/S is deprecated, and HRD is the new hotness (and it conveniently costs more).
- moobirubi 14y ago
  I think it cost the same, when it was first out it cost more, but now it cost the same.
  - Ensorceled 14y ago
    For small applications it costs more because of thresholds for free services is lower. In our case it costs a lot more since some of things we were doing need another instance on HRD that we didn't need on M/S

edent 14y ago

Why would anyone expect customer support from Google? They have made it clear time and time again that they don't provide it. http://shkspr.mobi/blog/2013/02/googles-customer-contempt-conundrum/ http://shkspr.mobi/blog/2013/02/googles-customer-contempt-co...

Ironlink 14y ago
Support packages are available. http://googleenterprise.blogspot.com/2013/02/google-cloud-platform-introduces-new.html http://googleenterprise.blogspot.com/2013/02/google-cloud-pl...
- EwanToo 14y ago
  The problem with saying things like "Support packages are available", is that time and again we see paying Google customers with support packages being treated awfully. For example, this guy was a paying Google customer but couldn't get help http://www.sultansolutions.com/google-voice-lost-number/ http://www.sultansolutions.com/google-voice-lost-number/ These are paying customers who are paying a non-trivial amount of money for support (though not the "Premium" support in this case, which is an extra $500 per month for GAE).
  - Ironlink 14y ago
    We're a paying customer of GAE. I think it's quite clear that paying for the basic service doesn't include support beyond the public issue tracker and the forums. Support packages start at $150 per month, and at that point you get a 4 hour response time. I think that's entirely reasonable. We have yet to sign up for a support level, but then again we're not really seeing any troubles with the service.
    - EwanToo 14y ago
      I guess the question is, if you're paying and you find what you believe is a system wide issue, do you expect a resolution? In this case, Google's response is seemingly "It might (or might not) be a system-wide issue, but we don't care - we won't fix it". There's no indication in this case that someone paying $500 a month for the premium support would get a better answer.

pyalot2 14y ago

Ok, so here's the deal. If your app runs exclusively on GAE you've essentially tied yourself to one cloud vendor. Now disregarding the respective benefits and drawbacks of google as a hosting company for your app (I would never do that), being dependent on one cloud provider is a very bad idea. No matter if you run on EC2, Azure or GAE, if you can't seamlessly switch to another provider, you're screwed. These all go down regularly and have issues. They're big companies, you're a small company, you have no such thing as "recourse". The court of public opinion will not save your company.

rbanffy 14y ago
Not really. You can host your own applications using AppScale or Typhoonae. Moving applications is easy. Moving data is much worse.
kaolinite 14y ago
Agree with this to an extent however the company I work for deploys on AWS and is far too cautious about vendor lock-in, to the point where we use AWS basically as a VPS, not a cloud service, and get none of the advantages (and all of the disadvantages, e.g. worse performance, higher price).

rachelbythebay 14y ago

Remember the old "thundering herd" problem with Apache children and things of that nature? You'd basically have a whole bunch of processes which had a listening fd from an earlier call to listen(). When a new connection would come in, the kernel would wake all of them, even though only one of them would actually have something to get. The others would go through the process for nothing. It caused a big performance hit back in the day. Well, imagine now that you have a directory or lock service where you can store things and perform atomic updates. When you do a write to something in it, it fans out to all of its clients, and they all wake up (nearly) simultaneously and receive the update. They then have to do whatever processing you do with new data of that type. If they all do this at the same time, then you have no processes left to service incoming requests. They're all identically busy with whatever mutexes held in order to apply those config changes safely, so no other work happens on those clients while they load in the new data. It's not so much that it's taking a mutex and is getting stuck for a little bit, since that's going to happen no matter what. It's that all of the children do it at the same time, so there's nobody to service your hit, and you're guaranteed to get stuck. If it was spread out, then only some percentage of incoming requests would get stuck behind this. The others would get lucky and would hit another instance which either had already run it or hadn't yet run it. I'm not saying this is what's going on here, but it sure sounds familiar.

stingraycharles 14y ago
On what basis do you think these issues are related? The bug report provides very little insight in what's going on, only that there's a severe performance degradation at 9AM. The thundering herd problem applied to waking up child processes is one possible explanation, but there are dozens of other explanations that are just as likely, based on the information we're provided with.
- brown9-2 14y ago
  The commenter you are replying to is a former Google employee; the description of the lock service sounds like Chubby (http://research.google.com/archive/chubby.html http://research.google.com/archive/chubby.html), App Engine likely uses some sort of distributed directory service for keeping track of things like quotas.

mnml_ 14y ago

Gae is cool but its not worth the money. I shouldn't be out of beta.

afhof 14y ago

A GAE user sees a problem of his service being slow, writes a frantic bug report with caps and exclamation marks and threatens to leave GAE. As a GAE user myself, two questions come to mind: 1. Is GAE outside of their .9995 SLA* uptime? If they aren't, then it probably isn't important enough spend time looking into it. Customers cannot expect better than the agreed upon uptime percent, and hosting companies are obligated to reimburse customers if they go below SLA. Both of these are covered in the SLA doc. 2. Is it reproducible? So far, the bug report mentions 2 people out of GAE users. Is 2 people enough to say its a problem with GAE? One person is panicked, and the other provides few details for the bug report. *https://developers.google.com/appengine/sla https://developers.google.com/appengine/sla

stickfigure 14y ago
Not trying to apologize for G here, but just making sure the facts are straight: There is no SLA for M/S applications (which run on completely different infrastructure). GAE has only ever offered a SLA for HRD applications.
efdee 14y ago
1. 0.9995 SLA means about 6 minutes of downtime a month. Since it's a daily event, I'm guessing that yes, the SLA is violated. 2. It's a problem that is occurring daily, with a test case that has pretty much no code at all. That in itself does not prove anything, but it really makes me wonder how it could be a problem on the user's side.
- mark-r 14y ago
  My math for downtime per month works out a little differently: (1-0.9995)x30x24x60 = 21.6 minutes. Still I mostly agree with you.

lucb1e 14y ago

To their credit, people are apparently using something that's been deprecated and should be changed regardless. At least, that was their conclusion when it was changed to wontfix. The replies are very rare and curt though, I can't really say it's quality service when you're paying for a product. Customer support from Google has always been like this as far as I've experienced and heard. There is no way to actually reach and converse with anyone, regardless whether you are paying them for the service or what kind of request it is. Once a Google employee randomly replied to a complaint of mine about Google+ (I didn't even +mention them). After a few comments and him confirming that it was added to the bugs list, I asked if it was okay to +mention him in the future with similar issues. It was okay. I did. He never showed his face again. (His profile still says "Works at Google+".) Another Google employee I know online also never replies to anything concerning Google. I know he works on the Google+ project, but I can only hope he passes on any bugs I +mentioned him in. For Youtube, you can post in their forums but merely hope for a reply. Copyright complaint disputes are no priority, either. I haven't used many paid products, but I have read about their customer support being one of the very worst and also have never been able to find a single e-mail address or phone number to get support at for any service. Edit: By the way, I would have moved away from the Google Apps Engine a long time ago if my app went down every morning during rush hour for 10 days straight.

timme 14y ago

The headline blows this out of proportion. Few people (who act obnoxious as hell) report a problem that can be solved by moving away from a deprecated system, yet they fail to even read the note because they're busy smashing exclamation marks into the issue tracker.

Ironlink 14y ago
I couldn't agree more. In fact, it was deprecated just about eleven months ago: http://googleappengine.blogspot.com/2012/04/masterslave-datastore-thanks-for-all.html http://googleappengine.blogspot.com/2012/04/masterslave-data... When your datastore gets deprecated, you act sooner rather than later.
- jfoster 14y ago
  The two datastores even have the same API. As long as your app doesn't depend on the exact performance characteristics of the old one, the migration is very straightforward. I did it for one of my apps in a morning and was done well before lunch.
efdee 14y ago
The problem apparently also occurs on the non-deprecated system. I can understand their frustration after not getting a reply for X days on what seems to be a critical issue for them. That's not "obnoxious as hell", that's customers panicking. You really don't want your customers panicking about your service.

benjaminwootton 14y ago

Google support is an absolute disgrace. I had a Nexus 7 go AWOL at Christmas and I've never had such a shambolic customer service experience. They have absolutely no respect or customer service ethos when it comes to people who are actually paying them real cash money. Not in a million years would I sign off on hosting a production project on App Engine.

PostOnce 14y ago
Google support borders upon the farcical. It doesn't appear to be costing them too much money in the grand scheme of things, which is sort of surprising to me. I had a support guy tell me I had to get Apple's legal team to contact Google so I could use the "Mac" trademark, because I happened to be selling a piece of software that ran on OS X. Like that's ever going to happen. My ad simply said "Try ____, a better way to _____ on Windows and Mac.", linking them to http://www.apple.com/legal/trademark/guidelinesfor3rdparties.html http://www.apple.com/legal/trademark/guidelinesfor3rdparties... didn't quite cut it, apparently, even though it clearly states that such use is acceptable under "2. Compatibility" near the top of the page. i.e. I can say my product runs on Mac if in fact it runs on Mac. Approving a ten word ad takes Google over a week, in my experience. Baffling. All this with their adwords $100 free trial. All that trial did was convince me that I should never ever in the life of the universe commit any money to Google, because they made it starkly apparent that I would never get what I paid for... running honest ads for honest products in a reasonable timeframe. I went with other ad networks in the end and had zero trouble whatsoever, and infinitely faster approval times. I suppose I may have had a smaller audience, but the headaches Google causes aren't worth the extra money. Someone is going to come along and pull the rug out from under Google eventually. You can't rest on your laurels forever. /rant
Ironlink 14y ago
This customer is complaining about a service component which has been deprecated since almost 11 months ago. There is a tool which migrates application data from the old datastore to the new one. When you don't move off of deprecated infrastructure, I'd say you've set yourself up for problems.
- 14y ago

okku 14y ago

I am also a GAE-user, I have had no problems like the OP. But I start to miss a fundamental feature, sockets. I have worked around it by using other services and polling. Maybe wrong forum, but is there any infrastructure templates for setting up a scalable web/db/loadbalancer/memcached for a simple tradional webservice, in my case a game? I want to be able to sleep at night, and easily scale up by adding some more machines in case of higher load. I could use denormalized myslq/postgre or mongodb for speed. Preferred language is Python (or maybe c# or java). Any ideas?

ThingTwo 14y ago
Channels.
- okku 14y ago
  Channels are for communicating with javascript clients. I want to communicate with other servers and applications.
  - petersmagnusson 14y ago
    stay tuned
EwanToo 14y ago
Depending on your budget (isn't it always..?), speak to Rightscale - they provide a set of frameworks to deploy infrastructure to various cloud platforms, and can handle auto-scaling and all that stuff.

chris_wot 14y ago

Well, there's a good reason not to use this service.

killermonkeys 14y ago

Many on the thread say the reporters are over-reacting. They are not. What would amazon do? They would not consider this an issue, would respond in less than 24 hours, and would take complete responsibility. GAE is a pay service. I think this level of service is pathetic. As noted the only attempt at diagnosis is completely wrong (even the reporter is not on MS) and very late.

kushti 14y ago

Never pay Google. It has terrible support for all products

rbanffy 14y ago
Not my experience at all. While it's true I never had much trouble with App Engine, when I had problems, they were solved rather quickly.

Al-Khwarizmi 14y ago

Not surprising... the second most-voted bug in Google Code, reported exactly a year ago ( http://code.google.com/p/support/issues/detail?id=24324 http://code.google.com/p/support/issues/detail?id=24324 ) deplores the removal of a feature that was already there (the Updates page) and was the single most useful feature in Google Code for many of us. After one year and more than 800 people registering their interest on the issue, they haven't even explained why they removed it or whether there are any plans of brinding it back.

mrerrormessage 14y ago

It happens that 9 AM Brussels time is midnight pacific time. I'm sure Google is running some maintentance cron at midnight thinking "This is a low demand time," and it is, across the US, but not in Brussels. These are old instances, and Google probably doesn't want to re-time or rewrite the cron job to be more efficient.

guard-of-terra 14y ago
Do people go to bed that early in the US? Looked at my project's charts, the low demand time is definitely 4AM and midnight is 40% of peak.