imbriaco

15 ms·

91.
by imbriaco 14y ago
Thanks for reminding me, my apologies. I was focused entirely on the network components since that's the part of the problem I personally responded to and overlooked it. I've updated the blog post: Note: I initially forgot to mention that w
92.
by imbriaco 14y ago
Have you contacted support? We are not having any known problems today and our traffic patterns look normal.
93.
by imbriaco 14y ago
Because your responsibility is to your kids and not to some idealistic vision of what you could be doing to improve the school system as a whole.
94.
by imbriaco 14y ago
It's because we update the status site from our chat room via Hubot and whoever posted that particular update didn't realize that they didn't need to put quotes around the body of the message.
95.
by imbriaco 15y ago
It's entirely possible that the status site had an unrelated issue. It happens. That said, I ran their stack until 2 weeks ago, so I can speak authoritatively on this: Nothing from their stack is shared with the status site. Period.
96.
by imbriaco 15y ago
I am probably biased, since I wrote many of them, but I like to think Heroku has a pretty good reputation for writing meaningful public post-mortems when there are large scale outages. It's safe to give them the benefit of the doubt that th
97.
by imbriaco 15y ago
The devcenter site is absolutely hosted on the Heroku platform. I don't have first hand knowledge of this specific incident, but I do have a very deep understanding how the Heroku platform as a whole works. It is incredibly likely for many
98.
by imbriaco 15y ago
The Heroku status site is indeed hosted outside of both the Heroku platform and EC2. It has been that way for at least a couple of years that I have direct knowledge of, and probably longer.
99.
by imbriaco 15y ago
While BGP is certainly better, the vast majority of DNS servers honor TTL nicely these days. In the old days, there used to be many problems, but I have been shocked at how well DNS failover has worked in the past few years.
100.
by imbriaco 15y ago
It's definitely not running on Heroku.
101.
by imbriaco 15y ago
Also, if you're handling user uploaded content you shouldn't just be writing it to local disk. You're going to have to figure out how to do off-instance blob storage at some point if you scale beyond a single server or want any redundancy.
102.
by imbriaco 15y ago
I agree with the author that hstore is very interesting, but the data structures are not the key selling point in the NoSQL space in my opinion. The most overlooked advantage to things like Cassandra and Riak are the fact that you have no s
103.
by imbriaco 15y ago
It turns out, that's exactly what I want in the world of a configuration management system like Puppet or Chef. I don't want my package to try to be clever and configure itself or make lots of assumptions about how I intend to use it. I'll
104.
by imbriaco 15y ago
You actually don't have to live in _that_ large of a city. The Raleigh/Durham MSAs combine for about 1.6M in total population spread over a pretty sprawling area and we have had LTE service for a few months. I frankly don't get the complain
105.
by imbriaco 15y ago
You're exactly right about log streaming to S3. We open sourced the tool that we use to perform that: https://github.com/heroku/wal-e
106.
by imbriaco 15y ago
Not if the cloud you're running your workload on is large enough. The Amazon Web Services cloud could absorb many Reddit sized sites without any other customers experiencing an impact. AWS is _enormous_.
107.
by imbriaco 15y ago
Did you try contacting support? Looking at that log line, I can tell you that it looks like your app took too long to respond to the request (service=30000ms) and our routing mesh timed it out (status=503, Error H12 (Request timeout). If th
108.
by imbriaco 15y ago
Closing the loop and replying to myself, I got carried away and wound up with a full Sinatra demo app that does this: https://github.com/imbriaco/heroku-redis-sshtunnel
109.
by imbriaco 15y ago
This motivated me to play around with Net::SSH port forwarding this afternoon and evening. What I ended up with was a tiny patch that allows you to forward a local UNIX domain socket to a remote TCP port over an SSH connection. It's a fun h
110.
by imbriaco 15y ago
Interestingly, my two Kinesis keyboards were also purchased in 1999 and 2000 respectively. One is a Classic and the other is a Professional since the Advantage didn't exist yet. Both keyboards are still going strong with very heavy usage. L
111.
by imbriaco 15y ago
For those who say that 37signals has allowed Basecamp to stagnate, I'd trot out as exhibit A the changelog: http://basecamphq.com/changes Being very deliberate about making sweeping changes to an application with an extremely large number
112.
by imbriaco 15y ago
Definitely get in touch with me, we (Heroku) are hiring exactly that kind of person for the operations team that I run. The company is based in San Francisco, but my team is all remote and a couple of us live here in the triangle.
113.
by imbriaco 15y ago
Um, no.
114.
by imbriaco 15y ago
The patch in the bug report seems solves the problem and allows you to stay on 10.04 LTS. It's a tiny patch and has been stable for us under production load.
115.
by imbriaco 15y ago
Not exactly. The 3 IPs that are returned for proxy.heroku.com are not fixed. They are a just subset of the available front-end proxy servers.
116.
by imbriaco 16y ago
I was one of those experiments, as the Operations Manager, and I like to think it went pretty well. As many people here have rightly pointed out, though, being hands-on is the key. I continued to do a lot of the day-to-day technical work. T
117.
by imbriaco 16y ago
You can do the same thing with the local or aggregation tier syslog daemon if desired.
118.
by imbriaco 16y ago
I would argue that you want to do the filtering at the aggregation layer and not at the application level. So while you may want to set different flags or parameters that will be logged (ie: severity, component, etc) those parameters should
119.
by imbriaco 16y ago
The streaming is from CDN, but the links to the CDN with the authorization to stream comes from EC2. The video source for the CDNs is S3.
120.
by imbriaco 16y ago
Nope, the main site is also hosted on EC2. I saw Adrian Cockroft give an abbreviated overview of their architecture recently. There's very little that isn't already on EC2 and they're actively working to move those things.

More ›