mattrobenolt

9 ms·

61.
by mattrobenolt 10y ago
> You can't add a new thing without adding a new point of failure. Every point is a point of failure. Correct, but it's an existing process. So you're right, we could ship a blatantly bad config. > Who deploys the thing
62.
by mattrobenolt 10y ago
Yep, it's definitely possible to go this route as well. We just didn't have to.
63.
by mattrobenolt 10y ago
You missed the major point where we're strictly improving and not thinking we're more available. In fact, I assume that we're less available, hence why we have a fallback from our local to S3 directly since I assume our singl
64.
by mattrobenolt 10y ago
Sure, there's definitely risk. I'm not asserting that it's literally 0 chance. But this risk of this is also tied up with other things that leverage this proxy. So it's not adding a new dependency or a new point of failu
65.
by mattrobenolt 10y ago
This is a trusted, internal, private network. The only one who could do this is the application itself, or something rogue on our network. If something were running rogue on our network, there'd be worse things it could get access to.
66.
by mattrobenolt 10y ago
Yeah, we could do something like this as well if we cared. We are easily caching 95+% of our active data in a small amount of disk space. It's not that valuable for us to have a complete replica of the full data set.
67.
by mattrobenolt 10y ago
Ironically, in our case, our availability is very core to our business. In this exact scenario, if S3 would have blocked our processing pipeline because it was down, that means we couldn't have alerted users as reliably that they were
68.
by mattrobenolt 10y ago
Technically yes. But we're pretty accustomed to this level of risk. For something like this, the pros far outweigh the cons involved. Yeah, it could fail. The maintenance overhead of this is absolutely minimal and took a handful of hou
69.
by mattrobenolt 10y ago
Fortunately, I am top engineer and am on-call for what I implement and rely on. :) This is also why I build things in a way that don't add more risk to production infrastructure. If you'd read the post, you'd see that this is
70.
by mattrobenolt 10y ago
This is correct assessment. HAProxy is running on localhost, and it strictly falls back to hitting public S3 directly if our cache is down.
71.
by mattrobenolt 10y ago
It's running on localhost to each server. So the failure event here is that somehow haproxy process would explode with the rest of the server being fine. It's much much more likely that a whole machine will die instead or a networ
72.
by mattrobenolt 10y ago
Sorry, and by reliable, I explicitly mean, the network connection from our datacenter to halfway across the country hiccuped more than I liked and was slower than I liked. Not reliability of the service itself.
73.
by mattrobenolt 10y ago
This is mostly unrelated to our original goals, which was reliability and performance. This wasn't to protect ourselves in the event of S3 going down, the timing just worked out that it saved us during that as well while also accomplis
74.
by mattrobenolt 10y ago
That's another problem that'd require much more work than just a few hours of hacking around. :)
75.
by mattrobenolt 10y ago
I'll have to look into this. I wasn't aware. Either way, I don't think that'd replace our current setup since the original intent wasn't to increase our availability. But good to know!
76.
by mattrobenolt 10y ago
Yeah, one of our goals was to not add a new/weaker point of failure. We gracefully fall back to S3 directly if our cache server is down without a hiccup. So there is no operational overhead of this additional cog. If the server has a f
77.
by mattrobenolt 10y ago
I have tons of experience with Varnish and a long history there. Varnish is really bad for this since it's memory only. I wanted to use 750GB of disk space, not the 32GB of RAM we had. 750GB of RAM is significantly more expensive. And
78.
by mattrobenolt 10y ago
Definitely. If our dataset was absolutely massive and we couldn't hold a reasonable amount on disk, it'd make more sense. Fortunately, we are getting a 90+% hit ratio out of a very small amount of space relative to the size of our
79.
by mattrobenolt 10y ago
It directs traffic conditionally based on if our cache server is up or not, falling back to S3 directly. In theory, we could use nginx for this, but nginx isn't as good as a generic load balancer since it doesn't have as rich of i
80.
by mattrobenolt 10y ago
Or we can do what we did and save money instead and gain the guarantees. Plus achieve our original goal of performance by bringing the bytes inside our datacenter instead of going out through public internet.
81.
by mattrobenolt 10y ago
S3 bucket replication is a bit flawed, and doesn't buy us anything on top of what we implemented. And would cost double for storage. Plus more complex application logic which I wanted to avoid. Tools like haproxy are pretty good at thi
82.
by mattrobenolt 10y ago
In our case, we don't need to ever revalidate. We store things forever since our file blobs are immutable.
83.
by mattrobenolt 10y ago
That's DataDog. :)
84.
by mattrobenolt 10y ago
As mentioned, we are saving a lot in S3 transfer cost. :)
85.
by mattrobenolt 10y ago
This would unfortunately be slower, since it's not within our datacenter and on our network, and significantly more expensive. Probably at least 100x more expensive for the bandwidth than S3 by itself.
86.
by mattrobenolt 10y ago
I guess this is just phrased a bit oddly. It's just a factor in helping us scale by requiring us to use our hardware more efficiently. Not sure I'd say it's horizontally or vertically. We've just made each unit of work c
87.
by mattrobenolt 10y ago
Nope. That's me. :)
88.
by mattrobenolt 10y ago
What's your PayPal address?
89.
Debuggable JavaScript in Production with Source Maps(blog.getsentry.com)
5 points by mattrobenolt 11y ago|0 comments
90.
Sentry emails get a facelift(blog.getsentry.com)
2 points by mattrobenolt 11y ago|0 comments

More ›