lawrjone

24 ms·

241.
by lawrjone 6y ago
Not who you are replying to, but I'd imagine your point: > If you so value your company information, maybe you should start reading what you sign. Is exactly what people are up in arms about. Even reading the EULA, you may not expec
242.
by lawrjone 6y ago
Such an awesome company, with great people. Probably one of the most exciting start-ups in Europe right now! Only negative is they keep taking my great colleagues ;)
243.
by lawrjone 6y ago
To be fair, network latency across all the providers is reasonably similar. The network throughput was not even close though, with GCP winning by a huge margin. I'd say both statements are fair and not mutually exclusive.
244.
by lawrjone 6y ago
Author here, and happy to discuss anything about the content. It might be useful to know this work was driven by the need to provide self-service tooling to developers, removing the SRE team from the loop when it came to bootstrapping new s
245.
Deploying Software at GoCardless: Open-Sourcing Our “Getting Started” Tutorial(medium.com)
2 points by lawrjone 6y ago|1 comments
246.
by lawrjone 6y ago
Not the author, but I am an engineer at GoCardless. This article is about how the Product Development group at GoCardless works. From team setup, how we make decisions, how we promote, etc. I always like insights into other companies like t
247.
An overview of engineering at GoCardless, a fintech scale-up(medium.com)
3 points by lawrjone 6y ago|1 comments
248.
by lawrjone 6y ago
Postgres change-capture device that supports high-throughput and low-latency capture to a variety of sinks (at first release, just Google BigQuery): https://github.com/lawrencejones/pgsink I know there's debezium
249.
by lawrjone 6y ago
There are a load of ways NY might have contributed to this, but it may not be a direct cause. What's more likely, Slack forgetting to scale their deployment back up after too much mulled wine, or a number of people on holiday meaning a
250.
by lawrjone 6y ago
Having been in this situation before, with a totally-down-and-not-coming-back-up outage of a payments system, I really feel for their incident response team. I'll take this moment to remind everyone of their human tendency to read mean
251.
by lawrjone 6y ago
Author here! A few others have pointed this out, but to restate: in our situation, compression was costing us almost nothing. 60TB of logs was 60 days worth of retention, so 1TB a day. That means we process about 11MB/s on average, pea
252.
by lawrjone 6y ago
I'm not quite sure what you mean by message framing. If you mean marking messages as having been compressed, then absolutely yet. The Pub/Sub messages were tagged with `compress=true|false` so we could update downstream and upstre
253.
by lawrjone 6y ago
You've nailed this! This logging system was for all https://gocardless.com/ systems. We're a B2C company which means we have different economies of scale than many scale-ups of our size, but you were close with yo
254.
by lawrjone 6y ago
Author here! These 10B log lines are from the last 60 days of activity from https://gocardless.com/ systems. It includes: - System logs, such as our Kubernetes VM host logs, or our Chef Postgres machines - Application logs
255.
by lawrjone 6y ago
Hey all- I'm the author and just noticed this post. Thanks for the repost! If you're interested, there was a nice discussion in /r/devops about this the other day: https://www.reddit.com/r/devops&#x2
256.
by lawrjone 6y ago
Hey! I didn't implement the fluentd piece, but I can point you at the PR that did just that for our fork of the gcloud-pubsub plugin: https://github.com/gocardless/fluent-plugin-gcloud-pubsub-cu... One comment I c
257.
by lawrjone 6y ago
Author here, happy to answer any questions people may have!
258.
We compress Pub/Sub messages and more, saving a load of money(blog.lawrencejones.dev)
9 points by lawrjone 6y ago|4 comments
259.
by lawrjone 6y ago
I'm not sure I see that as a fair comparison. I think it's best to use the same durations for this, as an entire day changes the level of impact. <1hr outages have a fraction of the impact that an entire days outage might have.
260.
by lawrjone 6y ago
Google do a very good job of defining their SLAs. In answer to your question, they'll accept evidence from your own monitoring system when you claim on the SLA. They pair that up to their own knowledge about how the system was performi
261.
by lawrjone 6y ago
Would they have less impact? Or would it have the same impact, just distributed across many more outages? You can rely on Google outages being very few and far between, and recovering pretty fast. For the benefits you get from such a connec
262.
by lawrjone 6y ago
Not the author, but work with Ben (post author) on an SRE team at GoCardless. Happy to answer any questions here. As a side note, we're actively searching for people with an interest in these technologies- message me if this sounds lik
263.
Service outage due to Vault certificate expiry, leading to Kubernetes pods dying(gocardless.com)
11 points by lawrjone 6y ago|1 comments
264.
by lawrjone 6y ago
pg_basebackup defaults to streaming WAL from whatever server it is taking a copy from, as backups are otherwise inconsistent. There's no need for the source database to be paused while doing it, and even if your backup takes several ho
265.
by lawrjone 6y ago
Author of the post here, in case anyone has any questions!
266.
Connected data: Using BigQuery to analyse user behaviour in response to webhooks(blog.lawrencejones.dev)
4 points by lawrjone 6y ago|1 comments
267.
by lawrjone 6y ago
Yep, as you point out, the idea of Elasticsearch having a uniform type for each field can get problematic. That can be fixed by: - logging to different indexes - preprocessing your logs so the keys have their own schema prefix, as you menti
268.
by lawrjone 6y ago
We run about 10 Kubernetes clusters, and each cluster comes with our company's 'kernel'. That kernel includes an HA Prometheus deployment of two replicas and all the monitoring components that to with it. The 2TB of data is w
269.
by lawrjone 6y ago
We run Prometheus with 60d retention (2TB), and an Elasticsearch cluster for logs (also 60d, about 30TB cluster) with Kibana on top. Prometheus is by far the cheapest, both for infrastructure spend and for human cost. The only time we spend
270.
An incident response starter-pack: how do you handle production outages?(blog.lawrencejones.dev)
5 points by lawrjone 6y ago|0 comments

More ›