shad42

5 ms·

31.
We decreased our LLM costs by switching to Opus(mendral.com)
2 points by shad42 7mo ago|0 comments
32.
by shad42 7mo ago
Mendral co-founder here. What happens at PostHog is not uncommon. While building Mendral, we talked to hundreds of team and they all have a similar situation. Initially they come to us to make their CI pipelines faster. But as the agent div
33.
What CI looks like at a 100-person team (PostHog)(mendral.com)
56 points by shad42 7mo ago|30 comments
34.
We upgraded to a frontier model and our costs went down(mendral.com)
1 points by shad42 7mo ago|0 comments
35.
by shad42 8mo ago
In some ways: we use their product and they use Mendral
36.
by shad42 8mo ago
100% and LLMs have tons of related training data
37.
by shad42 8mo ago
very interesting, curious if there is any downside to running this at scale (compute?)
38.
by shad42 8mo ago
We did not want to make the post engineering-focused, but we have 18 companies in production today (we wrote about PostHog in the blog). At some point we should post some case studies. The metric we track for usefulness is our monthly reven
39.
by shad42 8mo ago
Mendral is replacing a human Platform Engineer. It debugs the CI logs, look at the commit associated, look at the implementation of the tests, etc... It then proposes fixes and takes care of opening a PR. We wrote about how this works for P
40.
by shad42 8mo ago
There is a cost associated with each investigation (that the Mendral agent is doing). And we spend time tuning the orchestration between agents. Yes expensive but we're making money on top of what it costs us. So far we were able to ta
41.
by shad42 8mo ago
I agree, we automated in the Mendral agent what is time consuming for human (like debugging a flaky test), but it will need permission to confirm the remediation and open a PR. But it's night and day to fix your CI when someone (in thi
42.
by shad42 8mo ago
LLMs are better now at pulling the context (as opposed to feeding everything you can inside the prompt). So you can expose enough query primitives to the LLM so it's able to filter out the noise. I don't think implementing filteri
43.
by shad42 8mo ago
I am sure you heard before: there are only two hard things in CS: cache invalidation and naming things. In the history of this company, I can honestly say that this SQL/LLM thing wasn't the hardest :)
44.
by shad42 8mo ago
From own experience it's true, and I think it's due to the amount of SQL content (docs, best practices, code) that you can find online, which is now in all LLM's corpus data. Same applies when picking a programming language n
45.
by shad42 8mo ago
Yeah it sounds very familiar with what we went through while building this agent. We're focused on CI logs for now because we wanted something that works really well for things like flaky tests, but planning to expand the context to i
46.
by shad42 8mo ago
We started writing very recently: https://www.mendral.com/blog - there is a another post we made yesterday about the overall architecture. And we have a long list of things we're planning to write about in more details
47.
by shad42 8mo ago
What we learned while building this is every token matters in the context, we spend lot of time watching logs of agent sessions, changing the tool params, errors returned by tools, agent prompts, etc... We noticed for example the importance
48.
by shad42 8mo ago
Mendral co-founder here, we built this infra to have our agent detect CI issues like flaky tests and fix them. Observing logs are useful to detect anomalies but we also use those to confirm a fix after the agent opens a PR (we have long cod
49.
We gave terabytes of CI logs to an LLM(mendral.com)
233 points by shad42 8mo ago|111 comments
50.
Anatomy of a Production AI Agent(mendral.com)
2 points by shad42 8mo ago|0 comments
51.
What CI Looks Like at a 100-Person Team(mendral.com)
3 points by shad42 8mo ago|0 comments
52.
What CI looks like at a 100-person team(mendral.com)
4 points by shad42 8mo ago|0 comments
53.
by shad42 1y ago
LLMs are powerful assistants—as long as the user keeps a firm mental model of the problem. That’s why, for now, they complement software engineers rather than replace them (at least today). When you already know exactly what needs to be bui
54.
Evals as Code: CI for LLMs with Dagger(dagger.io)
2 points by shad42 1y ago|1 comments
55.
by shad42 1y ago
This goes over how we built LLM support into Dagger, and more importantly what it took to implementing testing for it.
56.
by shad42 4y ago
Replaced Desktop with `colima` as well few months ago. I've been using it daily since then. I did not have any issue, sometimes I just delete / start a instance to upgrade the docker version, it only takes few minutes. I like the
57.
by shad42 6y ago
There is always a container in the end. But do you actually provide a Dockerfile along with your app code, or do you let buildpack create the container for you (ie. no Dockerfile in your source code)?
58.
by shad42 6y ago
Given your Dokku experience, do you mostly interact with Buildpacks or Containers? And why?
59.
by shad42 10y ago
The complexity is the reason why I came back to gevent, especially since it now supports Python 3... Gevent monkey patching isn't perfect but it works and gets you closer to how an event loop should be used with standard libs IMO, clos
60.
by shad42 10y ago
Did you have to upgrade clusters for your clients yet? Is there a way upgrade clusters to a newer version of k8s safely?

More ›