11 ms·

> The technical fix was embarrassingly simple: stop pushing to main every ten minutes. Wait, you push straight to main? > We added a rule — batch related chan

by yokuze 6mo ago

> The technical fix was embarrassingly simple: stop pushing to main every ten minutes. Wait, you push straight to main? > We added a rule — batch related changes, avoid rapid-fire pushes. It's in our CLAUDE.md (the governance file that all our AI agents follow): > Avoid rapid-fire pushes to main — 11 pushes in 2h caused overlapping Kamal deploys with concurrent SQLite access. Wait, you let _Claude_ push your e-commerce code straight to main which immediately results in a production deploy?

crabmusket 6mo ago
Patient: doctor, my app loses data when I deploy twice during a 10 minute interval! Doctor: simply do not do that
- pavel_lishin 6mo ago
  Doctor: solution is simple, stop letting that stupid clown Pagliacci define how you do your work! Patient: but doctor,
  - pjc50 6mo ago
    pAIgliacci: as a large language model, I am unable to experience live comedy.
- rcakebread 6mo ago
  Bob Newhart did it best https://www.youtube.com/watch?v=LhQGzeiYS_Q https://www.youtube.com/watch?v=LhQGzeiYS_Q
bombcar 6mo ago
Hey, Apple still takes their store down during product launches!
- pstuart 6mo ago
  I assumed that it was to ensure that the announced products were revealed in a controlled manner rather than because they aren't able to do updates to their product listings as a regular thing.
  - bombcar 6mo ago
    My reading of the tea leaves is it started out as the latter and continues as the former as part of the “mystique”.
tensegrist 6mo ago
i hate to be so blunt but look around the site and then tell me you're surprised
xnorswap 6mo ago
I'm fairly confident they let it write the blog post too.
- simonw 6mo ago
  "Not as a proof of concept. Not for a side project with three users. A real store" - suggestion for human writers, don't use "not X, not Y" - it carries that LLM smell whether or not you used an LLM.
  - xnorswap 6mo ago
    And that's just the opening paragraph, the full text is rounded off with: "The constraint is real: one server, and careful deploy pacing." Another strong LLM smell, "The <X> is real", nicely bookends an obviously generated blog-post.
- These335 6mo ago
  You're absolutely right, this was an AI post
  - yokuze 6mo ago
    I see what you did there XD
chasil 6mo ago
This is the actual problem: "Kamal runs blue-green deploys — it starts a new container, health-checks it, then stops the old one. During the switchover, both containers are running. Both mount ultrathink_storage. Both have the SQLite files open." WAL mode requires shared access to System V IPC mapped memory. This is unlikely to work across containers. In case anybody needs a refresher: https://en.wikipedia.org/wiki/Shared_memory https://en.wikipedia.org/wiki/Shared_memory https://en.wikipedia.org/wiki/CB_UNIX https://en.wikipedia.org/wiki/CB_UNIX https://www.ibm.com/docs/en/aix/7.1.0?topic=operations-system-v-interprocess-communication-ipc https://www.ibm.com/docs/en/aix/7.1.0?topic=operations-syste...
- Retr0id 6mo ago
  > This is unlikely to work across containers. Why not?
- simonw 6mo ago
  Thanks for this, the anecdote with the lost data was very concerning to me. I think you're exactly right about the WAL shared memory not crossing the container boundary. EDIT: It looks like WAL works fine across Docker boundaries, see https://news.ycombinator.com/item?id=47637353#47677163 https://news.ycombinator.com/item?id=47637353#47677163 I don't know much about Kamal but I'd look into ways of "pausing" traffic during a deploy - the trick where a proxy pretends that a request is taking another second to finish when it's actually held in the proxy while the two containers switch over. From https://kamal-deploy.org/docs/upgrading/proxy-changes/ https://kamal-deploy.org/docs/upgrading/proxy-changes/ it looks like Kamal 2's new proxy doesn't have this yet, they list "Pausing requests" as "coming soon".
  - chasil 6mo ago
    You might consider taking the database(s) out of WAL mode during a migration. That would eliminate the need for shared memory.
  - Retr0id 6mo ago
    > I think you're exactly right about the WAL shared memory not crossing the container boundary. I don't, fwiw (so long as all containers are bind mounting the same underlying fs).
littlestymaar 6mo ago
> Wait, you let _Claude_ push your e-commerce code straight to main which immediately results in a production deploy? Yikes. Thank you I'm not going to read “Lessons learned” by someone this careless.
- 66yatman 6mo ago
  The issue wasn’t done by the ai but their lack of architectural knowledge
  - okkdev 6mo ago
    Goes hand in hand
- whateveracct 6mo ago
  stupid is as stupid does
burnt-resistor 6mo ago
I suspect they don't wear helmets or seatbelts either. Sigh. The "I'm so proud and ignorant of unnecessarily risky behaviors" meme is tiring. The Meta dev model of diff reviews merge into main (rebase style) after automated tests run is pretty good. Also, staging and canary, gradual, exponential prod deployment/rollback approaches help derisk change too. Finally, have real, tested backups and restore processes (not replicated copies) and ability to rollback.

by yokuze 6mo ago

crabmusket 6mo ago

Patient: doctor, my app loses data when I deploy twice during a 10 minute interval! Doctor: simply do not do that

pavel_lishin 6mo ago
Doctor: solution is simple, stop letting that stupid clown Pagliacci define how you do your work! Patient: but doctor,
- pjc50 6mo ago
  pAIgliacci: as a large language model, I am unable to experience live comedy.
rcakebread 6mo ago
Bob Newhart did it best https://www.youtube.com/watch?v=LhQGzeiYS_Q https://www.youtube.com/watch?v=LhQGzeiYS_Q

bombcar 6mo ago

Hey, Apple still takes their store down during product launches!

pstuart 6mo ago
I assumed that it was to ensure that the announced products were revealed in a controlled manner rather than because they aren't able to do updates to their product listings as a regular thing.
- bombcar 6mo ago
  My reading of the tea leaves is it started out as the latter and continues as the former as part of the “mystique”.

tensegrist 6mo ago

i hate to be so blunt but look around the site and then tell me you're surprised

xnorswap 6mo ago

I'm fairly confident they let it write the blog post too.

simonw 6mo ago
"Not as a proof of concept. Not for a side project with three users. A real store" - suggestion for human writers, don't use "not X, not Y" - it carries that LLM smell whether or not you used an LLM.
- xnorswap 6mo ago
  And that's just the opening paragraph, the full text is rounded off with: "The constraint is real: one server, and careful deploy pacing." Another strong LLM smell, "The <X> is real", nicely bookends an obviously generated blog-post.
These335 6mo ago
You're absolutely right, this was an AI post
- yokuze 6mo ago
  I see what you did there XD

chasil 6mo ago

This is the actual problem: "Kamal runs blue-green deploys — it starts a new container, health-checks it, then stops the old one. During the switchover, both containers are running. Both mount ultrathink_storage. Both have the SQLite files open." WAL mode requires shared access to System V IPC mapped memory. This is unlikely to work across containers. In case anybody needs a refresher: https://en.wikipedia.org/wiki/Shared_memory https://en.wikipedia.org/wiki/Shared_memory https://en.wikipedia.org/wiki/CB_UNIX https://en.wikipedia.org/wiki/CB_UNIX https://www.ibm.com/docs/en/aix/7.1.0?topic=operations-system-v-interprocess-communication-ipc https://www.ibm.com/docs/en/aix/7.1.0?topic=operations-syste...

Retr0id 6mo ago
> This is unlikely to work across containers. Why not?
simonw 6mo ago
Thanks for this, the anecdote with the lost data was very concerning to me. I think you're exactly right about the WAL shared memory not crossing the container boundary. EDIT: It looks like WAL works fine across Docker boundaries, see https://news.ycombinator.com/item?id=47637353#47677163 https://news.ycombinator.com/item?id=47637353#47677163 I don't know much about Kamal but I'd look into ways of "pausing" traffic during a deploy - the trick where a proxy pretends that a request is taking another second to finish when it's actually held in the proxy while the two containers switch over. From https://kamal-deploy.org/docs/upgrading/proxy-changes/ https://kamal-deploy.org/docs/upgrading/proxy-changes/ it looks like Kamal 2's new proxy doesn't have this yet, they list "Pausing requests" as "coming soon".
- chasil 6mo ago
  You might consider taking the database(s) out of WAL mode during a migration. That would eliminate the need for shared memory.
- Retr0id 6mo ago
  > I think you're exactly right about the WAL shared memory not crossing the container boundary. I don't, fwiw (so long as all containers are bind mounting the same underlying fs).

littlestymaar 6mo ago

> Wait, you let _Claude_ push your e-commerce code straight to main which immediately results in a production deploy? Yikes. Thank you I'm not going to read “Lessons learned” by someone this careless.

66yatman 6mo ago
The issue wasn’t done by the ai but their lack of architectural knowledge
- okkdev 6mo ago
  Goes hand in hand
whateveracct 6mo ago
stupid is as stupid does

burnt-resistor 6mo ago

I suspect they don't wear helmets or seatbelts either. Sigh. The "I'm so proud and ignorant of unnecessarily risky behaviors" meme is tiring. The Meta dev model of diff reviews merge into main (rebase style) after automated tests run is pretty good. Also, staging and canary, gradual, exponential prod deployment/rollback approaches help derisk change too. Finally, have real, tested backups and restore processes (not replicated copies) and ability to rollback.