shayonj

19 ms·

91.
Faster PostgreSQL partition look ups with client-side integer hashing in Ruby(github.com)
5 points by shayonj 1y ago|0 comments
92.
by shayonj 1y ago
Redis, PG, Clickhouse sounds like a good combo to scale workloads that require OLTP and OLAP workloads, at scale (?).
93.
by shayonj 1y ago
Likewise - that’s why I mentioned that orchestration backed by PG and treating sidekiq + Redis message bus goes a long way. You get the durability, and the ability to re-enqueue jobs in case of availability issues with Redis. Sidekiq’s reli
94.
by shayonj 1y ago
very nice! cool project
95.
by shayonj 1y ago
IIUC - Does this mean if I am making a network call, or performing some expensive task from within the job, the transaction against the database is held open for the duration of the job?
96.
by shayonj 1y ago
very interesting and TIL about the project, thanks for sharing. What DB is River Queue for 46k jobs/sec ( https://riverqueue.com/docs/benchmarks ) UPDATE: I see its PG - https://riverqueue.com/docs&#
97.
by shayonj 1y ago
I like Solid Queue and the direction things are heading, but its hard to overlook the performance. A system that does tens to hundreds of thousands of jobs/s on Sidekiq + Redis, will now get bottlenecked by transactional performance wi
98.
by shayonj 1y ago
Very nicely written post! I'd love to start running these in production on NVMe and hope its something major cloud providers start to offer ASAP. The performance gains are _extremely_ attractive
99.
Another look into PostgreSQL CTE materialization and non-idempotent subqueries(shayon.dev)
3 points by shayonj 1y ago|0 comments
100.
by shayonj 1y ago
Also, I believe using an `ORDER BY` is only as good as `=` where you are giving the planner a hint on the route to choose, but it doesn't necessarily solve the issue like using `AS MATERIALIZED` does. The issue being - it should only d
101.
by shayonj 1y ago
That is correct and our curiosities are very much aligned here. I wrote this post in passing on a bus, so it def lacks some critical detail :D. I can very much solve this with a AS MATERIALIZED like you pointed, and as also mentioned in -
102.
by shayonj 1y ago
That's correct, if you want to keep using CTE, the gurantee can be achieved via something like this WITH deleted_tasks AS MATERIALIZED ( SELECT id FROM task_queue WHERE queue_group_id = 5 AND status = 'pending' L
103.
A PostgreSQL Planner Gotcha with CTEs Delete and Limit(shayon.dev)
2 points by shayonj 1y ago|0 comments
104.
pg_hash_func Ruby gem: Calculate PostgreSQL partition index without querying(github.com)
3 points by shayonj 1y ago|0 comments
105.
by shayonj 2y ago
There is something to be said about using SET LOCAL SESSION maybe. Esp. when using pgbouncer in between rails app and PostgreSQL and using session var
106.
by shayonj 2y ago
exactly! those are the places I have found it to be super useful. Glad you found it useful
107.
by shayonj 2y ago
+1
108.
by shayonj 2y ago
Yes, i touched a bit on it towards the end. If you turn it off, Aurora won't wait for 4 out of 6 nodes to acknowledge the commit ( https://aws.amazon.com/blogs/database/amazon-aurora-postgres... )
109.
by shayonj 2y ago
+1 - I have found its better to instead use it on paths that you know are idempotent and recoverable. Liking expiring data on a cron and so on.
110.
Selective async commits in PostgreSQL – balancing durability and performance(shayon.dev)
1 points by shayonj 2y ago|0 comments
111.
Selective async commits in PostgreSQL – balancing durability and performance(shayon.dev)
109 points by shayonj 2y ago|37 comments
112.
by shayonj 2y ago
that's true, you can get very far with COPY
113.
by shayonj 2y ago
thank you for the response and stats! very cool
114.
by shayonj 2y ago
> I insert about 150,000 rows a second That's amazing - I would love to know if you have done careful data modeling, indexing, etc that allows you to get to this and what kind of data is being insert ed?
115.
Challenging AI generated code from first principles(shayon.dev)
1 points by shayonj 2y ago|0 comments
116.
Challenging AI generated code from first principles(shayon.dev)
4 points by shayonj 2y ago|1 comments
117.
by shayonj 2y ago
oops! (:p) fixed, thanks!
118.
by shayonj 2y ago
hey hey! thanks for reading and posting to HN - (post author)
119.
by shayonj 2y ago
You do make a good point - I think there is something to be said about making things more easier for application/product engineers such that these things are sort of "auto pilot"
120.
by shayonj 2y ago
Starting each test in its own transaction and then rolling back at the end, and ensuring no 3rd party network calls in application logic in rails and Postgres apps have worked with a lot of success in my experience.

More ›