3 ms·

This quote stuck out to me: >Today, the Python backends to the web services communicate directly with PostgreSQL via SQLAlchemy, but it is my intention to buil

This quote stuck out to me: >Today, the Python backends to the web services communicate directly with PostgreSQL via SQLAlchemy, but it is my intention to build out experimental replacement backends which are routed through GraphQL instead. This way, the much more performant and robust GraphQL backends become the single source of truth for all information in SourceHut. I wonder how adding a layer of indirection can significantly improve performance. If I were writing this service, I would go all in on GraphQL and have the frontend talk to the GraphQL services directly rather than routing the requests from Python through to a GraphQL service then presumably to PostgreSQL. Perhaps I am missing something. Indeed good luck to Drew here.

deleted 6y ago
[deleted]
ianamartin 6y ago
That quote is sort of exactly what's conceptually wrong with what's goin on in my opinion. Yes, I know, armchair quarterback and I'm not the one out there building stuff like this for free, etc., etc. But claiming some nebulous backend that's more performant and robust than Postgres is like, WTF? Are you using an actual GraphDB like Neo4J? Are you putting a graph frontend on Postgres like PostGraphQL? None of the post really makes any sense because GraphQL is a Query Language, not a data store. What are the CAP theorem tradeoffs in the new backend? What does more robust mean? What does more performant mean? This is a source control app. Those tradeoffs are meaningful. There seems to be a lot of conflation between API design and data store and core programming tools all mixed into a big post that mostly sounds to me like, "I don't get how to make this (extremely popular and well-known platform that drives many websites 10000x my size) work well, so I'm trying something different that sounds cool." Which, again, the author has always said this is an experiment, and that's cool. But the conceptual confusion in the post makes me think that moving away from boring tools and trying new tools is not going to end up going well. But this is a source control app, and it's hopefully backed up somewhere besides sourcehut so it should be fine if he needs to backtrack.
- zapf 6y ago
  Well said! Couldn't agree more. The GraphQL confusion is one more bullshit in the world of web dev.
  - ianamartin 6y ago
    I'm not totally against GraphQL in general. As an alternative to REST it can sometimes make sense. And let's be real, most REST APIs are absolute garbage. Anything would be better than a bad REST API. And if, in fact, you are storing a graph in a graph database, the QL makes a bit of sense. But nothing in the post makes any sense out of any of that. It's just Python bad; REST bad; I read too much hacker news, and I feel like it's time for a change. Like, when I complain about other people's REST APIs, that's out of my control. This guy is saying that his API is garbage, and instead of fixing it to make it better, he's just going to redo everything with a worse result. I don't get it.
- vertex-four 6y ago
  The goal here is to generate a typed API across a bunch of microservices (written in some typed language suited for the job) that are consumed by a Python frontend. The current design is a pile of vertically-integrated monoliths that touch the disk, database, perform backend operations and rendering all in one process. Python's single-threaded design makes it difficult to be responsive to small queries quickly while simultaneously serving large, time-consuming queries (i.e. git operations). You can get around this using worker queues to separate interpreter processes and an async design, or otherwise splitting your workload up... or you can use a language where "have a threadpool" is actually a properly supported concept, and an architecture where sharding git/email/etc backends is feasible.
  - pknopf 6y ago
    > The goal here is to generate a typed API across a bunch of microservices You are describing gRPC.
    - vertex-four 6y ago
ddevault 6y ago
It will be performing this indirection over localhost. I don't see it as much different from the indirection of SQLAlchemy. Yes, there is the question of parsing the GQL and so on, but I think that they're surmountable and fit well within our desired performance budget. Performance is also just one of many reasons why this approach is being considered.

deleted 6y ago

[deleted]

ianamartin 6y ago

That quote is sort of exactly what's conceptually wrong with what's goin on in my opinion. Yes, I know, armchair quarterback and I'm not the one out there building stuff like this for free, etc., etc. But claiming some nebulous backend that's more performant and robust than Postgres is like, WTF? Are you using an actual GraphDB like Neo4J? Are you putting a graph frontend on Postgres like PostGraphQL? None of the post really makes any sense because GraphQL is a Query Language, not a data store. What are the CAP theorem tradeoffs in the new backend? What does more robust mean? What does more performant mean? This is a source control app. Those tradeoffs are meaningful. There seems to be a lot of conflation between API design and data store and core programming tools all mixed into a big post that mostly sounds to me like, "I don't get how to make this (extremely popular and well-known platform that drives many websites 10000x my size) work well, so I'm trying something different that sounds cool." Which, again, the author has always said this is an experiment, and that's cool. But the conceptual confusion in the post makes me think that moving away from boring tools and trying new tools is not going to end up going well. But this is a source control app, and it's hopefully backed up somewhere besides sourcehut so it should be fine if he needs to backtrack.

zapf 6y ago
Well said! Couldn't agree more. The GraphQL confusion is one more bullshit in the world of web dev.
- ianamartin 6y ago
  I'm not totally against GraphQL in general. As an alternative to REST it can sometimes make sense. And let's be real, most REST APIs are absolute garbage. Anything would be better than a bad REST API. And if, in fact, you are storing a graph in a graph database, the QL makes a bit of sense. But nothing in the post makes any sense out of any of that. It's just Python bad; REST bad; I read too much hacker news, and I feel like it's time for a change. Like, when I complain about other people's REST APIs, that's out of my control. This guy is saying that his API is garbage, and instead of fixing it to make it better, he's just going to redo everything with a worse result. I don't get it.
vertex-four 6y ago
The goal here is to generate a typed API across a bunch of microservices (written in some typed language suited for the job) that are consumed by a Python frontend. The current design is a pile of vertically-integrated monoliths that touch the disk, database, perform backend operations and rendering all in one process. Python's single-threaded design makes it difficult to be responsive to small queries quickly while simultaneously serving large, time-consuming queries (i.e. git operations). You can get around this using worker queues to separate interpreter processes and an async design, or otherwise splitting your workload up... or you can use a language where "have a threadpool" is actually a properly supported concept, and an architecture where sharding git/email/etc backends is feasible.
- pknopf 6y ago
  > The goal here is to generate a typed API across a bunch of microservices You are describing gRPC.
  - vertex-four 6y ago

ddevault 6y ago

It will be performing this indirection over localhost. I don't see it as much different from the indirection of SQLAlchemy. Yes, there is the question of parsing the GQL and so on, but I think that they're surmountable and fit well within our desired performance budget. Performance is also just one of many reasons why this approach is being considered.