8 ms·

Well, that's too bad. I always thought this was a cool project. But if you can't dev your way into decent performance for a small alpha project using python/fla

Well, that's too bad. I always thought this was a cool project. But if you can't dev your way into decent performance for a small alpha project using python/flask/sql, I don't think your tools are the problem. And I guarantee that a graphql isn't the solution. So, I mean, good luck.

ghoshbishakh 6y ago
what do you feel the problem is?
- ianamartin 6y ago
  There are lots of production sites that serve 10,000x as much traffic as sourcehut that are built on Python/flask/sqlalchemy serving RESTful APIs. If you can't make that combination work well, there's another place to look for problems besides your tool kit. You might need to ask yourself if you really understand the tools you're trying to use. But like I said, this has always been a very cool project. My "good luck" was meant more as actual good luck than a Morgan Freeman You're-trying-to-blackmail-batman kind of good luck.
  - deleted 6y ago
    [deleted]
  - xrisk 6y ago
    What are some production websites that run Python stacks? I’m curious. On a tangential note: if anybody has blog posts on scaling Flask/SQLAlchemy or Django stacks I would appreciate it.
    - searchableguy 6y ago
      Dropbox is a notable one born here.
      - procinct 6y ago
        Especially considering they employed the BDFL.
        square_usual 6y ago
        No longer the BDFL, unfortunately.
        iso-8859-1 6y ago
        The BDFL is a position that you have for life. It is embedded in the definition. So it is impossible to shed the title.
        dragonwriter 6y ago
        Like most “for life” positions (the papacy, US federal judges, the British monarchy, etc.), it can be shed though it is not regularly expected to be lost other than at the holder's decision.
    - alexchamberlain 6y ago
      Don't Instagram and YouTube still have large Python code bases?
    - ianamartin 6y ago
      Reddit is one you might have heard of. pypi.org is another that's familiar. You know, every time you type `pip install x` yeah, that's pypi. Although I think those are both powered mainly by Pyramid rather than flask. Still, same concept. As others mention, large parts of google and youtube are still python. Dropbox was so invested in python that they employed Guido van Rossum for a while. Instagram, a lot of Yahoo! back when they were a thing, Spotify, Quora, Pinterest, Hipmunk, Disqus, and this really obscure satire site called The Onion that totally never gets any traffic at all. All of them powered by python at their core, many of them Django, some Pyramid, and some Flask. Yes, getting that big does require big teams. Becoming one of the top 100 or so sites on the internet always requires some special sauce as well as dedicated teams. But most of these companies started with Python and a framework and got to massive web scale along the way and never changed the core platform because there really wasn't a need. Handling scale isn't about your core language or framework. It's about dozens of other things that you can offload to other things if you're smart. But let's be real: sourcehut isn't close to any of that level of traffic. My negativity on this isn't about stanning a particular language. I'm an agnostic in multiple ways. I'll use whatever tool seems like the best fit. I'm down on this because the explanation is tool-blaming, murky, unclear, and doesn't provide a lot of the detail I would want to have if I were depending on this service. On the other hand, the guy has always said this is an alpha project and you should expect major changes. That's all fine. It's just weird to me to see a "why I'm changing from X to Y" post that doesn't really explain anything other than "I might be bad at this."
      - kupaka 6y ago
        jfyi, The Onion hasn't run on Python for a few years now. In 2017, they migrated over to Gizmodo's Scala stack.
searchableguy 6y ago
I didn't read the post that way. I feel scaling is a small issue more so organization. Graphql does make sense for something like source hunt. Why not use type hints in python? Isn't that a good enough substitute? I wonder why go instead of rust if he wanted static typing, long term ease of maintanence and performance. Go's type system is not great especially for something like graphql. Gqlgen relies heavily on code generation. Last time I used it, I ran into so many issues. I ditched go together after several painful clashes with it that community always responded with: oh you don't need this. (yeah except they implemented all those parts in arguably worse ways and ditched community solutions in the next few years) One major benefit the GP fails to mention is that with graphql, it is easy to generate types for frontend. This makes your frontend far more sane. It's also way easier to test graphql since there are tools to automatically generate queries for performance testing unlike rest. There is no need to add something for docs or interactivity like swagger.
- CameronNemo 6y ago
  >One major benefit the GP fails to mention is that with graphql, it is easy to generate types for frontend. This makes your frontend far more sane. By GP (grandparent?) do you mean the article / blog post? Because if so I see no indication that Drew plans to adopt a SPA architecture -- he seems intent on continuing to use server side rendering with little javascript, which would make "frontend types" sort of irrelevant.
  - searchableguy 6y ago
    You don't need spa to take benefits of the generated types from graphql. If you use something like typescript on the backend (SSR app), the types will be stripped out in the end so it doesn't affect your bundle size.
    - CameronNemo 6y ago
      You must understand my confusion when your original comment explicitly states that frontend types can be generated, but your reply here seems to be talking about a javascript/typescript backend service.
CameronNemo 6y ago
This quote stuck out to me: >Today, the Python backends to the web services communicate directly with PostgreSQL via SQLAlchemy, but it is my intention to build out experimental replacement backends which are routed through GraphQL instead. This way, the much more performant and robust GraphQL backends become the single source of truth for all information in SourceHut. I wonder how adding a layer of indirection can significantly improve performance. If I were writing this service, I would go all in on GraphQL and have the frontend talk to the GraphQL services directly rather than routing the requests from Python through to a GraphQL service then presumably to PostgreSQL. Perhaps I am missing something. Indeed good luck to Drew here.
- deleted 6y ago
  [deleted]
- ianamartin 6y ago
  That quote is sort of exactly what's conceptually wrong with what's goin on in my opinion. Yes, I know, armchair quarterback and I'm not the one out there building stuff like this for free, etc., etc. But claiming some nebulous backend that's more performant and robust than Postgres is like, WTF? Are you using an actual GraphDB like Neo4J? Are you putting a graph frontend on Postgres like PostGraphQL? None of the post really makes any sense because GraphQL is a Query Language, not a data store. What are the CAP theorem tradeoffs in the new backend? What does more robust mean? What does more performant mean? This is a source control app. Those tradeoffs are meaningful. There seems to be a lot of conflation between API design and data store and core programming tools all mixed into a big post that mostly sounds to me like, "I don't get how to make this (extremely popular and well-known platform that drives many websites 10000x my size) work well, so I'm trying something different that sounds cool." Which, again, the author has always said this is an experiment, and that's cool. But the conceptual confusion in the post makes me think that moving away from boring tools and trying new tools is not going to end up going well. But this is a source control app, and it's hopefully backed up somewhere besides sourcehut so it should be fine if he needs to backtrack.
  - zapf 6y ago
ddevault 6y ago
Performance is a secondary concern. SourceHut already has the best performance in the industry, built on Python: https://forgeperf.org https://forgeperf.org But I think it could be even better, and this work will help. It will make it easier to write performant code without explicitly hand-optimizing everything. There are more important reasons to consider GraphQL than performance, which I cover in detail in TFA.
pknopf 6y ago
> And I guarantee that a graphql isn't the solution. I agree. Out of the pan and into the frier. He had a good idea though.

ghoshbishakh 6y ago

what do you feel the problem is?

ianamartin 6y ago
There are lots of production sites that serve 10,000x as much traffic as sourcehut that are built on Python/flask/sqlalchemy serving RESTful APIs. If you can't make that combination work well, there's another place to look for problems besides your tool kit. You might need to ask yourself if you really understand the tools you're trying to use. But like I said, this has always been a very cool project. My "good luck" was meant more as actual good luck than a Morgan Freeman You're-trying-to-blackmail-batman kind of good luck.
- deleted 6y ago
  [deleted]
- xrisk 6y ago
  What are some production websites that run Python stacks? I’m curious. On a tangential note: if anybody has blog posts on scaling Flask/SQLAlchemy or Django stacks I would appreciate it.
  - searchableguy 6y ago
    Dropbox is a notable one born here.
    - procinct 6y ago
      Especially considering they employed the BDFL.
      - square_usual 6y ago
        No longer the BDFL, unfortunately.
        iso-8859-1 6y ago
        The BDFL is a position that you have for life. It is embedded in the definition. So it is impossible to shed the title.
        dragonwriter 6y ago
        Like most “for life” positions (the papacy, US federal judges, the British monarchy, etc.), it can be shed though it is not regularly expected to be lost other than at the holder's decision.
  - alexchamberlain 6y ago
    Don't Instagram and YouTube still have large Python code bases?
  - ianamartin 6y ago
    Reddit is one you might have heard of. pypi.org is another that's familiar. You know, every time you type `pip install x` yeah, that's pypi. Although I think those are both powered mainly by Pyramid rather than flask. Still, same concept. As others mention, large parts of google and youtube are still python. Dropbox was so invested in python that they employed Guido van Rossum for a while. Instagram, a lot of Yahoo! back when they were a thing, Spotify, Quora, Pinterest, Hipmunk, Disqus, and this really obscure satire site called The Onion that totally never gets any traffic at all. All of them powered by python at their core, many of them Django, some Pyramid, and some Flask. Yes, getting that big does require big teams. Becoming one of the top 100 or so sites on the internet always requires some special sauce as well as dedicated teams. But most of these companies started with Python and a framework and got to massive web scale along the way and never changed the core platform because there really wasn't a need. Handling scale isn't about your core language or framework. It's about dozens of other things that you can offload to other things if you're smart. But let's be real: sourcehut isn't close to any of that level of traffic. My negativity on this isn't about stanning a particular language. I'm an agnostic in multiple ways. I'll use whatever tool seems like the best fit. I'm down on this because the explanation is tool-blaming, murky, unclear, and doesn't provide a lot of the detail I would want to have if I were depending on this service. On the other hand, the guy has always said this is an alpha project and you should expect major changes. That's all fine. It's just weird to me to see a "why I'm changing from X to Y" post that doesn't really explain anything other than "I might be bad at this."
    - kupaka 6y ago
      jfyi, The Onion hasn't run on Python for a few years now. In 2017, they migrated over to Gizmodo's Scala stack.

searchableguy 6y ago

I didn't read the post that way. I feel scaling is a small issue more so organization. Graphql does make sense for something like source hunt. Why not use type hints in python? Isn't that a good enough substitute? I wonder why go instead of rust if he wanted static typing, long term ease of maintanence and performance. Go's type system is not great especially for something like graphql. Gqlgen relies heavily on code generation. Last time I used it, I ran into so many issues. I ditched go together after several painful clashes with it that community always responded with: oh you don't need this. (yeah except they implemented all those parts in arguably worse ways and ditched community solutions in the next few years) One major benefit the GP fails to mention is that with graphql, it is easy to generate types for frontend. This makes your frontend far more sane. It's also way easier to test graphql since there are tools to automatically generate queries for performance testing unlike rest. There is no need to add something for docs or interactivity like swagger.

CameronNemo 6y ago
>One major benefit the GP fails to mention is that with graphql, it is easy to generate types for frontend. This makes your frontend far more sane. By GP (grandparent?) do you mean the article / blog post? Because if so I see no indication that Drew plans to adopt a SPA architecture -- he seems intent on continuing to use server side rendering with little javascript, which would make "frontend types" sort of irrelevant.
- searchableguy 6y ago
  You don't need spa to take benefits of the generated types from graphql. If you use something like typescript on the backend (SSR app), the types will be stripped out in the end so it doesn't affect your bundle size.
  - CameronNemo 6y ago
    You must understand my confusion when your original comment explicitly states that frontend types can be generated, but your reply here seems to be talking about a javascript/typescript backend service.

CameronNemo 6y ago

This quote stuck out to me: >Today, the Python backends to the web services communicate directly with PostgreSQL via SQLAlchemy, but it is my intention to build out experimental replacement backends which are routed through GraphQL instead. This way, the much more performant and robust GraphQL backends become the single source of truth for all information in SourceHut. I wonder how adding a layer of indirection can significantly improve performance. If I were writing this service, I would go all in on GraphQL and have the frontend talk to the GraphQL services directly rather than routing the requests from Python through to a GraphQL service then presumably to PostgreSQL. Perhaps I am missing something. Indeed good luck to Drew here.

deleted 6y ago
[deleted]
ianamartin 6y ago
That quote is sort of exactly what's conceptually wrong with what's goin on in my opinion. Yes, I know, armchair quarterback and I'm not the one out there building stuff like this for free, etc., etc. But claiming some nebulous backend that's more performant and robust than Postgres is like, WTF? Are you using an actual GraphDB like Neo4J? Are you putting a graph frontend on Postgres like PostGraphQL? None of the post really makes any sense because GraphQL is a Query Language, not a data store. What are the CAP theorem tradeoffs in the new backend? What does more robust mean? What does more performant mean? This is a source control app. Those tradeoffs are meaningful. There seems to be a lot of conflation between API design and data store and core programming tools all mixed into a big post that mostly sounds to me like, "I don't get how to make this (extremely popular and well-known platform that drives many websites 10000x my size) work well, so I'm trying something different that sounds cool." Which, again, the author has always said this is an experiment, and that's cool. But the conceptual confusion in the post makes me think that moving away from boring tools and trying new tools is not going to end up going well. But this is a source control app, and it's hopefully backed up somewhere besides sourcehut so it should be fine if he needs to backtrack.
- zapf 6y ago

ddevault 6y ago

Performance is a secondary concern. SourceHut already has the best performance in the industry, built on Python: https://forgeperf.org https://forgeperf.org But I think it could be even better, and this work will help. It will make it easier to write performant code without explicitly hand-optimizing everything. There are more important reasons to consider GraphQL than performance, which I cover in detail in TFA.

pknopf 6y ago

> And I guarantee that a graphql isn't the solution. I agree. Out of the pan and into the frier. He had a good idea though.