14 ms·

How Shopify implemented its secure authentication service

174 points by frankpf 7y ago

ivankolev 7y ago
So is Shopify the biggest fish still on the Ruby stack? Nice article detailing how they did an upgrade to openId connect to allow SSO on multiple shops within a client company.
- noodle 7y ago
  IIRC there are still many large companies using Ruby/Rails still, they've just also diversified their tech stacks (as larger companies tend to do). AFAIK the list includes: GitHub (MS has a few Rails-based acquisitions now), Airbnb, Groupon, Square, Cookpad, Kickstarter, Hulu, etc..
  - keithwhor 7y ago
    Pretty sure Stripe is a Ruby shop as well.
    - multiplegeorges 7y ago
      One of the biggest. They recently released Sorbet, a gradual type checker for Ruby.
      - ivankolev 7y ago
        I sometimes forget how vast and diverse the whole industry is, so many tech. stacks and ecosystems.
    - yellow_lead 7y ago
      Gitlab too it seems.
    - riffraff 7y ago
      IIRC it's ruby, but not rails
    - faintrain 7y ago
      JP Morgan Chase and McKesson both use Ruby with McKesson leading the pack in RoR LOC among the Fortune 50
  - itake 7y ago
    Insiders at Airbnb told me they are (almost) completely off Rails now. They moved their website to the jvm.
    - notyourwork 7y ago
      Seems at some scale this usually happens.
    - cutler 7y ago
      If all you want is the JVM JRuby and its Truffle variant are viable options.
  - ksec 7y ago
    I believe AirBnB and Groupon are off Rails ( And Ruby ). So the big one for Rails are Shopify, Github, Cookpad, Gitlab. For Ruby ( Not Rails ) That would be Stripe.
- uyuioi 7y ago
  They’re still using a rails setup for their core services. But, this core falls over with him RPM. It’s failed in nearly all flash sales it has had to handle. Rails is great. But commerce is heavy and I don’t believe Shopify can keep its existing core code base around much longer without a significant change to aid with performance.
- stickfigure 7y ago
  Funny you mention this. I just today had to implement a painful workaround for Shopify's insanely short timeout on product image uploads. On submitting an image url, you apparently get 4s to complete the whole transfer. I found hundreds of people complaining about this in the community forums, going back years. If you're dynamically generating images, or on a congested network, 4s is far too short. Since this is a simple config property, the only justification I can imagine is that they are trying to restrict the amount of time that their (single-threaded, memory-hungry) instances are occupied. Because of Ruby's poor resource management, a core part of their API is barely usable. I'm pretty disappointed.
  - uyuioi 7y ago
    I do a lot of developing with Shopify and it’s a mess. One of the worst development experiences of all time. Because they’re so monolithic focused. The API’s you can use are pretty sluggish and poorly documented. Rails is just not meant for heavy transactional load. And e-commerce needs async to handle what can be a huge load. Taobao is java or php and they handle load far greater without fault. Shopify is much better than Magento though.
    - kenrose 7y ago
      > One of the worst development experiences of all time... The API’s you can use are pretty sluggish and poorly documented. Can you elaborate? Shopify released their GraphQL Admin API in 2018, which has built in documentation, as well as an interactive IDE (Graphiql) that you can run on your shop. As long as you're not doing anything too crazy that exceeds the throttle (e.g., syncing 1000s of products), they're pretty good about listening to developer requests. https://help.shopify.com/en/api/graphql-admin-api https://help.shopify.com/en/api/graphql-admin-api > Rails is just not meant for heavy transactional load. Shopify has invested heavily in their sharding setup so that it can be handle high load and scale quickly. e.g., flash sales where baseline traffic will 3x in a matter of seconds. See the discussion of pods: https://engineering.shopify.com/blogs/engineering/e-commerce-at-scale-inside-shopifys-tech-stack https://engineering.shopify.com/blogs/engineering/e-commerce... > And e-commerce needs async to handle what can be a huge load. Long running processes are async. However, having commerce modeled by a transactional database that provides atomicity is a boon for simplicity. You don't want to deal with eventual consistency when updating inventory or placing orders. Note: Am ex-Shopify. Ran the API team.
- Thaxll 7y ago
  They're moving slow parts to Go.
  - uyuioi 7y ago
    Hopefully crystal Lang will make its way into these ruby heavy shops. They’ll get 100x performance without needing to really think in a whole different programming experience.
    - chrisseaton 7y ago
      > without needing to really think in a whole different programming experience But Crystal has entirely different semantics to Ruby. They look vaguely similar at a superficial level, but the semantics are not even remotely similar.
      - rurounijones 7y ago
        Maybe but depending on what you are doing they are minor considerations. I am porting a relatively simple ruby app to Crystal to see how it is and most of it is copy/paste and then some fixing and adding type definitions where needed. The only issues I have had are where I am using a rubygem that doesn't have a crystal counterpart. For example a pretty simple Gem that connects to a socket and parses incoming data was exceedingly easy to port, including tests. * https://gitlab.com/overlord-bot/tacview-ruby-client https://gitlab.com/overlord-bot/tacview-ruby-client * https://gitlab.com/overlord-bot/tacview-crystal-client https://gitlab.com/overlord-bot/tacview-crystal-client For a more specific example these two files are almost identical: * https://gitlab.com/overlord-bot/tacview-ruby-client/blob/master/lib/tacview_client/client.rb https://gitlab.com/overlord-bot/tacview-ruby-client/blob/mas... * https://gitlab.com/overlord-bot/tacview-crystal-client/blob/master/src/tacview_client/client.cr https://gitlab.com/overlord-bot/tacview-crystal-client/blob/... The only thing not done was CRC hash calculating for a password because there was no crystal shard for it and it was low priority so I didn't write one.
        chrisseaton 7y ago
- Nextgrid 7y ago
  Remember that the hyped-up companies you hear about on HN & other social media aren't the entire world. There are plenty of companies out there that stay quiet and outside of the spotlight and use the language just fine. The same applies for PHP and other languages that are considered (unfairly IMO) "old-school".
  - eropple 7y ago
    PHP is rarely considered "old school." It's considered bad. And not without reason given its history of hostility to its own developers and the sysadmins who have to manage it. I think pretty much everyone has acknowledged that it's improved. Where opinions differ is in how much it has improved and whether that's enough to entertain its use (my answers to which are "not enough" and "not even if you paid me", respectively).
    - Nextgrid 7y ago
      I agree that PHP is bad in certain ways, even though I started with PHP before transitioning to other languages. But honestly, every language has to make some trade-offs. Even if PHP has some things that are bad design choices as opposed to trade-offs, it can still be worthwhile to put up with them if you have an existing codebase written in it or want to take advantage of libraries that don't exist in other languages. As a result I don't consider any language as bad or old school. They might have downsides but 1) the upsides might outweigh them in certain use-cases and 2) any competent developer should be able to make something good with any Turing-complete language so I don't judge by the language alone, and especially not by the "hype factor" of the language.
    - planetzero 7y ago
      "PHP is rarely considered "old school." It's considered bad" PHP hasn't been considered 'bad' for a few years now. It has been battle tested on many large-scale websites. Now, One of the main issues is that anyone can write a few lines of code and call themselves a developer, so you have horrible code bases still out there...but this has more to do with the developer than the language. "my answers to which are "not enough" and "not even if you paid me", respectively I love hearing answers like this. This is why I'm still paid so well to write PHP code after 15 years in the industry.
pm90 7y ago
It’s not mentioned but I’m assuming that they built their own OIDC/OAuth backend and not use existing ones (eg okta, Auth0 etc). It would be interesting to know the details of how they’re doing authorization. It appears that it’s all or nothing but I might be mistaken.
- YawningAngel 7y ago
  Running an OAuth2 server isn't tremendously involved. There are good open-source projects like https://github.com/ory/hydra https://github.com/ory/hydra that are pretty easy to configure.
  - inferiorhuman 7y ago
    Oh god, at megacorp we implemented our own OAuth2 stack. Much sadness ensued.
    - masonhensley 7y ago
      Been there, done that - wish it upon no one. If anyone ever brings up the idea of building out oauth or even vaguely user management, I try to point them to at least try a POC (Proof of Concept) with https://www.keycloak.org/ https://www.keycloak.org/ (Apache 2.0 License) or https://www.gluu.org/ https://www.gluu.org/ (MIT License) before they considering building.
      - corford 7y ago
        Another solution is OpenLDAP (or JumpCloud) at the root and then supporting software: OpenLDAP ├── PrivacyIDEA (TOTP/MFA with LDAP auth backend) ├──---└── SAML iDp (e.g. SimpleSAMLphp or Shibboleth) for SSO: AWS, Google, Github, Atlassian, Snowflake, Azure etc. ├── Dex (https://github.com/dexidp/dex) for anything that wants Oauth flow ├── Native LDAP for apps that support it (e.g. Metabase, Grafana) ├── Any other custom authT that supports LDAP as a backend OpenLDAP itself isn't for the faint hearted but I've had a lot of success with JumpCloud (and Okta also have an LDAP directory service... though starting price is high).
delidumrul 7y ago
Well design of one-to-one relationship between users and shops. Then, they solve the problem that approach has brought. Nothing particular at the article.
deleted 7y ago
[deleted]
derptron 7y ago
This is really more about how they migrated legacy accounts. Everything else is just leveraging Open ID. Nothing interesting in this article at all.

174 points by frankpf 7y ago

So is Shopify the biggest fish still on the Ruby stack? Nice article detailing how they did an upgrade to openId connect to allow SSO on multiple shops within a client company.

noodle 7y ago
IIRC there are still many large companies using Ruby/Rails still, they've just also diversified their tech stacks (as larger companies tend to do). AFAIK the list includes: GitHub (MS has a few Rails-based acquisitions now), Airbnb, Groupon, Square, Cookpad, Kickstarter, Hulu, etc..
- keithwhor 7y ago
  Pretty sure Stripe is a Ruby shop as well.
  - multiplegeorges 7y ago
    One of the biggest. They recently released Sorbet, a gradual type checker for Ruby.
    - ivankolev 7y ago
      I sometimes forget how vast and diverse the whole industry is, so many tech. stacks and ecosystems.
  - yellow_lead 7y ago
    Gitlab too it seems.
  - riffraff 7y ago
    IIRC it's ruby, but not rails
  - faintrain 7y ago
    JP Morgan Chase and McKesson both use Ruby with McKesson leading the pack in RoR LOC among the Fortune 50
- itake 7y ago
  Insiders at Airbnb told me they are (almost) completely off Rails now. They moved their website to the jvm.
  - notyourwork 7y ago
    Seems at some scale this usually happens.
  - cutler 7y ago
    If all you want is the JVM JRuby and its Truffle variant are viable options.
- ksec 7y ago
  I believe AirBnB and Groupon are off Rails ( And Ruby ). So the big one for Rails are Shopify, Github, Cookpad, Gitlab. For Ruby ( Not Rails ) That would be Stripe.
uyuioi 7y ago
They’re still using a rails setup for their core services. But, this core falls over with him RPM. It’s failed in nearly all flash sales it has had to handle. Rails is great. But commerce is heavy and I don’t believe Shopify can keep its existing core code base around much longer without a significant change to aid with performance.
stickfigure 7y ago
Funny you mention this. I just today had to implement a painful workaround for Shopify's insanely short timeout on product image uploads. On submitting an image url, you apparently get 4s to complete the whole transfer. I found hundreds of people complaining about this in the community forums, going back years. If you're dynamically generating images, or on a congested network, 4s is far too short. Since this is a simple config property, the only justification I can imagine is that they are trying to restrict the amount of time that their (single-threaded, memory-hungry) instances are occupied. Because of Ruby's poor resource management, a core part of their API is barely usable. I'm pretty disappointed.
- uyuioi 7y ago
  I do a lot of developing with Shopify and it’s a mess. One of the worst development experiences of all time. Because they’re so monolithic focused. The API’s you can use are pretty sluggish and poorly documented. Rails is just not meant for heavy transactional load. And e-commerce needs async to handle what can be a huge load. Taobao is java or php and they handle load far greater without fault. Shopify is much better than Magento though.
  - kenrose 7y ago
    > One of the worst development experiences of all time... The API’s you can use are pretty sluggish and poorly documented. Can you elaborate? Shopify released their GraphQL Admin API in 2018, which has built in documentation, as well as an interactive IDE (Graphiql) that you can run on your shop. As long as you're not doing anything too crazy that exceeds the throttle (e.g., syncing 1000s of products), they're pretty good about listening to developer requests. https://help.shopify.com/en/api/graphql-admin-api https://help.shopify.com/en/api/graphql-admin-api > Rails is just not meant for heavy transactional load. Shopify has invested heavily in their sharding setup so that it can be handle high load and scale quickly. e.g., flash sales where baseline traffic will 3x in a matter of seconds. See the discussion of pods: https://engineering.shopify.com/blogs/engineering/e-commerce-at-scale-inside-shopifys-tech-stack https://engineering.shopify.com/blogs/engineering/e-commerce... > And e-commerce needs async to handle what can be a huge load. Long running processes are async. However, having commerce modeled by a transactional database that provides atomicity is a boon for simplicity. You don't want to deal with eventual consistency when updating inventory or placing orders. Note: Am ex-Shopify. Ran the API team.
Thaxll 7y ago
They're moving slow parts to Go.
- uyuioi 7y ago
  Hopefully crystal Lang will make its way into these ruby heavy shops. They’ll get 100x performance without needing to really think in a whole different programming experience.
  - chrisseaton 7y ago
    > without needing to really think in a whole different programming experience But Crystal has entirely different semantics to Ruby. They look vaguely similar at a superficial level, but the semantics are not even remotely similar.
    - rurounijones 7y ago
      Maybe but depending on what you are doing they are minor considerations. I am porting a relatively simple ruby app to Crystal to see how it is and most of it is copy/paste and then some fixing and adding type definitions where needed. The only issues I have had are where I am using a rubygem that doesn't have a crystal counterpart. For example a pretty simple Gem that connects to a socket and parses incoming data was exceedingly easy to port, including tests. * https://gitlab.com/overlord-bot/tacview-ruby-client https://gitlab.com/overlord-bot/tacview-ruby-client * https://gitlab.com/overlord-bot/tacview-crystal-client https://gitlab.com/overlord-bot/tacview-crystal-client For a more specific example these two files are almost identical: * https://gitlab.com/overlord-bot/tacview-ruby-client/blob/master/lib/tacview_client/client.rb https://gitlab.com/overlord-bot/tacview-ruby-client/blob/mas... * https://gitlab.com/overlord-bot/tacview-crystal-client/blob/master/src/tacview_client/client.cr https://gitlab.com/overlord-bot/tacview-crystal-client/blob/... The only thing not done was CRC hash calculating for a password because there was no crystal shard for it and it was low priority so I didn't write one.
      - chrisseaton 7y ago
Nextgrid 7y ago
Remember that the hyped-up companies you hear about on HN & other social media aren't the entire world. There are plenty of companies out there that stay quiet and outside of the spotlight and use the language just fine. The same applies for PHP and other languages that are considered (unfairly IMO) "old-school".
- eropple 7y ago
  PHP is rarely considered "old school." It's considered bad. And not without reason given its history of hostility to its own developers and the sysadmins who have to manage it. I think pretty much everyone has acknowledged that it's improved. Where opinions differ is in how much it has improved and whether that's enough to entertain its use (my answers to which are "not enough" and "not even if you paid me", respectively).
  - Nextgrid 7y ago
    I agree that PHP is bad in certain ways, even though I started with PHP before transitioning to other languages. But honestly, every language has to make some trade-offs. Even if PHP has some things that are bad design choices as opposed to trade-offs, it can still be worthwhile to put up with them if you have an existing codebase written in it or want to take advantage of libraries that don't exist in other languages. As a result I don't consider any language as bad or old school. They might have downsides but 1) the upsides might outweigh them in certain use-cases and 2) any competent developer should be able to make something good with any Turing-complete language so I don't judge by the language alone, and especially not by the "hype factor" of the language.
  - planetzero 7y ago
    "PHP is rarely considered "old school." It's considered bad" PHP hasn't been considered 'bad' for a few years now. It has been battle tested on many large-scale websites. Now, One of the main issues is that anyone can write a few lines of code and call themselves a developer, so you have horrible code bases still out there...but this has more to do with the developer than the language. "my answers to which are "not enough" and "not even if you paid me", respectively I love hearing answers like this. This is why I'm still paid so well to write PHP code after 15 years in the industry.

pm90 7y ago

It’s not mentioned but I’m assuming that they built their own OIDC/OAuth backend and not use existing ones (eg okta, Auth0 etc). It would be interesting to know the details of how they’re doing authorization. It appears that it’s all or nothing but I might be mistaken.

YawningAngel 7y ago
Running an OAuth2 server isn't tremendously involved. There are good open-source projects like https://github.com/ory/hydra https://github.com/ory/hydra that are pretty easy to configure.
- inferiorhuman 7y ago
  Oh god, at megacorp we implemented our own OAuth2 stack. Much sadness ensued.
  - masonhensley 7y ago
    Been there, done that - wish it upon no one. If anyone ever brings up the idea of building out oauth or even vaguely user management, I try to point them to at least try a POC (Proof of Concept) with https://www.keycloak.org/ https://www.keycloak.org/ (Apache 2.0 License) or https://www.gluu.org/ https://www.gluu.org/ (MIT License) before they considering building.
    - corford 7y ago
      Another solution is OpenLDAP (or JumpCloud) at the root and then supporting software: OpenLDAP ├── PrivacyIDEA (TOTP/MFA with LDAP auth backend) ├──---└── SAML iDp (e.g. SimpleSAMLphp or Shibboleth) for SSO: AWS, Google, Github, Atlassian, Snowflake, Azure etc. ├── Dex (https://github.com/dexidp/dex) for anything that wants Oauth flow ├── Native LDAP for apps that support it (e.g. Metabase, Grafana) ├── Any other custom authT that supports LDAP as a backend OpenLDAP itself isn't for the faint hearted but I've had a lot of success with JumpCloud (and Okta also have an LDAP directory service... though starting price is high).

delidumrul 7y ago

Well design of one-to-one relationship between users and shops. Then, they solve the problem that approach has brought. Nothing particular at the article.

deleted 7y ago

[deleted]

derptron 7y ago

This is really more about how they migrated legacy accounts. Everything else is just leveraging Open ID. Nothing interesting in this article at all.