9 ms·

Moving persistent data out of Redis

220 points by samlambert 10y ago

coldcode 10y ago
I wonder if this will allow better scaling of GitHub enterprise. We are pegging our usage; if we could we would migrate everything to Gitlab Enterprise (which we also have) which seems to have better scalability.
- gjtorikian 10y ago
  > which seems to have better scalability Out of curiosity: how many users do you have, and what measurements are you using to determine scalability? Disclaimer: I work at GitHub.
- sytse 10y ago
  How can we help you move to GitLab EE? (As you indicated it scales to 100k users so that shouldn't we the problem)
  - john832 10y ago
    Does GitLab really handle 100k users? Where is this indicated? https://gitlab.com/gitlab-org/gitlab-ce/issues/26405#note_20968879 https://gitlab.com/gitlab-org/gitlab-ce/issues/26405#note_20...
    - sytse 10y ago
      Thanks for asking. The issue you referred to talks about users on a single machine. Unlike GitHub you can run a cluster of application servers with GitLab https://about.gitlab.com/high-availability/ https://about.gitlab.com/high-availability/ Some of our users have 25k+ users on their cluster. We know GitLab can scale to 100k users because we run GitLab Enterprise Edition without modifications on GitLab.com GitLab.com currently has much more than 100k users and the performance leaves much to be desired https://gitlab.com/gitlab-com/infrastructure/issues/947 https://gitlab.com/gitlab-com/infrastructure/issues/947 But we're comfortable that you can run 100k users on a cluster of machines without much tuning.
      - marcinkuzminski 10y ago
        can you share how many machines handles gitlab.com currently?
        sytse 10y ago
        It is around 100 of machines. Commonly it is a couple of thousand active users per application server, but your mileage may vary based on many things.
      - john832 10y ago
        > Unlike GitHub GitHub seems to have a clustering product that came out a year ago, and it looks like IBM has reported they're running over 13,000 users on it (back in August, i imagine it's closer to 20,000 by now): https://www.ibm.com/blogs/bluemix/2016/08/ibm-internal-github-enterprise-system-running-blue-box/ https://www.ibm.com/blogs/bluemix/2016/08/ibm-internal-githu... > We know GitLab can scale to 100k users because we run GitLab Enterprise Edition without modifications on GitLab.com What does "modification" mean in this context? Beyond recommended specifications listed at https://docs.gitlab.com/ee/install/requirements.html#cpu https://docs.gitlab.com/ee/install/requirements.html#cpu and https://docs.gitlab.com/ee/install/requirements.html#memory https://docs.gitlab.com/ee/install/requirements.html#memory? it doesnt list anything above 40,000 users. Beyond that, the HA documentation (https://docs.gitlab.com/ce/administration/high_availability/README.html https://docs.gitlab.com/ce/administration/high_availability/...) isn't _really_ active/active HA, it says it is but it's not true. True active/active would mean that you wouldn't rely on a shared NFS server, postgres, or redis server. However, with https://help.github.com/enterprise/2.8/admin/guides/clustering/about-cluster-nodes/#services-required-for-clustering https://help.github.com/enterprise/2.8/admin/guides/clusteri... it sounds like my organization can scale to well over 100,000 by adding more clustering nodes instead of trying to figure out independently how to scale services like redis or postgres or a shared NFS server. > without much tuning. I'd love to hear a comparison between the two products, and also what kind of tuning you've done on gitlab.com to support those user numbers. Would love to see that on the documentation to support the open source way!
antirez 10y ago
Wow I'm learning today that Github used Redis for persistent data, now that they moved away :-) Anyway very happy that Redis helped to run such an important site. From the blog post it looks like that for certain things to move away from Redis was hard even if they are very skilled with MySQL, this is a good thing from the POV of Redis since it means that Redis allows to model certain things easily. However they wanted to move away as an important priority, so I wish to know why they wanted to move away so badly and how Redis could be improved in order to serve better the users. If Redis was better for their use case, they could have avoided to move to MySQL I guess. Unfortunately the blog post is short of details on that regard, perhaps because the blog post author(s) are too gentle to bash Redis after using it for a long time.
- clarkenheim 10y ago
  Reading more into this point in the post than i maybe should but "Take advantage of our expertise operating MySQL." sounds like they have more engineers familiar and comfortable working with MySQL than they are with Redis.
  - antirez 10y ago
    That's a valid reason indeed. Also technological consolidation, that is, if I can do everything with a single DB / language / ... I always tend to use a single thing.
    - virmundi 10y ago
      Watch how far you generalize that view. I realize that "I can do everything with a single...language" can make "everything" mean anything. I had a coworker that helped a middle eastern country with security software. He went full Javascript on it: Node, Mongo, etc. It worked. JS did everything. While he didn't go into details, afterwards he thought it was a bad idea. A great learning experience on when to define "everything" properly as well as what is outside of it.
      - ohstopitu 10y ago
epberry 10y ago
Cool stuff. I found it especially interesting how they removed 30% of writes with new logic to compose some timelines of events in other timelines. It's a thought provoking optimization that calls to mind graph partitioning. For example you have 10 people in your organization with various permissions on repos. Some people (CTO let's say) can see every repo while others might only be able to see some repos. Or you might have consultants or open source projects which non-employees contribute to. Then you construct a graph where each node is an contributor that is connected to other contributors by the permissions they have on repos (or are the repos the nodes and the contributor permissions the connections?). Finally you run a graph partitioning algorithm where the number of partitions is the number of unique timelines you have to write for an organization. Thinking about an organization with closer to 500 contributors I can see how this could reduce the number of timelines by 30%.
matharmin 10y ago
Working with large amounts of persistent data is hard. Limiting the architecture to a single database system (MySQL in this case) generally makes managing and scaling much easier, versus having to know/learn how to scale multiple systems independently. Even if Redis was a better fit for some of their use cases, it just makes it much easier to not have the additional persistent database system to manage.
rch 10y ago
> We changed up how writing to and reading from Redis keys worked for [the organization] timeline before even thinking about MySQL ... This resulted in a dramatic 65% reduction of the write operations in for this feature. Interesting. Is there a comparison of overall performance between the intermediate design (w/ Redis) and what they ended up with?
tschellenbach 10y ago
Wonder why they didn't use Cassandra for this use case.
- vacri 10y ago
  Last year I was setting up a trial of Cassandra for something, going through the usual swearing of a new tool not quite working as expected (eg by default picking a random port for inter-node communication)... and the next desk over, a non-tech colleague called Cassandra kept hearing me mutter angrily about 'cassandra' and wondered what she'd done. Whoops :)
  - flurdy 10y ago
    Do you sit close to Ezekiel as well? Though having someone sit near me not knowing what I am/we are working on would surprise me. But it does happen especially if there are hotdesks nearby for people from other offices to work on temporarily. I do swear loudly often so probably not a good choice to have those too near me...
- lostcolony 10y ago
  They told you their motivations. Reduce complexity, and take advantage of their MySQL knowledge. Cassandra would fail both of those if they didn't have a lot of MySQL knowledge, since setting it up and maintaining it would increase complexity by introducing a tool they did not know.
jhgg 10y ago
We've recently had to move away from redis for persistent data storage at work too - opting instead to write a service layer ontop of cassandra for storing data. Redis was tremendous in our journey up there - but one of the shortcomings is that it isn't as easy to scale-up as cassandra is if you haven't designed your system to scale-up on redis from when it was built (which we didn't) - instead of re-architecting for a redis-cluster setup, we decided to move the component to a clustered microservice written in go, that sits as a memory-cache & write buffer infront of cassandra for hot, highly mutated data. Would anyone be interested in a blog post about our struggles & journey?
- meritt 10y ago
  I'd be interested in seeing that, also exactly what bottlenecks were you running into? CPU? I/O? Memory?
  - jhgg 10y ago
    Memory & CPU. We knew we'd be running into both eventually - without adding additional redis instances (memory growing way faster than CPU usage in this specific scenario). When it was initially built, it was basically a bunch of redis lua scripts to handle updating the data - on redis configured in master/slave managed by sentinels. Given the nature of the data too, only the data for active users would be hot - but users that were inactive would have stuck around in memory needlessly. Our new system keeps only the hot set of users in memory. We also built it to transparently migrate users from redis to cassandra when they were accessed.
- cookiecaper 10y ago
  >instead of re-architecting for a redis-cluster setup, we decided to move the component to a clustered microservice written in go, that sits as a memory-cache & write buffer infront of cassandra for hot, highly mutated data. Somehow setting up a Redis cluster and doing whatever you have to do to distribute/shard your keys effectively (which afaik is not much) does sound a little more efficient than rewriting a clustered microservice in Go with a Cassandra backend. Redis clustering is actually quite easy. Forgive me if I seem grumpy. My recent experiences have caused the "We had a minor issue, so we redid everything in a Totally Cool Super-Neato New Stack That Integrates All The Hiring Manager's Favorite Buzzwords!" perspective to become a bit grating. Redis is one of the few new pieces of infrastructure over the last 10 years that's truly deserving of its position.
activatedgeek 10y ago
Can anybody here help me understand why many teams are using MySQL as a KV store? (Uber did it recently, so assuming many others probably did it too, network effect) I personally love MySQL. Just want to understand what makes MySQL a great KV store as opposed to more seemingly specialized systems like Redis?
- Sanddancer 10y ago
  Less complexity, I imagine. If you've already got a beefy MySQL setup, then maintaining a separate Redis setup will introduce quite a bit of redundancy and chances for failure. Given that the big SQL databases have key value engines these days, you're going to get decent performance out of them as well.
- adrr 10y ago
  All the tooling around SQL databases that allow you to easily setup fixtures, backups/restores, monitoring,viewing data etc. Also you can scale wide through read slaves easily.
- rhizome 10y ago
  For me a primary reason would be to avoid platform spread.
- nvartolomei 10y ago
  Operational complexity at scale and prior experience with it. It is one thing to run one instance system and completely different story to run a clustered setup for large amounts of data with performance consideration. MySQL also proved very good at scale, at Facebook, YouTube, Uber, etc. and there are a lot of people with experience running it. Uber wrote a very good article about their switch, for them it was more about performance, though a little bit controversial.
- toomuchtodo 10y ago
  > Just want to understand what makes MySQL a great KV store Its not, it just happens to be good enough, which matters a lot for operational expertise/costs/etc. For example, you can store hundreds of millions of KV rows in an InnoDB table and still have <1-3ms response times on queries, while having persistence built in. Perfect is the enemy of good enough.
adrr 10y ago
I don't know why people use redis as an LRU cache. Its a terrible LRU cache. Its eviction algorithm isn't true LRU and does sampling which may cause new keys to get incorrectly convicted. LRU is also really slow being single threaded.
- zaius 10y ago
  What should people use?
  - adrr 10y ago
    Memcached is a LRU cache.
    - eriknstr 10y ago
      I went and searched for memcached redis because up until now I haven't gotten around to checking out the differences. Here are some things I found. http://antirez.com/news/94 http://antirez.com/news/94 http://stackoverflow.com/questions/10558465/memcached-vs-redis http://stackoverflow.com/questions/10558465/memcached-vs-red... http://www.infoworld.com/article/3063161/application-development/why-redis-beats-memcached-for-caching.html http://www.infoworld.com/article/3063161/application-develop...
  - aphextron 10y ago
    use Redis when you need advanced data structures built in. Memcached is just a KV store.
- twblalock 10y ago
  A lot of uses of Redis are suboptimal, but easy to set up and good enough. For example, "true" LRU is not always needed -- an approximation of LRU is sometimes sufficient. It's often not worth the time and money to do better. When you need to scale past what Redis can provide, you can move on to a different solution, as Github has done.
- ec109685 10y ago
  Why does single threaded make it slow? Also, 3.0 has a better LRU algorithm. https://redis.io/topics/lru-cache https://redis.io/topics/lru-cache
amalag 10y ago
Activity streams are such a common use case. It is very interesting that Soundcloud chose to do something different: https://developers.soundcloud.com/blog/roshi-a-crdt-system-for-timestamped-events https://developers.soundcloud.com/blog/roshi-a-crdt-system-f... Assembling the inbox on demand is quite interesting. I don't quite understand the querying and operations involved with Roshi for doing that.
buremba 10y ago
It would be nice if they can share some info about Github::KV.
aerialcombat 10y ago
I love Redis!
antman 10y ago
Has Github stopped allowing free searching for code? All I get is "Must include at least one user, organization, or repository". I think that its a greater problem than the speed of its streams.
- avian 10y ago
  I believe this restriction was put in place to limit searching for accidentally committed AWS keys.
  - whatismyip 10y ago
    AWS keys are already being detected and the account owners get notified when they are found. Private key material was a bigger issue at one point in time (i.e dotfiles/.ssh/id_rsa).
- j_s 10y ago
  Have to be logged in.
sandGorgon 10y ago
its fairly unfortunate that they are tied to mysql, because this is pretty much the usecase that postgresql jsonb was built for. it has first class support in most ORM, and works quite well.

220 points by samlambert 10y ago

I wonder if this will allow better scaling of GitHub enterprise. We are pegging our usage; if we could we would migrate everything to Gitlab Enterprise (which we also have) which seems to have better scalability.

gjtorikian 10y ago
> which seems to have better scalability Out of curiosity: how many users do you have, and what measurements are you using to determine scalability? Disclaimer: I work at GitHub.
sytse 10y ago
How can we help you move to GitLab EE? (As you indicated it scales to 100k users so that shouldn't we the problem)
- john832 10y ago
  Does GitLab really handle 100k users? Where is this indicated? https://gitlab.com/gitlab-org/gitlab-ce/issues/26405#note_20968879 https://gitlab.com/gitlab-org/gitlab-ce/issues/26405#note_20...
  - sytse 10y ago
    Thanks for asking. The issue you referred to talks about users on a single machine. Unlike GitHub you can run a cluster of application servers with GitLab https://about.gitlab.com/high-availability/ https://about.gitlab.com/high-availability/ Some of our users have 25k+ users on their cluster. We know GitLab can scale to 100k users because we run GitLab Enterprise Edition without modifications on GitLab.com GitLab.com currently has much more than 100k users and the performance leaves much to be desired https://gitlab.com/gitlab-com/infrastructure/issues/947 https://gitlab.com/gitlab-com/infrastructure/issues/947 But we're comfortable that you can run 100k users on a cluster of machines without much tuning.
    - marcinkuzminski 10y ago
      can you share how many machines handles gitlab.com currently?
      - sytse 10y ago
        It is around 100 of machines. Commonly it is a couple of thousand active users per application server, but your mileage may vary based on many things.
    - john832 10y ago
      > Unlike GitHub GitHub seems to have a clustering product that came out a year ago, and it looks like IBM has reported they're running over 13,000 users on it (back in August, i imagine it's closer to 20,000 by now): https://www.ibm.com/blogs/bluemix/2016/08/ibm-internal-github-enterprise-system-running-blue-box/ https://www.ibm.com/blogs/bluemix/2016/08/ibm-internal-githu... > We know GitLab can scale to 100k users because we run GitLab Enterprise Edition without modifications on GitLab.com What does "modification" mean in this context? Beyond recommended specifications listed at https://docs.gitlab.com/ee/install/requirements.html#cpu https://docs.gitlab.com/ee/install/requirements.html#cpu and https://docs.gitlab.com/ee/install/requirements.html#memory https://docs.gitlab.com/ee/install/requirements.html#memory? it doesnt list anything above 40,000 users. Beyond that, the HA documentation (https://docs.gitlab.com/ce/administration/high_availability/README.html https://docs.gitlab.com/ce/administration/high_availability/...) isn't _really_ active/active HA, it says it is but it's not true. True active/active would mean that you wouldn't rely on a shared NFS server, postgres, or redis server. However, with https://help.github.com/enterprise/2.8/admin/guides/clustering/about-cluster-nodes/#services-required-for-clustering https://help.github.com/enterprise/2.8/admin/guides/clusteri... it sounds like my organization can scale to well over 100,000 by adding more clustering nodes instead of trying to figure out independently how to scale services like redis or postgres or a shared NFS server. > without much tuning. I'd love to hear a comparison between the two products, and also what kind of tuning you've done on gitlab.com to support those user numbers. Would love to see that on the documentation to support the open source way!

antirez 10y ago

Wow I'm learning today that Github used Redis for persistent data, now that they moved away :-) Anyway very happy that Redis helped to run such an important site. From the blog post it looks like that for certain things to move away from Redis was hard even if they are very skilled with MySQL, this is a good thing from the POV of Redis since it means that Redis allows to model certain things easily. However they wanted to move away as an important priority, so I wish to know why they wanted to move away so badly and how Redis could be improved in order to serve better the users. If Redis was better for their use case, they could have avoided to move to MySQL I guess. Unfortunately the blog post is short of details on that regard, perhaps because the blog post author(s) are too gentle to bash Redis after using it for a long time.

clarkenheim 10y ago
Reading more into this point in the post than i maybe should but "Take advantage of our expertise operating MySQL." sounds like they have more engineers familiar and comfortable working with MySQL than they are with Redis.
- antirez 10y ago
  That's a valid reason indeed. Also technological consolidation, that is, if I can do everything with a single DB / language / ... I always tend to use a single thing.
  - virmundi 10y ago
    Watch how far you generalize that view. I realize that "I can do everything with a single...language" can make "everything" mean anything. I had a coworker that helped a middle eastern country with security software. He went full Javascript on it: Node, Mongo, etc. It worked. JS did everything. While he didn't go into details, afterwards he thought it was a bad idea. A great learning experience on when to define "everything" properly as well as what is outside of it.
    - ohstopitu 10y ago

epberry 10y ago

Cool stuff. I found it especially interesting how they removed 30% of writes with new logic to compose some timelines of events in other timelines. It's a thought provoking optimization that calls to mind graph partitioning. For example you have 10 people in your organization with various permissions on repos. Some people (CTO let's say) can see every repo while others might only be able to see some repos. Or you might have consultants or open source projects which non-employees contribute to. Then you construct a graph where each node is an contributor that is connected to other contributors by the permissions they have on repos (or are the repos the nodes and the contributor permissions the connections?). Finally you run a graph partitioning algorithm where the number of partitions is the number of unique timelines you have to write for an organization. Thinking about an organization with closer to 500 contributors I can see how this could reduce the number of timelines by 30%.

matharmin 10y ago

Working with large amounts of persistent data is hard. Limiting the architecture to a single database system (MySQL in this case) generally makes managing and scaling much easier, versus having to know/learn how to scale multiple systems independently. Even if Redis was a better fit for some of their use cases, it just makes it much easier to not have the additional persistent database system to manage.

rch 10y ago

> We changed up how writing to and reading from Redis keys worked for [the organization] timeline before even thinking about MySQL ... This resulted in a dramatic 65% reduction of the write operations in for this feature. Interesting. Is there a comparison of overall performance between the intermediate design (w/ Redis) and what they ended up with?

tschellenbach 10y ago

Wonder why they didn't use Cassandra for this use case.

vacri 10y ago
Last year I was setting up a trial of Cassandra for something, going through the usual swearing of a new tool not quite working as expected (eg by default picking a random port for inter-node communication)... and the next desk over, a non-tech colleague called Cassandra kept hearing me mutter angrily about 'cassandra' and wondered what she'd done. Whoops :)
- flurdy 10y ago
  Do you sit close to Ezekiel as well? Though having someone sit near me not knowing what I am/we are working on would surprise me. But it does happen especially if there are hotdesks nearby for people from other offices to work on temporarily. I do swear loudly often so probably not a good choice to have those too near me...
lostcolony 10y ago
They told you their motivations. Reduce complexity, and take advantage of their MySQL knowledge. Cassandra would fail both of those if they didn't have a lot of MySQL knowledge, since setting it up and maintaining it would increase complexity by introducing a tool they did not know.

jhgg 10y ago

We've recently had to move away from redis for persistent data storage at work too - opting instead to write a service layer ontop of cassandra for storing data. Redis was tremendous in our journey up there - but one of the shortcomings is that it isn't as easy to scale-up as cassandra is if you haven't designed your system to scale-up on redis from when it was built (which we didn't) - instead of re-architecting for a redis-cluster setup, we decided to move the component to a clustered microservice written in go, that sits as a memory-cache & write buffer infront of cassandra for hot, highly mutated data. Would anyone be interested in a blog post about our struggles & journey?

meritt 10y ago
I'd be interested in seeing that, also exactly what bottlenecks were you running into? CPU? I/O? Memory?
- jhgg 10y ago
  Memory & CPU. We knew we'd be running into both eventually - without adding additional redis instances (memory growing way faster than CPU usage in this specific scenario). When it was initially built, it was basically a bunch of redis lua scripts to handle updating the data - on redis configured in master/slave managed by sentinels. Given the nature of the data too, only the data for active users would be hot - but users that were inactive would have stuck around in memory needlessly. Our new system keeps only the hot set of users in memory. We also built it to transparently migrate users from redis to cassandra when they were accessed.
cookiecaper 10y ago
>instead of re-architecting for a redis-cluster setup, we decided to move the component to a clustered microservice written in go, that sits as a memory-cache & write buffer infront of cassandra for hot, highly mutated data. Somehow setting up a Redis cluster and doing whatever you have to do to distribute/shard your keys effectively (which afaik is not much) does sound a little more efficient than rewriting a clustered microservice in Go with a Cassandra backend. Redis clustering is actually quite easy. Forgive me if I seem grumpy. My recent experiences have caused the "We had a minor issue, so we redid everything in a Totally Cool Super-Neato New Stack That Integrates All The Hiring Manager's Favorite Buzzwords!" perspective to become a bit grating. Redis is one of the few new pieces of infrastructure over the last 10 years that's truly deserving of its position.

activatedgeek 10y ago

Can anybody here help me understand why many teams are using MySQL as a KV store? (Uber did it recently, so assuming many others probably did it too, network effect) I personally love MySQL. Just want to understand what makes MySQL a great KV store as opposed to more seemingly specialized systems like Redis?

Sanddancer 10y ago
Less complexity, I imagine. If you've already got a beefy MySQL setup, then maintaining a separate Redis setup will introduce quite a bit of redundancy and chances for failure. Given that the big SQL databases have key value engines these days, you're going to get decent performance out of them as well.
adrr 10y ago
All the tooling around SQL databases that allow you to easily setup fixtures, backups/restores, monitoring,viewing data etc. Also you can scale wide through read slaves easily.
rhizome 10y ago
For me a primary reason would be to avoid platform spread.
nvartolomei 10y ago
Operational complexity at scale and prior experience with it. It is one thing to run one instance system and completely different story to run a clustered setup for large amounts of data with performance consideration. MySQL also proved very good at scale, at Facebook, YouTube, Uber, etc. and there are a lot of people with experience running it. Uber wrote a very good article about their switch, for them it was more about performance, though a little bit controversial.
toomuchtodo 10y ago
> Just want to understand what makes MySQL a great KV store Its not, it just happens to be good enough, which matters a lot for operational expertise/costs/etc. For example, you can store hundreds of millions of KV rows in an InnoDB table and still have <1-3ms response times on queries, while having persistence built in. Perfect is the enemy of good enough.

adrr 10y ago

I don't know why people use redis as an LRU cache. Its a terrible LRU cache. Its eviction algorithm isn't true LRU and does sampling which may cause new keys to get incorrectly convicted. LRU is also really slow being single threaded.

zaius 10y ago
What should people use?
- adrr 10y ago
  Memcached is a LRU cache.
  - eriknstr 10y ago
    I went and searched for memcached redis because up until now I haven't gotten around to checking out the differences. Here are some things I found. http://antirez.com/news/94 http://antirez.com/news/94 http://stackoverflow.com/questions/10558465/memcached-vs-redis http://stackoverflow.com/questions/10558465/memcached-vs-red... http://www.infoworld.com/article/3063161/application-development/why-redis-beats-memcached-for-caching.html http://www.infoworld.com/article/3063161/application-develop...
- aphextron 10y ago
  use Redis when you need advanced data structures built in. Memcached is just a KV store.
twblalock 10y ago
A lot of uses of Redis are suboptimal, but easy to set up and good enough. For example, "true" LRU is not always needed -- an approximation of LRU is sometimes sufficient. It's often not worth the time and money to do better. When you need to scale past what Redis can provide, you can move on to a different solution, as Github has done.
ec109685 10y ago
Why does single threaded make it slow? Also, 3.0 has a better LRU algorithm. https://redis.io/topics/lru-cache https://redis.io/topics/lru-cache

amalag 10y ago

Activity streams are such a common use case. It is very interesting that Soundcloud chose to do something different: https://developers.soundcloud.com/blog/roshi-a-crdt-system-for-timestamped-events https://developers.soundcloud.com/blog/roshi-a-crdt-system-f... Assembling the inbox on demand is quite interesting. I don't quite understand the querying and operations involved with Roshi for doing that.

buremba 10y ago

It would be nice if they can share some info about Github::KV.

aerialcombat 10y ago

I love Redis!

antman 10y ago

Has Github stopped allowing free searching for code? All I get is "Must include at least one user, organization, or repository". I think that its a greater problem than the speed of its streams.

avian 10y ago
I believe this restriction was put in place to limit searching for accidentally committed AWS keys.
- whatismyip 10y ago
  AWS keys are already being detected and the account owners get notified when they are found. Private key material was a bigger issue at one point in time (i.e dotfiles/.ssh/id_rsa).
j_s 10y ago
Have to be logged in.

sandGorgon 10y ago

its fairly unfortunate that they are tied to mysql, because this is pretty much the usecase that postgresql jsonb was built for. it has first class support in most ORM, and works quite well.