3 ms·

Can existing LDAP implementations scale a single directory to hundreds of millions of users? I can find no clear answer after a few minutes of searching.

by bagels 12y ago

viraptor 12y ago
OpenLDAP is limited by LMDB, which is pretty much limited only by what your filesystem and address space can handle. For scaling out it does support multi-master and slave configurations. So yes, it should.
beagle3 12y ago
You can probably list the organizations (outside of China, perhaps) that have over a hundred million of users in a twitter message - (Facebook;Twitter;Google;Walmart;WhatsApp;Instagram;Hotmail;Snapchat;Yahoo;IRS[0];Blizzard;AOL;AT&T;Verizon) is probably more than 30% of the list already. And AFAIK not a single one of them uses LDAP - but assume one of them does. So, practically speaking - does it matter? [0] And equivalent for each country with >70M people
- hyc_symas 12y ago
  You're quite wrong here. Google, AT&T and Verizon on your list use OpenLDAP. OpenLDAP scales to billions of users in a single instance, and of course it distributes across multiple nodes for even greater scaling as needed. Most of the telcos around the world use it now; it scales far beyond every other data storage technology out there. http://symas.com/docs/2014-LOADays-jkozyra.pdf http://symas.com/docs/2014-LOADays-jkozyra.pdf 80% of the US population that owns cellphones are OpenLDAP users, whether they realize it or not. 90% of the US population that subscribes to cable TV (Comcast, Time-Warner, etc.) as well. Nothing else actually works for hundreds of millions of users. These companies know, they've tried a lot of "solutions" already.
  - beagle3 12y ago
    Then I'm glad to stand corrected - I suppose I'm horribly out of date from the days they tried other "solutions". I'm less surprised about AT&T and Verizon, more surprised about Google. My own personal experience with LDAP in general (and some with OpenLDAP in particular) dates back to 2000 or so -- and at some point I just gave up on the complex and unreliable beast that it was (for whatever reason, BDB would occasionally lock up or get corrupted) - but I know a lot has changed since, LMDB is amazing -- I guess it is time for another look.
    - hyc_symas 12y ago
      OpenLDAP didn't start to become seriously usable until about 2003 (that's when HP dropped SunDS and adopted OpenLDAP for all of their global infrastructure). A lot of work went into it in the 2001-2002 timeframe to get it there. Yeah, a lot has changed, you've missed a couple generations of advancements. BDB's complexity was definitely a pain, but that's all in the past.
Spooky23 12y ago
I'm pretty sure that with the appropriate infrastructure, The answer is yes. From a scaling limits POV, AD can support billions of objects. Pretty sure the Netscape derivatives (Oracle, the former Sun Directory Server, Red Hat) have implementations that large. I've worked with one with 40M user accounts and we weren't near any scaling limits.

Can existing LDAP implementations scale a single directory to hundreds of millions of users? I can find no clear answer after a few minutes of searching.

by bagels 12y ago

Can existing LDAP implementations scale a single directory to hundreds of millions of users? I can find no clear answer after a few minutes of searching.

viraptor 12y ago

OpenLDAP is limited by LMDB, which is pretty much limited only by what your filesystem and address space can handle. For scaling out it does support multi-master and slave configurations. So yes, it should.

beagle3 12y ago

You can probably list the organizations (outside of China, perhaps) that have over a hundred million of users in a twitter message - (Facebook;Twitter;Google;Walmart;WhatsApp;Instagram;Hotmail;Snapchat;Yahoo;IRS[0];Blizzard;AOL;AT&T;Verizon) is probably more than 30% of the list already. And AFAIK not a single one of them uses LDAP - but assume one of them does. So, practically speaking - does it matter? [0] And equivalent for each country with >70M people

hyc_symas 12y ago
You're quite wrong here. Google, AT&T and Verizon on your list use OpenLDAP. OpenLDAP scales to billions of users in a single instance, and of course it distributes across multiple nodes for even greater scaling as needed. Most of the telcos around the world use it now; it scales far beyond every other data storage technology out there. http://symas.com/docs/2014-LOADays-jkozyra.pdf http://symas.com/docs/2014-LOADays-jkozyra.pdf 80% of the US population that owns cellphones are OpenLDAP users, whether they realize it or not. 90% of the US population that subscribes to cable TV (Comcast, Time-Warner, etc.) as well. Nothing else actually works for hundreds of millions of users. These companies know, they've tried a lot of "solutions" already.
- beagle3 12y ago
  Then I'm glad to stand corrected - I suppose I'm horribly out of date from the days they tried other "solutions". I'm less surprised about AT&T and Verizon, more surprised about Google. My own personal experience with LDAP in general (and some with OpenLDAP in particular) dates back to 2000 or so -- and at some point I just gave up on the complex and unreliable beast that it was (for whatever reason, BDB would occasionally lock up or get corrupted) - but I know a lot has changed since, LMDB is amazing -- I guess it is time for another look.
  - hyc_symas 12y ago
    OpenLDAP didn't start to become seriously usable until about 2003 (that's when HP dropped SunDS and adopted OpenLDAP for all of their global infrastructure). A lot of work went into it in the 2001-2002 timeframe to get it there. Yeah, a lot has changed, you've missed a couple generations of advancements. BDB's complexity was definitely a pain, but that's all in the past.

Spooky23 12y ago

I'm pretty sure that with the appropriate infrastructure, The answer is yes. From a scaling limits POV, AD can support billions of objects. Pretty sure the Netscape derivatives (Oracle, the former Sun Directory Server, Red Hat) have implementations that large. I've worked with one with 40M user accounts and we weren't near any scaling limits.