Most of the core platforms these companies use are built on open source. I think the fundamental reason there isn't something comparable in the free and open ar

Most of the core platforms these companies use are built on open source. I think the fundamental reason there isn't something comparable in the free and open arena is because at the end of the day to coral the data volumes you will find your self well on your way down the road to hell. There is a minimum physical hardware cost in the multi-million dollar range just to get started. Not to mention the labor cost to operate and architect said hardware. Any thing in the Petabyte scale is looking at a minimum skeleton crew of (if they are top talent, more people if they aren't top talent) maybe 3-4 people between sysadmin architect and data engineers to keep things production worthy. The industry is still light on people with those skills so ontop of the bare hardware you are probably talking about 600 to 800 K in annual labor cost as a starting point. This would all be before you get that much usage, the minute that utilization becomes high that skeleton crew won't cut it any more. The SysAdmin will need to turn into an on-call ops team. Data Engineering will need specialists for on-boarding vs analytics vs access layer. If you are doing anything controversial don't forget the importance of a solid security organization which is highly challenging in the distributed computing space. These are just the technical hurdles. Depending on the data you would be bringing in there are quite possibly legal barriers as well depending on the locality. So again its not so much that this isn't possible, more that there hasn't been someone willing to endow the kind of funding that would be required to scale something of this nature solely for the benefit of the public good.