4 ms·

The semantic web currently scales poorly in two different dimensions. As long as this remains the case, it will see little adoption. First, the software techno

The semantic web currently scales poorly in two different dimensions. As long as this remains the case, it will see little adoption. First, the software technology scales very poorly unless the size of your data model is trivial. The semantic web is a type of graph search problem with all of the hard algorithm and data structure challenges that entails. Most interesting semantic web applications require data models operating at scales where existing graph platforms have pathological performance characteristics. The gap between capability and requirements is several orders of magnitude. You can't throw hardware at it; when I was first hired to do CS research on scaling the semantic web, I was working on literal supercomputers. Second, the human element of the semantic web doesn't scale. Semantics are contextual and subjective. As the number of people contributing to the data model increases, the consistency and quality of the data model decreases. This becomes an extremely expensive global coordination problem at scale that may be unsolvable. It is broadly recognized that you need a single, global arbiter of semantics in any scalable system just to maintain semantic consistency, which basically means strong AI. If we have the technology to solve these two problems, I suspect the Semantic Web will no longer be interesting.

pfraze 6y ago
I think you said it pretty well, but I'll re-iterate some of your second point. The human element defies any solution other than social coordination such as a specs registry or org. At some point the software needs to understand each other and that requires people to work together. A technical solution around schemas can assist the coordination work, but it can't replace it. EDIT to add: Since we can't remove human coordination, I question the value of arbitrary schema composition when it incurs the huge DX cost of using a URL to represent every attribute.
- infinite8s 6y ago
  Alan Kay had a comment on here from a few years ago on how this could scale: https://news.ycombinator.com/item?id=15320789 https://news.ycombinator.com/item?id=15320789
- bigmanwalter 6y ago
  IEML encodes a flexible and upgradeable semantic schema in a git repo where anyone can pull request! https://pierrelevyblog.com/my-research-in-a-nutshell/ https://pierrelevyblog.com/my-research-in-a-nutshell/ https://github.com/IEMLdev/ieml https://github.com/IEMLdev/ieml
zozbot234 6y ago
The "semantic web" is not a special technology, it's just a bunch of standards for data interchange. You can definitely expose highly proven and scalable technology stacks that have nothing to do with silly "graph database" models/platforms, while still using semantic web and Linked Data standards to great effect.
bigmanwalter 6y ago
I follow the IEML project which combines semantic ontologies with folksonomies into a recursive data structure hosted in a Git repo. Interesting ideas, would highly recommend checking it out! https://pierrelevyblog.com/my-research-in-a-nutshell/ https://pierrelevyblog.com/my-research-in-a-nutshell/
samirillian 6y ago
Yes but couldn't you imagine a web and a society that doesn't run on big data? What if the unscalability of the semantic web were a feature rather than a bug? The current political and economic fallout of the big data web that we all know and love arises precisely due to the ungodly effects of network size on power laws. E.g., the rich get richer and the poor get poorer on a global scale. Why not a more semantic and a more decentralized web.
thinkloop 6y ago
The whole world coordinated doing identical SEO. If there was a clear path of proper incentive there may be a chance.

pfraze 6y ago

I think you said it pretty well, but I'll re-iterate some of your second point. The human element defies any solution other than social coordination such as a specs registry or org. At some point the software needs to understand each other and that requires people to work together. A technical solution around schemas can assist the coordination work, but it can't replace it. EDIT to add: Since we can't remove human coordination, I question the value of arbitrary schema composition when it incurs the huge DX cost of using a URL to represent every attribute.

infinite8s 6y ago
Alan Kay had a comment on here from a few years ago on how this could scale: https://news.ycombinator.com/item?id=15320789 https://news.ycombinator.com/item?id=15320789
bigmanwalter 6y ago
IEML encodes a flexible and upgradeable semantic schema in a git repo where anyone can pull request! https://pierrelevyblog.com/my-research-in-a-nutshell/ https://pierrelevyblog.com/my-research-in-a-nutshell/ https://github.com/IEMLdev/ieml https://github.com/IEMLdev/ieml

zozbot234 6y ago

The "semantic web" is not a special technology, it's just a bunch of standards for data interchange. You can definitely expose highly proven and scalable technology stacks that have nothing to do with silly "graph database" models/platforms, while still using semantic web and Linked Data standards to great effect.

bigmanwalter 6y ago

I follow the IEML project which combines semantic ontologies with folksonomies into a recursive data structure hosted in a Git repo. Interesting ideas, would highly recommend checking it out! https://pierrelevyblog.com/my-research-in-a-nutshell/ https://pierrelevyblog.com/my-research-in-a-nutshell/

samirillian 6y ago

Yes but couldn't you imagine a web and a society that doesn't run on big data? What if the unscalability of the semantic web were a feature rather than a bug? The current political and economic fallout of the big data web that we all know and love arises precisely due to the ungodly effects of network size on power laws. E.g., the rich get richer and the poor get poorer on a global scale. Why not a more semantic and a more decentralized web.

thinkloop 6y ago

The whole world coordinated doing identical SEO. If there was a clear path of proper incentive there may be a chance.