5 ms·

I have spent the last 7 years building ontologies (knowledge bases), small and large. Like with most tech articles, this one is pushing a very specific point of

I have spent the last 7 years building ontologies (knowledge bases), small and large. Like with most tech articles, this one is pushing a very specific point of view that is relevant to megacorporations but certainly not to most companies that would benefit from the deployment of graph-structured knowledge bases. Unless a company has tons of extra cash to burn using technologies like neo4j, Spark, etc., they are better off just building an ontology on top of their existing datastores. The work on the semantic web in the 2000s added a lot of valuable ideas and I see a tendency to downplay these ideas in favor of new technologies. Storing relationships as RDF triples and building indices that are well-suited to your queries will be good enough for the majority of people who need an ontology. I have found that graph-based systems don't model time well. When you are building a knowledge base, you often need to understand how knowledge has changed over time. You may want to relate facts across time or understand what the state of knowledge was within a given period of time. This usually takes extra work. Another thing that I have found these graph-based systems to do poorly is the management of storage classes for different types of data. If you are building a graph knowledge base, not all the information in your knowledge base will be consumed in the same way. If your knowledge base is sufficiently large, you will want to store less frequently accessed data on cheaper storage (e.g. HDD or even object storage like S3) and hotter data on more expensive storage (SSD or even maybe keep it in memory). I have always had to write this code by myself and haven't found a tool that helps to manage this very importance concern in ontology building. That turned into a bit of a ramble, but damn ontologies are cool. :)

amw-zero 5y ago
I am honestly intrigued any time I hear anyone mention RDF. You have had success using it? It wasn't exactly clear how you used it from what you said.
- zomglings 5y ago
  I only use RDF triples as a storage format for edges. I don't normally take advantage of triple stores (e.g. apache Jena), sparql query engines, etc. It's just a simple format and it's easy to build different types of query/analysis capabilities around. The only thing that's a little awkward is applying types to items in an rdf triple (in a way that's computationally cheap). Typing is a general problem, though.
  - LukeEF 5y ago
    I strongly agree with this - RDF is fine and there are some great concepts in the SemWeb world (IRIs for example), but OWL and SPARQL are confusing and can be off-putting, especially when it is all rolled together. The latest iteration of our triple store actually moves radically towards linked-document or a 'document graph' to make it more understandable while keeping RDF at the base.
    - rambojazz 5y ago
      What is your triple store?
  - pbourke 5y ago
    Can you express graph attributes such as edge weight in RDF? I am familiar with RDF triples such as "Book", author, "Jane Doe". How would you express "10 minute phone call at time t between persons A and B"?
    - zomglings 5y ago
      You could do it, but a triple wouldn't be the best choice of data structure. I would just add a new numerical column called "weight" to each relation in your RDF store: "Harry Potter and the Philosopher's Stone","author","Jane Doe",0.005 "Harry Potter and the Philosopher's Stone","author","J.K. Rowling",1.0 ... You could easily export RDF triples from this if you needed to using rules like "The author for each book is determined by the '<book>-author-<name>' relation of highest weight". Edit: Sorry I didn't answer your question about "10 minute phone call at time t between person A and person B". The best way I can think of to model this with triples is: "call:<call_id>","started_at",t "call:<call_id>","ended_at",t + <duration> "person A","participant","call:<call_id>" "person B","participant","call:<call_id>" But this may be overly complicated depending on how you want to query the data and what you want to do with it. Edit 2: mhitza already answered it better! https://news.ycombinator.com/item?id=28503097 https://news.ycombinator.com/item?id=28503097
    - mhitza 5y ago
      Not previous commenter, but as with other problems in computing, I find that you need to add a level of indirection. "CallSessionX" -> participant -> "person A" "CallSessionX" -> participant -> "person B" "CallSessionX" -> duration -> "10 minute" "CallSessionX" -> at_time -> t Haven't used Sparql or Cypher in many years (used in Neo4j), but with the later you could do a query along the lines "person A" <- participant <- call_session -> participant -> "person B" To find calls between person A and person B, then you can add a variable binding on the exact node you're interested in (eg call_session) to extract/aggregate attributes.
- FpUser 5y ago
  I've used RDF triples for some specific part of a bigger business data model. It was implemented on a regular database and used to encode things like "Joe Hates Propaganda" which is exactly what was needed and hence was a success (meaning that it had worked as intended).
jcims 5y ago
>The work on the semantic web in the 2000s added a lot of valuable ideas and I see a tendency to downplay these ideas in favor of new technologies. Storing relationships as RDF triples and building indices that are well-suited to your queries will be good enough for the majority of people who need an ontology. Is this basically the idea of using RDF to create explicit joins between datasets in other systems? E.g. each tuple has a foreign key reference to a row/doc that represents that node in the graph? >That turned into a bit of a ramble, but damn ontologies are cool. :) This is a major hurdle in infosec as enterprises adopt cloud products. The traditional ontologies around the risk profiles and state of operating systems and networks have been quite stable over the past 10-20 years and leverage things like DMTF's CIM and IETF's SNMP. However, what I'm finding is that 'cloud' products and services surface entirely new namespaces and concepts in nearly every instance. Sometimes, as in the case of AWS EC2, it's pretty simple to map these back to an existing schema, but in other cases like say AWS Chime or Glue, it gets really tricky. It feels like I'm looking for a system that can rapidly ingest the service's native information model then adapt it with overlays that transform or aggregate various services in a way that allows consistent data modelling. Any suggestions there? :)
- zomglings 5y ago
  re: RDF for joins. That's exactly right. You can keep things as light or as heavy as you need them to be regarding when/how/whether you validate foreign key constraints, whether you use a schema registry, etc. The simplicity of the format makes it really easy to build around. My only suggestion re: infosec ontologies is to not push too much functionality into the ontology all at once. I have had ontology projects end in spectacular failure, and all those failures have resulted from putting too much responsibility on the ontology from the very beginning. The ontology should be as dumb as possible to start with. In your case, the approach I would suggest is: 1. Ontology stores mapping between cloud provider resource schemas to your internal schema. These mappings can be represented in multiple ways - as a URL to a service which implements the mapping or as a JSON object in some DSL which defines the transformations or even as the code/binary that should be run to perform the transformation. 2. From the ontology, it should be easy to query/update the following things: current cloud provider schema for a given resource, internal schema corresponding to a cloud resource, and current mapping to the internal schema, the number of failed transformations from cloud schema to internal schema in the past {1 hour | 1 day | 1 week | ...}. 3. At first, you build the transformations from the cloud provider schemas to internal schema by hand. 4. A worker which alerts you when the cloud provider schemas change (should be easy to build on top of your ontology after step 2 - check the number of failures in transformation). 5. Now write a worker which statically analyzes the terraform code base to see when they changed the code related to a particular cloud API you are working with. Represent this in the ontology (e.g. "AWS Chime" - "terraform:vblah.blah.blah" -> "<json resource definition>" 6. Write a worker which monitors updates to - "terraform:*" edges and propagates alerts you to update the mapping given the new JSON resource definition. 7. For some services, it will be easy to automate the transformation update. For these things, create a new ontology worker. I have found that it's better to let workers run idempotently on cronjobs/systemd timers than to implement event-driven workers. The whole thing will probably take a few months, but you will have production-ready code at the end of each step and start benefitting from the existence of the ontology right away. (Sounds like you are already at step 3 or 4?)
ctvo 5y ago
> If your knowledge base is sufficiently large, you will want to store less frequently accessed data on cheaper storage (e.g. HDD or even object storage like S3) and hotter data on more expensive storage (SSD or even maybe keep it in memory). I have always had to write this code by myself and haven't found a tool that helps to manage this very importance concern in ontology building. This is a general concern for a lot of areas, including observability. Lots of data, need to query it, need to store it cheaply with a time component to help differentiate. Take a look at Thanos and Grafana Labs Cortex. It’s a mostly generalize architecture.
- zomglings 5y ago
  Thank you. Didn't know about Thanos.
leeuw01 5y ago
Any word from the wise for those who want to get into ontologies? Furthermore, do you know about any good resources regarding the process of capturing information into models? I've found books about solutions like Ecore & OWL, but I fail to find books about the practice of modelling in general.
- zomglings 5y ago
  I'm not so wise, my friend, but I do have some tips from hard experience: 1. Most people don't understand the importance of ontologies. It is an uphill battle to convince them. Build your ontologies quietly. They are a superpower. 2. A real ontology must extend far beyond the confines of the data store. You must think of it as also including the workers that bring data into the ontology, the workers which verify the consistency of the ontology, and the query engines through which users get data out of the ontology. 3. Start simple. Your ontologies should be as dumb as possible to start out with but they should support extension. 4. Learn one datastore really well and build your ontologies there. Chances are you don't need to know anything about OWL or neo4j or Spark or anything hyped. I use Postgres. 5. Idempotency is very important. This means, for example, that if you run a data collection worker twice with the same arguments, it should not create duplicate data in your ontology. 6. Build for a specific use case. Don't try to build a general purpose ontology. The kind of data you will be storing and the access patterns will determine a lot about the structure of your ontology and you should be accounting for these things. 7. Avoid event-drive architectures (e.g. IFTTT, AWS Lambda, etc.). Put your faith in idempotence and cron (actually, I use systemd but you get the picture). Do you have an idea what kind of information you would want to build an ontology for?
  - leeuw01 5y ago
    Thanks for the tips! Yes, would you perhaps be up to grab a cup of coffee and have a chat about ontologies? :)
    - zomglings 5y ago
      Definitely. Here is a link to my calendly: https://calendly.com/neeraj-simiotics/bugout-30 https://calendly.com/neeraj-simiotics/bugout-30
btown 5y ago
> better off just building an ontology on top of their existing datastores. I just made a view with three copypasta’d levels of joins on a parent-child relationship table to build up an ancestor_names column to be used in an analytics. The analytics team is probably going to just use regexes or if-statements on that column in Tableau to understand our performance with top-level partner names. If it gets slow I’ll make it a materialized view. This. Is. Fine. You don’t need a graph database or explicit ontology or even recursive queries to answer graph questions. It’s nice to have one, of course, but not necessary.
- zozbot234 5y ago
  Yup, database views are exactly how to do 'ontology-based' inference in SQL, in the rare cases where that makes actual sense. It can do anything that a graph or "semantic" database can do, and with on-par or better performance. Even CTE recursive queries work just as well as a general-purpose graph DB, though the latter sometimes has a more convenient query language.
- zomglings 5y ago
  Totally. You shouldn't be allowing people to run graph traversals of arbitrary depth in your database anyway. :)

amw-zero 5y ago

I am honestly intrigued any time I hear anyone mention RDF. You have had success using it? It wasn't exactly clear how you used it from what you said.

zomglings 5y ago
I only use RDF triples as a storage format for edges. I don't normally take advantage of triple stores (e.g. apache Jena), sparql query engines, etc. It's just a simple format and it's easy to build different types of query/analysis capabilities around. The only thing that's a little awkward is applying types to items in an rdf triple (in a way that's computationally cheap). Typing is a general problem, though.
- LukeEF 5y ago
  I strongly agree with this - RDF is fine and there are some great concepts in the SemWeb world (IRIs for example), but OWL and SPARQL are confusing and can be off-putting, especially when it is all rolled together. The latest iteration of our triple store actually moves radically towards linked-document or a 'document graph' to make it more understandable while keeping RDF at the base.
  - rambojazz 5y ago
    What is your triple store?
- pbourke 5y ago
  Can you express graph attributes such as edge weight in RDF? I am familiar with RDF triples such as "Book", author, "Jane Doe". How would you express "10 minute phone call at time t between persons A and B"?
  - zomglings 5y ago
    You could do it, but a triple wouldn't be the best choice of data structure. I would just add a new numerical column called "weight" to each relation in your RDF store: "Harry Potter and the Philosopher's Stone","author","Jane Doe",0.005 "Harry Potter and the Philosopher's Stone","author","J.K. Rowling",1.0 ... You could easily export RDF triples from this if you needed to using rules like "The author for each book is determined by the '<book>-author-<name>' relation of highest weight". Edit: Sorry I didn't answer your question about "10 minute phone call at time t between person A and person B". The best way I can think of to model this with triples is: "call:<call_id>","started_at",t "call:<call_id>","ended_at",t + <duration> "person A","participant","call:<call_id>" "person B","participant","call:<call_id>" But this may be overly complicated depending on how you want to query the data and what you want to do with it. Edit 2: mhitza already answered it better! https://news.ycombinator.com/item?id=28503097 https://news.ycombinator.com/item?id=28503097
  - mhitza 5y ago
    Not previous commenter, but as with other problems in computing, I find that you need to add a level of indirection. "CallSessionX" -> participant -> "person A" "CallSessionX" -> participant -> "person B" "CallSessionX" -> duration -> "10 minute" "CallSessionX" -> at_time -> t Haven't used Sparql or Cypher in many years (used in Neo4j), but with the later you could do a query along the lines "person A" <- participant <- call_session -> participant -> "person B" To find calls between person A and person B, then you can add a variable binding on the exact node you're interested in (eg call_session) to extract/aggregate attributes.
FpUser 5y ago
I've used RDF triples for some specific part of a bigger business data model. It was implemented on a regular database and used to encode things like "Joe Hates Propaganda" which is exactly what was needed and hence was a success (meaning that it had worked as intended).

jcims 5y ago

>The work on the semantic web in the 2000s added a lot of valuable ideas and I see a tendency to downplay these ideas in favor of new technologies. Storing relationships as RDF triples and building indices that are well-suited to your queries will be good enough for the majority of people who need an ontology. Is this basically the idea of using RDF to create explicit joins between datasets in other systems? E.g. each tuple has a foreign key reference to a row/doc that represents that node in the graph? >That turned into a bit of a ramble, but damn ontologies are cool. :) This is a major hurdle in infosec as enterprises adopt cloud products. The traditional ontologies around the risk profiles and state of operating systems and networks have been quite stable over the past 10-20 years and leverage things like DMTF's CIM and IETF's SNMP. However, what I'm finding is that 'cloud' products and services surface entirely new namespaces and concepts in nearly every instance. Sometimes, as in the case of AWS EC2, it's pretty simple to map these back to an existing schema, but in other cases like say AWS Chime or Glue, it gets really tricky. It feels like I'm looking for a system that can rapidly ingest the service's native information model then adapt it with overlays that transform or aggregate various services in a way that allows consistent data modelling. Any suggestions there? :)

zomglings 5y ago
re: RDF for joins. That's exactly right. You can keep things as light or as heavy as you need them to be regarding when/how/whether you validate foreign key constraints, whether you use a schema registry, etc. The simplicity of the format makes it really easy to build around. My only suggestion re: infosec ontologies is to not push too much functionality into the ontology all at once. I have had ontology projects end in spectacular failure, and all those failures have resulted from putting too much responsibility on the ontology from the very beginning. The ontology should be as dumb as possible to start with. In your case, the approach I would suggest is: 1. Ontology stores mapping between cloud provider resource schemas to your internal schema. These mappings can be represented in multiple ways - as a URL to a service which implements the mapping or as a JSON object in some DSL which defines the transformations or even as the code/binary that should be run to perform the transformation. 2. From the ontology, it should be easy to query/update the following things: current cloud provider schema for a given resource, internal schema corresponding to a cloud resource, and current mapping to the internal schema, the number of failed transformations from cloud schema to internal schema in the past {1 hour | 1 day | 1 week | ...}. 3. At first, you build the transformations from the cloud provider schemas to internal schema by hand. 4. A worker which alerts you when the cloud provider schemas change (should be easy to build on top of your ontology after step 2 - check the number of failures in transformation). 5. Now write a worker which statically analyzes the terraform code base to see when they changed the code related to a particular cloud API you are working with. Represent this in the ontology (e.g. "AWS Chime" - "terraform:vblah.blah.blah" -> "<json resource definition>" 6. Write a worker which monitors updates to - "terraform:*" edges and propagates alerts you to update the mapping given the new JSON resource definition. 7. For some services, it will be easy to automate the transformation update. For these things, create a new ontology worker. I have found that it's better to let workers run idempotently on cronjobs/systemd timers than to implement event-driven workers. The whole thing will probably take a few months, but you will have production-ready code at the end of each step and start benefitting from the existence of the ontology right away. (Sounds like you are already at step 3 or 4?)

ctvo 5y ago

> If your knowledge base is sufficiently large, you will want to store less frequently accessed data on cheaper storage (e.g. HDD or even object storage like S3) and hotter data on more expensive storage (SSD or even maybe keep it in memory). I have always had to write this code by myself and haven't found a tool that helps to manage this very importance concern in ontology building. This is a general concern for a lot of areas, including observability. Lots of data, need to query it, need to store it cheaply with a time component to help differentiate. Take a look at Thanos and Grafana Labs Cortex. It’s a mostly generalize architecture.

zomglings 5y ago
Thank you. Didn't know about Thanos.

leeuw01 5y ago

Any word from the wise for those who want to get into ontologies? Furthermore, do you know about any good resources regarding the process of capturing information into models? I've found books about solutions like Ecore & OWL, but I fail to find books about the practice of modelling in general.

zomglings 5y ago
I'm not so wise, my friend, but I do have some tips from hard experience: 1. Most people don't understand the importance of ontologies. It is an uphill battle to convince them. Build your ontologies quietly. They are a superpower. 2. A real ontology must extend far beyond the confines of the data store. You must think of it as also including the workers that bring data into the ontology, the workers which verify the consistency of the ontology, and the query engines through which users get data out of the ontology. 3. Start simple. Your ontologies should be as dumb as possible to start out with but they should support extension. 4. Learn one datastore really well and build your ontologies there. Chances are you don't need to know anything about OWL or neo4j or Spark or anything hyped. I use Postgres. 5. Idempotency is very important. This means, for example, that if you run a data collection worker twice with the same arguments, it should not create duplicate data in your ontology. 6. Build for a specific use case. Don't try to build a general purpose ontology. The kind of data you will be storing and the access patterns will determine a lot about the structure of your ontology and you should be accounting for these things. 7. Avoid event-drive architectures (e.g. IFTTT, AWS Lambda, etc.). Put your faith in idempotence and cron (actually, I use systemd but you get the picture). Do you have an idea what kind of information you would want to build an ontology for?
- leeuw01 5y ago
  Thanks for the tips! Yes, would you perhaps be up to grab a cup of coffee and have a chat about ontologies? :)
  - zomglings 5y ago
    Definitely. Here is a link to my calendly: https://calendly.com/neeraj-simiotics/bugout-30 https://calendly.com/neeraj-simiotics/bugout-30

btown 5y ago

> better off just building an ontology on top of their existing datastores. I just made a view with three copypasta’d levels of joins on a parent-child relationship table to build up an ancestor_names column to be used in an analytics. The analytics team is probably going to just use regexes or if-statements on that column in Tableau to understand our performance with top-level partner names. If it gets slow I’ll make it a materialized view. This. Is. Fine. You don’t need a graph database or explicit ontology or even recursive queries to answer graph questions. It’s nice to have one, of course, but not necessary.

zozbot234 5y ago
Yup, database views are exactly how to do 'ontology-based' inference in SQL, in the rare cases where that makes actual sense. It can do anything that a graph or "semantic" database can do, and with on-par or better performance. Even CTE recursive queries work just as well as a general-purpose graph DB, though the latter sometimes has a more convenient query language.
zomglings 5y ago
Totally. You shouldn't be allowing people to run graph traversals of arbitrary depth in your database anyway. :)