publications

2022

1. ISWC

   Visual Relationship Detection using Knowledge Graphs for Neural-Symbolic AI

   David Herron

   In Proceedings of the Doctoral Consortium at ISWC 2022

   Abs Bib HTML

   Momentum is surging behind the consensus that neural-symbolic AI is the right road for AI to take today. We propose to travel this road using Semantic Web technologies to represent the symbolic AI tradition. Our objective is to investigate and compare the efficacy of a variety of strategies for combining the capabilities of deep neural networks for statistical learning from data with those of OWL ontologies and knowledge graphs for symbolic knowledge representation and reasoning. Our application area is visual relationship detection within images. Deep learning is data hungry and struggles to generalise to examples outside the training distribution. We seek to show that combining Semantic Web prior knowledge and reasoning with deep learning can deliver superior performance, can substitute for plentiful training data, and can deliver robust generalisation in few-shot/zero-shot learning scenarios.

   @inproceedings{herron2022,
     author = {Herron, David},
     title = {Visual {R}elationship {D}etection using {K}nowledge {G}raphs for {N}eural-{S}ymbolic {AI}},
     booktitle = {Proceedings of the Doctoral Consortium at ISWC},
     volume = {3165},
     year = {2022},
     publisher = {CEUR Workshop Proceedings},
   }

2023

1. arXiv

   NeSy4VRD: A Multifaceted Resource for Neurosymbolic AI Research using Knowledge Graphs in Visual Relationship Detection

   David Herron, Ernesto Jiménez-Ruiz, Giacomo Tarroni, and Tillman Weyde

   CoRR 2023

   Abs Bib HTML

   NeSy4VRD is a multifaceted resource designed to support the development of neurosymbolic AI (NeSy) research. NeSy4VRD re-establishes public access to the images of the VRD dataset and couples them with an extensively revised, quality-improved version of the VRD visual relationship annotations. Crucially, NeSy4VRD provides a well-aligned, companion OWL ontology that describes the dataset domain. It comes with open source infrastructure that provides comprehensive support for extensibility of the annotations (which, in turn, facilitates extensibility of the ontology), and open source code for loading the annotations to/from a knowledge graph. We are contributing NeSy4VRD to the computer vision, NeSy and Semantic Web communities to help foster more NeSy research using OWL-based knowledge graphs.

   @article{Herron/2023/nesy4vrd,
     author = {Herron, David and Jim{\'{e}}nez-Ruiz, Ernesto and Tarroni, Giacomo and Weyde, Tillman},
     title = {{NeSy4VRD: A Multifaceted Resource for Neurosymbolic AI Research using Knowledge Graphs in Visual Relationship Detection}},
     journal = {CoRR},
     year = {2023},
   }

2. CEUR

   On the Benefits of OWL-based Knowledge Graphs for Neural-Symbolic Systems

   David Herron, Ernesto Jiménez-Ruiz, and Tillman Weyde

   In NeSy 2023: 17th International Workshop on Neural-Symbolic Learning and Reasoning, July 3–5, La Certosa di Pontignano, Siena, Italy 2023

   Abs Bib HTML

   Knowledge graphs, as understood within the Semantic Web and Knowledge Representation communities, are more than just graph data. OWL-based knowledge graphs offer the benefits of being based on an ecosystem of open W3C standards that are implemented in a range of reusable existing resources (e.g. curated ontologies, software tools, web-wide linked data) and that also permit researchers to tailor resources for their unique needs (e.g. custom ontologies). Additionally, OWL-based knowledge graphs offer the benefits of formal, logical symbolic reasoning (e.g. reliable inference of new knowledge based on Description Logics, semantic consistency checking, extensions via user-defined Datalog rules). These capabilities allow OWL-based knowledge graphs to be leveraged in the form of active reasoning agents to guide deep learning during training and to participate in refining neural inference. We enumerate a host of such benefits to using OWL-based knowledge graphs in neural-symbolic systems. We illustrate several of these by drawing upon examples from our research in visual relationship detection within images, and we point to promising research directions and challenging opportunities.

   @inproceedings{Herron/NeSy2023,
     author = {Herron, David and Jim{\'{e}}nez-Ruiz, Ernesto and Weyde, Tillman},
     title = {{On the Benefits of OWL-based Knowledge Graphs for Neural-Symbolic Systems}},
     booktitle = {NeSy 2023: 17th International Workshop on Neural-Symbolic Learning and Reasoning, July 3--5, La Certosa di Pontignano, Siena, Italy},
     series = {{CEUR} Workshop Proceedings},
     volume = {3432},
     publisher = {CEUR-WS.org},
     year = {2023},
   }