A Simple Baseline for Inductive Knowledge Base Completion

This page describes the work done with AnyBURL to evaluate benchmark datasets for inductive knowledge base completion. As such, this webpage mostly covers the contents of the KR-26 paper "A Simple Baseline for Inductive Knowledge Base Completion" by Christian Meilicke, Rainer Gemulla, Julie Naegelen, and Heiner Stuckenschmidt. It also offers some supplementary ablation results, the results of our baseline on newer benchmarks suggested by Shomer et al. 2025, as well as quick instruction on how to run our baseline on these datasets.

TL;DR: We find that the process by which current IKGC benchmark datasets are created from their transductive successors introduces systematic biases into the graph structure. We also find that a model able to exploit these biases will be able to obtain a relatively high performance without performing any meaningful relational reasoning. Moroever, we build a minimal baseline which does not perform reasoning beyond 1-hop neighbourhood lookup and find it performs on-par with the current SOTA IKGC model, ULTRA.

• What is IKGC
• Flaws to Existing IKGC Benchmarks
• Our Baseline
• Main Results
• Ablation Results
• How to Run It
• Contact
• License
• References

The Shortest Introduction to IKGC

Knowledge graph completion (KGC) in its most common form aims to answer queries such about connections between entities such as married(John, ?) for a given graph. This query is interesting because while John and his potential marital partner might already be in the dataset, their (existing!) marriage (i.e. the link between them) is not. In short, we assume that the data in the KG is incomplete. Moreoever, we assume that by reasoning over present edges, we can add true but missing ones to the graph.

If this link prediction happens between entities that entities that we saw during training, it is called transductive. If we are to perform link prediction over a graph made up of new entities, it is called inductive.

The latter task of inductive knowledge graph completion (IKGC) is usually motivated either by dynamic KGs where new entities or relations are added to a graph over time, or in the context of foundation models. In the latter case we wish to pretrain a powerful model once, and then apply it on a multitude of new KGs out of the box. Practically speaking, IKGC differs from standard, transductive, KGC in that the entities (and sometimes relations) seen in training must be strictly disjoint from the ones seen at inference time on the target graph. There exist a number of well-established IKGC benchmark datasets, created from original transductive KGC datasets.

Biases in IKGC Benchmarks

The current main benchmark datasets for IKGC are created from their corresponding transductive benchmark by randomly sampling seed entities from the original full graph, then taking their k-hop neighbourhoods for two hops as training graph (snowball sampling). The induced induced k-hop neighbourhood subgraph is removed from the original graph. Then the process is repeated to obtain one or several inference graphs. Entity-and-relation-inductive benchmark datasets are obtained in a similar fashion. Shomer et al. 2025 describe this process in more detail.

Shomer et al. 2025 additionally find that using Personalised Page Rank agnostic of query relation type is enough to obtain IKGC results close to the SOTA on the resulting benchmark datasets. They suggest that this is due to the pipeline above causing entities in negative training samples to be much farther away from each other than in the original transductive baselines. They propose alternative IKGC benchmark datasets created from the original KGs via graph partitioning algorithms.

In our paper we find that the IKGC benchmark dataset pipeline additionally induces skewed entity distributions per relation type. For example, while in the original FB15k-237 dataset the relation profession was rather evenly distributed (abstracting from the 20.7% of entities whose profession is 'actor'), in the inductive splits up to 80% of all examples are covered by only three professions. This means that while in the original graph guessing a person's profession couldn't be done without performing some relational reasoning, in its inductive variants, just returning the most 'popular' entity can give an MRR of up to 84.8%. We argue that this problem is inherently introduced when applying the strict disjointness condition to an existing KG.

A Simple Baseline

To investigate whether good performance on current IKGC benchmark datasets implies complex relational reasoning, we purposefully set up a very limited baseline. Our simple baseline consists of running rule-miner AnyBURL in a restricted setting: we only allow Anyburl to mine rules with length two (i.e. of the form If I am married to John, he is married to me or If a company is based in the US, the company pays taxes there). Additionally, we allow AnyBurl to access the frequency distribution of the target graph. This allows it to exploit the frequency skew from above with a 'popularity rule'. As an example, if the main profession in a given target graph is 'singer', then a query of the form profession(John, ?) will always return 'singer', no matter the context we actually encounter the entity John in the inference graph.

Main Results

We find that our baseline performs very similar to the current SOTA zero-shot model ULTRA, even when we allow ULTRA to perform finetuning on the data domain. We also find that using the alternative inductive splits obtained via graph partitioning does not result in significantly more challenging benchmarks. Lastly, the amount of explanatory power provided by the "popularity rule" alone varies among benchmarks, but it is never insignificant. See the tables below for all results.

Ablation Results: New Benchmarks Through Graph Partitioning (Shomer et al. 2025)

Ablation Results: Popularity Rule Only

How to Run It

Within the file suppl-mat.zip you find the code/commands and (links to) all datasets that you need to generate the baseline results presented in the paper "A Simple Baseline for Inductive Knowledge Base Completion". Donwload the file and unzip it. There will be a folder suppl, which contains all relevant files.

To run AnyBURL you need to have Java installed on our device. Any version >=12 should work properly (e.g., openjdk12). Type java -version to check your installation.

Results for FB237

We prepared everything to generate the results for the splits of the FB237 datasets from [1]. At the end of this section we explain where to get the other datasets and how to generate results for them.

In each of the experiments, we first need to learn the rules sets. Make the suppl folder the working directory and run AnyBURL with this command:

java -Xmx12G -cp AnyBURL-23-1x.jar de.unima.ki.anyburl.Learn exp/fb237/config-learn.properties

This will run AnyBURL for 30 seconds in the restricted setting that corresponds to the baseline. You can take a look at the config file to see the parameters that result in this setting. The command generates the rule file rules-v1-30 in the exp/fb237 folder. Now you can use these rules to create a ranking file. Use this command:

java -Xmx12G -cp AnyBURL-23-1x.jar de.unima.ki.anyburl.Apply exp/fb237/config-apply.properties 

The resulting ranking file can be evaluated with this command:

java -Xmx12G -cp AnyBURL-23-1x.jar de.unima.ki.anyburl.Eval exp/fb237/config-eval.properties

This generates the following command line output. The first value is the hits@1 score, the second value is the hits@3 score, the third value is the hits@10 score and the last value is the MRR score, which we show in the results tables of our paper.

...
* loading result set at exp/fb237/preds-v1-30
0.4000   0.5341   0.5854   0.4720

There might a slight difference compared to the MRR number shown in the FBe v1 column (which is 46.7 in the paper) , which might be caused by the sampling done within AnyBURL. Differences are usually very small, exception might be the v1 splits of the datasets which are very small in number of test triples.

If you want to generate the results for the v2, v3, and v4 splits of this dataset, you have to change the config-files and rerun the commands above.

All Datasets

The other datasets can be obtained from:

• Inductive:

  • WNe, FBe, NELLe used in [1] are available at: https://github.com/kkteru/grail/tree/master/data
  • ILPC small and large used in [2] are available at: https://github.com/pykeen/ilpc2022/tree/main/data
  • The entity- and relation inductive versions used in [3] are available at: https://github.com/bdi-lab/InGram/tree/main/data
  • The data splits proposed by [4] are available at: https://github.com/HarryShomer/Better-Inductive-KGC/tree/master/new_data

• Transductive:

  • A back-up version of the NELL-995 dataset is available at: https://github.com/tmlr-group/one-shot-subgraph/tree/master/data/nell
  • The original version of FB15k-237 is available at: https://web.informatik.uni-mannheim.de/pi1/kge-datasets/fb15k-237.tar.gz
  • The original WN18RR dataset is available at: https://web.informatik.uni-mannheim.de/pi1/kge-datasets/wnrr.tar.gz
  • The original Codex-m and -l datasets are available at: https://zenodo.org/record/4281094/files/codex-m.tar.gz, https://zenodo.org/record/4281094/files/codex-l.tar.gz

The file names of the target graph differ between these datasets. In [1] it is the train.txt file in the _ind folders. In [2] this file is called inference.txt. In [3] it is the msg.txt file. Once the datasets are downloaded, the config files can be copied and easily adapted to generate the baseline results for all of the datasets. You have to add subfolders manually in the exp folder. [TODO: how to run on the [4] datasets from Shomer et al.]

Contact

If you have any questions, feel free to write an email at any time. We are also very interested if you happened to use our simple baseline!

• Christian Meilicke [christian AT informatik.uni-mannheim.de], University Mannheim, Data and Web Science Group

License

AnyBURL is available under the 3-clause BSD, sometimes referred to as modified BSD license:

Copyright (c) University Mannheim, Data and Web Science Group

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
Neither the name of the University nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS AS IS AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

References

[1] Teru, K.; Denis, E.; and Hamilton, W. 2020. Inductive relation prediction by subgraph reasoning. In International conference on machine learning, PMLR.

[2] Galkin, M.; Berrendorf, M.; and Hoyt, C. T. 2022. An open challenge for inductive link prediction on knowledge graphs. arXiv preprint arXiv:2203.01520.

[3] Lee, J.; Chung, C.; and Whang, J. J. 2023. Ingram: Inductive knowledge graph embedding via relation graphs. In International conference on machine learning, PMLR.

[4] Shomer, H; Revolinsky, J.; and Tang, J. 2025. Towards Better Benchmark Datasets for Inductive Knowledge Graph Completion. In Proceedings of the 32st ACM SIGKDD Conference on Knowledge Discovery and Data Mining, ACM.