What is KG4LLM?
Research, develop, and demonstrate methods for integrating and enriching LLMs through the injection of structured knowledge from Knowledge Graphs and access to symbolic reasoning and information retrieval services, with the ultimate goal of addressing the existing barriers hindering the widespread adoption of LLMs, including issues of factuality, bias, reliability, adaptability to specific domains, model size, and support for additional languages, particularly Spanish and co-official languages.
Multilingualism and Domain Adaptation
The project's goal is the adaptation and injection of domain knowledge into LLMs in the four co-official languages of the Spanish State (Spanish, Basque, Catalan, and Galician).
To choose which vertical domains this injection will take place in, a study was conducted following various criteria regarding the availability of linguistic resources for these domains: text corpora, verification databases with factual data, structured semantic resources, and evaluation datasets.
Following these criteria, the following domains were chosen: Health, Legal, Patents, and Insurance.
Evaluation and Improvement of Factuality
We propose a general framework for factuality evaluation with four fundamental phases, each of which will evaluate the model's factuality in a different and complementary domain.
Our goal is to increase the percentage of correct facts in the text generated by the model for the target domain and language pairs and to decrease the proportion of incorrect facts. Our proposal is based on the DPO algorithm to align a given LLM with a factuality target, relying on confidence-based estimators and references (when possible) to generate a preference dataset agnostically to domain and language.
We consider the use of knowledge-based model templates, as used for diagnostic or evaluation tasks, to include relevant information that has not been encoded in the LLMs.
Evaluation and Improvement of Bias
In addition to factuality, it is important to evaluate and mitigate bias, as models can reproduce social biases that exist in the data.
Therefore, our methodology will include an analysis of bias in the results of the generated texts using metrics such as imprecision, incoherence, stereotypes, etc. This will allow us to monitor the performance of the model and its adaptation to the languages and domains involved.
The results of this analysis will be used to retrain the LLMs to decrease the identified bias levels.
Use of External Tools
The LLMs will be complemented with other external tools for verification and information retrieval, which will be integrated into a coherent framework to allow external factual verification.
The application of these tools will be essential for producing a more reliable output for users, especially when dealing with sensitive topics that require a high level of accuracy and objectivity, such as legal or medical matters.
Generation of Knowledge Graphs
As part of the knowledge representation, we will create Knowledge Graphs to represent the domain knowledge of the selected verticals.
The main goal of these Knowledge Graphs is to facilitate information retrieval and enhance reasoning capabilities in the domain.
These will also be generated based on the previously mentioned linguistic resources, enabling the incorporation of the most relevant knowledge for each of the domains targeted by the project.
Deliverables
You can download the deliverables from the project down below:
State-of-the-art document, guides for knowledge injection into LLMs, and metrics.
Knowledge injection methods for LLMs.
Knowledge injection methods for LLMs.
Methods for Using External Tools by LLM.
Methods for Using External Tools by LLM.
Analysis and definition of application domains and use cases.
Domain-Adapted LLMs.
Methods for Extracting Knowledge Graphs from Text Using LLMs.
Evaluation framework and associated resources.
Demo Design
Project website.
Work plan and quality plan.