‘The Queen of England is not England’s Queen.’ (EACL’24)

When humans know “Rome is the capital of Italy”, they also know “The capital of Italy is Rome”.

This means, if humans know a fact, they can be either queried for the subject or the object of the relation and retrieve the knowledge. We would expect pre-trained language models (PLMS) to also be able to do this. But, can they?

The (Youssef et al., 2024) investigates the coherency of factual knowledge within pre-trained language models (PLMs). Highlighting a gap in PLMs’ ability to accurately predict related facts in reverse, it points to a need for improved training methods.

The research emphasizes the potential of retrieval-based approaches, which significantly enhance factual coherency, aiming to make PLMs more reliable sources of factual information. Additionally, this work calls for developing pre-training objectives which explicitly optimize PLMs for more coherent knowledge states.

Reference

1. Paul Youssef, Jörg Schlötterer, and Christin Seifert. The Queen of England is not England’s Queen: On the Lack of Factual Coherency in PLMs. Findings of the Association for Computational Linguistics: EACL 2024. 2024.
   BibTeX

   @inproceedings{Youssef2024_eacl_factual-coherency-PLMs,
     author = {Youssef, Paul and Schl{\"o}tterer, J{\"o}rg and Seifert, Christin},
     booktitle = {Findings of the Association for Computational Linguistics: EACL 2024},
     title = {The Queen of {E}ngland is not {E}ngland{'}s Queen: On the Lack of Factual Coherency in {PLM}s},
     year = {2024},
     address = {St. Julian{'}s, Malta},
     editor = {Graham, Yvette and Purver, Matthew},
     month = mar,
     pages = {2342--2354},
     publisher = {Association for Computational Linguistics},
     code = {https://github.com/paulyoussef/coherency},
     url = {https://aclanthology.org/2024.findings-eacl.155}
   }