Fact Retrieval from LLMs (EMNLP’23)

How do we know what Language Models know? And what are obstacles in using them as knowledge bases? In our recent paper presented at EMNLP, we survey methods and datasets for probing PLMs along a categorization scheme.

We identify three main prevalent obstacles for using PLMs as knowledge bases:

• PLMs’ sensitivity to input queries results in inconsistent fact retrieval.
• Understanding where facts are stored and how they are retrieved is necessary for trustworthy applications.
• Methods for reliably updating knowledge and/or enhancing facts with time-frames.

Reference

(Youssef et al., 2023)

1. Paul Youssef, Osman Koraş, Meijie Li, Jörg Schlötterer, and Christin Seifert. Give Me the Facts! A Survey on Factual Knowledge Probing in Pre-trained Language Models. Findings of the Association for Computational Linguistics: EMNLP 2023. 2023.
   BibTeX

   @inproceedings{Youssef2023_emnlp_survey-probing,
     author = {Youssef, Paul and Kora{\c{s}}, Osman and Li, Meijie and Schl{\"o}tterer, J{\"o}rg and Seifert, Christin},
     booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2023},
     title = {Give Me the Facts! A Survey on Factual Knowledge Probing in Pre-trained Language Models},
     year = {2023},
     address = {Singapore},
     editor = {Bouamor, Houda and Pino, Juan and Bali, Kalika},
     month = dec,
     pages = {15588--15605},
     publisher = {ACL},
     doi = {10.18653/v1/2023.findings-emnlp.1043},
     file = {:own-pdf/Youssef2023_emnlp_survey-probing_preprint.pdf:PDF},
     url = {https://aclanthology.org/2023.findings-emnlp.1043}
   }