xAI Lab, University of Marburg, Germany

Transparency in Bayesian Optimization

2026-05-05T00:00:00+00:00

On May 5th, Tanmay Chakraborty successfully defended his doctoral thesis on explainability and human–algorithm collaboration in Bayesian Optimization. His research addresses a key challenge in industrial optimization: while Bayesian Optimization is highly data-efficient for complex black-box systems, its lack of transparency often limits practical adoption. The thesis introduces novel explanation methods designed specifically for Bayesian, including TNTRules, and MOLONE, which help experts understand optimization recommendations, parameter interactions, uncertainty, and trade-offs during decision-making. The work further demonstrates through user studies that explanations significantly improve task performance, trust, and understanding without increasing cognitive load, highlighting the importance of explainability for effective human–AI collaboration in real-world optimization workflows. Tanmay even managed to hide eggs in his thesis — though not very subtly. Congrats on this eggselent achievement! Special thanks to Christian Wirth from Aumovio for his supervision and guidance. Key Publications (Chakraborty et al., 2025) (Chakraborty et al., 2025) (Chakraborty et al., 2025) Tanmay Chakraborty, Marion Koelle, Jörg Schlötterer, Nadine Schlicker, Christian Wirth, and Christin Seifert. Explanation Format does not Matter; but Explanations do - An Eggsbert study on explaining Bayesian Optimization tasks. Information Systems Frontiers. 2025. BibTeX @article{Chakraborty2025_isf_eggsbert, author = {Chakraborty, Tanmay and Koelle, Marion and Schlötterer, Jörg and Schlicker, Nadine and Wirth, Christian and Seifert, Christin}, journal = {Information Systems Frontiers}, title = {{Explanation Format does not Matter; but Explanations do - An Eggsbert study on explaining Bayesian Optimization tasks}}, year = {2025}, month = dec, doi = {10.1007/s10796-025-10671-6}, file = {:own-pdf/Chakraborty2025_isf_eggsbert_preprint.pdf:PDF}, publisher = {Springer Nature}, url = {https://link.springer.com/10.1007/s10796-025-10671-6} } Tanmay Chakraborty, Christian Wirth, and Christin Seifert. Comparative Explanations: Explanation Guided Decision Making for Human-in-the-Loop Preference Selection. Explainable Artificial Intelligence. 2025. BibTeX @inproceedings{Chakraborty2025_xaiw_comparitive-explanation, author = {Chakraborty, Tanmay and Wirth, Christian and Seifert, Christin}, booktitle = {Explainable Artificial Intelligence}, title = {Comparative Explanations: Explanation Guided Decision Making for Human-in-the-Loop Preference Selection}, year = {2025}, address = {Cham}, editor = {Guidotti, Riccardo and Schmid, Ute and Longo, Luca}, pages = {139--161}, publisher = {Springer Nature Switzerland}, doi = {10.1007/978-3-032-08317-3_7} } Tanmay Chakraborty, Christian Wirth, and Christin Seifert. Explainable Bayesian Optimization. Explainable Artificial Intelligence. 2025. BibTeX @inproceedings{Chakraborty2025_xaiw_explainable-bayesian-optimization, author = {Chakraborty, Tanmay and Wirth, Christian and Seifert, Christin}, booktitle = {Explainable Artificial Intelligence}, title = {Explainable Bayesian Optimization}, year = {2025}, address = {Cham}, editor = {Guidotti, Riccardo and Schmid, Ute and Longo, Luca}, pages = {53--77}, publisher = {Springer Nature Switzerland}, doi = {10.1007/978-3-032-08324-1_3} }

Tracing and Reversing Edits in LLMs (ICLR’26)

2026-04-23T00:00:00+00:00

Knowledge editing is used to update large language models when facts change. Instead of retraining an entire model, editing methods can directly modify the model’s parameters so that it responds with new information.

But what happens when such edits are unwanted, hidden, or malicious?

In this paper, we study how knowledge edits can be detected and undone. We introduce two tasks: tracing edits, where the goal is to recover which fact was inserted into a model from its edited weights, and reversing edits, where the goal is to restore the model’s original behavior.

Our results show that edits leave recognizable traces in model weights. By analyzing low-rank approximations of the parameter update, we can often identify the edited object and reverse the change while preserving much of the model’s original performance. This provides a step toward safer and more accountable use of knowledge editing in LLMs.

Paper and video

References

(Youssef et al., 2026)

Paul Youssef, Zhixue Zhao, Christin Seifert, and Jörg Schlötterer. Tracing and Reversing Edits in LLMs. International Conference on Learning Representations (ICLR). 2026.

BibTeX

Content Selection in Text Summarization and Simplification

2026-01-05T00:00:00+00:00

On December 17th, Jan Trienes successfully defended his PhD thesis. Over the past years, he investigated content selection in large language models (LLMs). While NLP tasks such as automatic text summarization and simplification can now be solved very effectively, the black-box nature of LLMs makes it difficult to understand and control how they select content. Key questions addressed in the thesis include: what information LLMs choose to keep or omit, how their strategies differ from those of human experts, and how content selection can be adapted to specialized domains such as clinical text. Since summarization and simplification inevitably involve information loss, an additional challenge lies in enabling readers to recover omitted information when needed. Jan’s thesis addresses these challenges by studying content selection behavior in LLMs and developing methods to make it more transparent, interpretable, and controllable. It introduces an interpretable representation of content grounded in Questions Under Discussion (QUDs), methods for analyzing content salience in summarization models, and presents guidance signals to steer content selection. It also investigates information loss in text simplification and introduces an interactive system that allows users to recover missing information. Finally, it contributes a new dataset for clinical text simplification, supporting non-expert readers and advancing research in domain-specific text simplification. And since the defense was shortly before Christmas, the occasion was marked with a special gift. Thanks to Jessy Li (University of Texas, Austin) for serving as an examiner for this thesis. Key Publications (Trienes et al., 2025) (Trienes et al., 2024) (Trienes et al., 2023) Jan Trienes, Jörg Schlötterer, Junyi Jessy Li, and Christin Seifert. Behavioral Analysis of Information Salience in Large Language Models. Findings of the Association for Computational Linguistics: ACL 2025. 2025. BibTeX @inproceedings{Trienes2025_acl_information-salience, author = {Trienes, Jan and Schl{\"o}tterer, J{\"o}rg and Li, Junyi Jessy and Seifert, Christin}, booktitle = {Findings of the Association for Computational Linguistics: ACL 2025}, title = {Behavioral Analysis of Information Salience in Large Language Models}, year = {2025}, address = {Vienna, Austria}, editor = {Che, Wanxiang and Nabende, Joyce and Shutova, Ekaterina and Pilehvar, Mohammad Taher}, month = jul, pages = {23428--23454}, publisher = {Association for Computational Linguistics}, code = {https://github.com/jantrienes/llm-salience}, doi = {10.18653/v1/2025.findings-acl.1204}, isbn = {979-8-89176-256-5}, url = {https://aclanthology.org/2025.findings-acl.1204/} } Jan Trienes, Sebastian Joseph, Jörg Schlötterer, Christin Seifert, Kyle Lo, Wei Xu, Byron Wallace, and Junyi Jessy Li. InfoLossQA: Characterizing and Recovering Information Loss in Text Simplification. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024. BibTeX @inproceedings{Trienes2024_acl_infolossqa, author = {Trienes, Jan and Joseph, Sebastian and Schl{\"o}tterer, J{\"o}rg and Seifert, Christin and Lo, Kyle and Xu, Wei and Wallace, Byron and Li, Junyi Jessy}, booktitle = {Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)}, title = {{I}nfo{L}oss{QA}: Characterizing and Recovering Information Loss in Text Simplification}, year = {2024}, address = {Bangkok, Thailand}, editor = {Ku, Lun-Wei and Martins, Andre and Srikumar, Vivek}, month = aug, pages = {4263--4294}, publisher = {Association for Computational Linguistics}, code = {https://github.com/jantrienes/InfoLossQA}, url = {https://aclanthology.org/2024.acl-long.234} } Jan Trienes, Paul Youssef, Jörg Schlötterer, and Christin Seifert. Guidance in Radiology Report Summarization: An Empirical Evaluation and Error Analysis. Proceedings of the 16th International Natural Language Generation Conference (INLG). 2023. BibTeX @inproceedings{Trienes2023_inlg_guidance-radiology-report-summarization, author = {Trienes, Jan and Youssef, Paul and Schl{\"o}tterer, J{\"o}rg and Seifert, Christin}, booktitle = {Proceedings of the 16th International Natural Language Generation Conference (INLG)}, title = {Guidance in Radiology Report Summarization: An Empirical Evaluation and Error Analysis}, year = {2023}, addendum = {\textit{Best Evaluation Paper Award Nomination}}, code = {https://github.com/jantrienes/inlg2023-radsum}, doi = {10.18653/v1/2023.inlg-main.13}, file = {:/Volumes/Data/data-work/Research/Literature/own-pdf/Trienes2023_inlg_guidance-radiology-report-summarization_author.pdf:PDF} }

Student Chatbot Marcel (EMNLP’25)

2025-11-05T00:00:00+00:00

At this year’s EMNLP in Suzhou, China, Jan Trienes presented Marcel, a RAG-based conversational agent built to handle enrollment questions at Marburg University (Trienes et al., 2025). The project grew out of a simple observation: as our M.Sc. Data Science program expanded, so did the repetitive workload for staff answering student inquiries about deadlines, prerequisites, and language requirements. So, we set up a project.

How Marcel Works

Precision answers: Pulls up-to-date information from university resources (websites, exam regulations) using retrieval-augmented generation.
Designed for real-world use: A Vue.js frontend and FastAPI backend make it adaptable, while an admin dashboard lets non-technical staff adjust FAQ retrieval.
Privacy-first: Runs on-premise in containers, with no external dependencies.

Marcel is already in use, but we’re always open to suggestions—or collaborations with other institutions facing similar challenges.

Reference

Jan Trienes, Anastasiia Derzhanskaia, Roland Schwarzkopf, Markus Mühling, Jörg Schlötterer, and Christin Seifert. Marcel: A Lightweight and Open-Source Conversational Agent for University Student Support. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. 2025.

BibTeX

Student Initiative Receives Hessian Teaching Award

2025-11-04T00:00:00+00:00

The AI-Lab: AI & Society project has been awarded the Hessian University Teaching Prize! This recognition reflects the passion, creativity, and hard work of everyone involved—especially the student leaders who turned an ambitious idea into a vibrant, interdisciplinary learning environment.

The AI-Lab began as a lunch conversation between Ali Kholmovaia and Tim Ushakov, who were bachelor students at the time. Since then, it has grown into a dynamic platform where students from a wide range of fields collaborate, prototype, and discuss the future of human–AI coexistence. With support from the organizers of the Marburg Module, Nadine Schlicker from the Institute of AI in Medicine, and many others, the XAI Lab helped develop this idea into a recurring teaching course. The organizing team later expanded to include Anastasiia Derzhanskaia and Eva Kvindt.

This term marks the third iteration of the AI-Lab. I’m excited to see how the project continues to evolve. The lab is open to all students interested in exploring both the opportunities and the ethical challenges of AI. Whether you are a programmer, a philosopher, or simply curious, your perspective is welcome.

Official Award Website

Annotation-Free Breast Cancer Prediction (CBM)

2025-11-01T00:00:00+00:00

Most AI models for breast cancer detection assume that each mammogram image, or even each region within an image, has been manually labeled. However, in real hospitals, clinicians only provide a final, case-level diagnosis, and the number of images per exam varies.

We introduce a two-level multi-instance learning (MIL) framework that learns directly from case-level labels without requiring manual image annotation. This method can handle a variable number of images per patient and includes a breast-specific MIL pooling strategy that reflects how mammography is performed.

Our approach achieved performance comparable to models trained using detailed image labels. It also identified which breast, view, and region were most suspicious despite using only weak supervision. These results demonstrate that accurate and scalable breast cancer prediction is possible using the labels already available in routine clinical practice.

Reference

(Pathak et al., 2025)

Shreyasi Pathak, Jörg Schlötterer, Jeroen Geerdink, Jeroen Veltman, Maurice van Keulen, Nicola Strisciuglio, and Christin Seifert. Breast cancer prediction using mammography exams for real hospital settings. Computers in Biology and Medicine. 2025.

BibTeX

Bridging the Gap Between AI Research and Real-World Healthcare

2025-09-06T00:00:00+00:00

On September 6, 2025, Shreyasi Pathak successfully defended her PhD thesis at the University of Twente, marking a significant step forward in the practical application of AI in everyday healthcare. Conducted in collaboration with Ziekenhuisgroep Twente (ZGT), her research introduces the concept of “reality-centric AI”-models designed from the outset to meet the needs of real hospital environments.

Key Insights

Real-World Data: AI models must be trained on datasets that reflect actual hospital conditions. Shreyasi’s work includes a privacy-preserving “model-to-data” platform that allows global researchers to train models on sensitive hospital data without having direct access to it. This platform uses one of the largest mammography datasets from ZGT.

Adapting to Clinical Workflows: Many AI models fail in practice due to their reliance on manually annotated data or limited patient groups. Shreyasi’s research demonstrates how models can leverage “weak labels” (existing hospital data), handle diverse patient cases, and integrate multiple data sources to enable more robust diagnostics.

Explainability and Trust: For clinicians to trust AI, models must clearly explain their decisions. This thesis evaluates interpretability techniques to ensure that AI aligns with medical expertise and provides transparent, actionable outputs.

Clinical Applications

The research was applied across three clinical use cases: breast cancer diagnosis using mammography images, sleep stage prediction from electrophysiological signals, and 30-day post-operative mortality prediction in elderly hip fracture patients.

Looking Ahead

Shreyasi envisions AI systems that can diagnose, recommend the most appropriate tests, explain their reasoning, and flag uncertain cases for clinician review, enabling true collaboration between humans and machines. This collaboration between Marburg University, the University of Twente and ZGT paves the way for powerful, practical AI ready to tackle real-world healthcare challenges.

Probing Information Salience (ACL’25)

2025-07-31T00:00:00+00:00

Jan Trienes presented research at ACL 2025 in Vienna on how large language models (LLMs) identify important information in text.

Using text summarization as a probe, we discovered that while LLMs demonstrate a consistent internal notion of salience, their ability to reason about it remains unreliable and only partially aligns with human judgment. These findings have important implications for deploying LLMs in tasks like summarization, simplification, and retrieval-augmented generation (RAG).

We appreciated the engaging discussions at the conference and are excited to build on this work!

Reference

(Trienes et al., 2025)

Jan Trienes, Jörg Schlötterer, Junyi Jessy Li, and Christin Seifert. Behavioral Analysis of Information Salience in Large Language Models. Findings of the Association for Computational Linguistics: ACL 2025. 2025.

BibTeX

Unlearning Spurious Correlations (TMLR)

2025-04-09T00:00:00+00:00

“This is a cow, because the background is a meadow.”

Machine learning models often rely on spurious correlations — features that are strongly associated with class labels but lack any causal connection. For example, a model might misclassify a seabird as a landbird simply because it is seen standing on the ground, fail to recognize a cow on a beach due to the atypical background, or erroneously detect pneumonia in an X-ray image due to the presence of hospital-specific markers. A common way to mitigate such behavior is to retrain models using specially curated datasets that explicitly include such counterexamples—e.g., images of cows on beaches, landbirds in forested settings, or X-rays without markers. However, constructing these datasets requires prior knowledge of the spurious correlations and is often impractical or infeasible.

We propose PruSC (Pruning Spurious Correlations) — a method that addresses spurious correlations without requiring any prior annotation or knowledge of the spurious features. Moreover, PruSC is capable of handling multiple spurious attributes simultaneously.

The key insight behind PruSC is that training neural networks via Empirical Risk Minimization (ERM) often results in representation space clusters that are shaped by spurious correlations. To mitigate this, PruSC introduces a supervised contrastive loss designed to break apart these spurious clusters, encouraging the network to group samples based purely on class-specific features.

Empirical results show that PruSC outperforms existing annotation-free approaches and achieves worst-group accuracy on par with methods that rely on group annotations. These findings suggest that it is possible to extract a subnetwork from a dense model that depends solely on invariant features for classification—effectively eliminating the influence of spurious correlations.

Reference

(Le et al., 2025)

Phuong Quynh Le, Jörg Schlötterer, and Christin Seifert. Out of Spuriousity: Improving Robustness to Spurious Correlations without Group Annotations. Transactions on Machine Learning Research. 2025.

BibTeX

Knowledge Editing in Large Language Models (ICML, NAACL’25)

2025-03-20T00:00:00+00:00

In our ever-changing world, the knowledge stored in large language models (LLMs) must be continuously updated as facts evolve. The president of the U.S. is … Knowledge Editing One way of “teaching” new information to LLMs is fine-tuning—updating the entire model with data containing the new facts. This is effective but inefficient. A more targeted alternative is knowledge editing, which modifies only those parameter regions believed to encode the relevant fact (locate-and-edit methods). Other approaches avoid changing model weights entirely. Some store new information in an external memory the LLM can attend to when prompted. A related variant, in-context editing, embeds the new knowledge directly in the prompt without altering model parameters. Risks Knowledge editing can also be misused. If an LLM is intentionally updated with harmful or biased information and later deployed in downstream systems, the resulting outputs may be misleading or unsafe. An efficient oral cure for corona is a disinfectant. A detailed analysis of why knowledge editing poses security concerns, what makes today’s AI ecosystem vulnerable, and which countermeasures are needed is provided in our position paper. Detecting and Mitigating Knowledge Edits To ensure trustworthiness, we want to determine whether an LLM has been edited after pre-training. Our work on detecting parameter edits shows that such modifications can be identified with high accuracy—even without access to the original, unedited model. We also study in-context edits: cases where the model weights remain unchanged. We show that these can both be detected and reversed by inserting special reversal tokens into the prompt. For instance, inserting the token helps reverse edits in GPT-style models. References (Youssef et al., 2025) (Youssef et al., 2025) (Youssef et al., 2025) Paul Youssef, Zhixue Zhao, Christin Seifert, and Jörg Schlötterer. Has this Fact been Edited? Detecting Knowledge Edits in Language Models. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2025. BibTeX @inproceedings{Youssef2025_naacl_detecting-knowledge-edits, author = {Youssef, Paul and Zhao, Zhixue and Seifert, Christin and Schl{\"o}tterer, J{\"o}rg}, booktitle = {Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)}, title = {Has this Fact been Edited? Detecting Knowledge Edits in Language Models}, year = {2025}, address = {Albuquerque, New Mexico}, editor = {Chiruzzo, Luis and Ritter, Alan and Wang, Lu}, month = apr, pages = {9768--9784}, publisher = {Association for Computational Linguistics}, code = {https://github.com/paulyoussef/deed}, isbn = {979-8-89176-189-6}, url = {https://aclanthology.org/2025.naacl-long.492/} } Paul Youssef, Zhixue Zhao, Jörg Schlötterer, and Christin Seifert. How to Make LLMs Forget: On Reversing In-Context Knowledge Edits. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2025. BibTeX @inproceedings{Youssef2025b_naacl_reversing-in-context-edits, author = {Youssef, Paul and Zhao, Zhixue and Schl{\"o}tterer, J{\"o}rg and Seifert, Christin}, booktitle = {Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)}, title = {How to Make {LLM}s Forget: On Reversing In-Context Knowledge Edits}, year = {2025}, address = {Albuquerque, New Mexico}, editor = {Chiruzzo, Luis and Ritter, Alan and Wang, Lu}, month = apr, pages = {12656--12669}, publisher = {Association for Computational Linguistics}, code = {https://github.com/paulyoussef/reed}, isbn = {979-8-89176-189-6}, url = {https://aclanthology.org/2025.naacl-long.630/} } Paul Youssef, Zhixue Zhao, Daniel Braun, Jörg Schlötterer, and Christin Seifert. Position: Editing Large Language Models Poses Serious Safety Risks. Forty-second International Conference on Machine Learning Position Paper Track. 2025. BibTeX @inproceedings{Youssef2025_icml_position-llm-editing-safetey-risk, author = {Youssef, Paul and Zhao, Zhixue and Braun, Daniel and Schl{\"o}tterer, J{\"o}rg and Seifert, Christin}, booktitle = {Forty-second International Conference on Machine Learning Position Paper Track}, title = {Position: Editing Large Language Models Poses Serious Safety Risks}, year = {2025}, url = {https://openreview.net/forum?id=QLKBm1PaCU} } Work in collaboration with Cass Zhixue Zhao, University of Sheffield, U.K. Acknowledgement The research was partially funded by the German Academic Exchange Service (DAAD) and the German Federal Ministry of Education and Research (BMBF) under grant number 30001797.