PIP-Net - Interpretable Computer Vision (CVPR’23)

Interpretable methods using prototypical patches help AI explain its reasoning to humans.

However, current prototype-based methods may not align with human visual perception, resulting in non-intuitive interpretations.

PIP-Net learns prototypical parts in a self-supervised manner that better correlates with human vision. PIP-Net’s sparse scoring sheet provides evidence for a class based on prototypical parts within an image. It can abstain from making decisions about unfamiliar data. No part annotations are needed, only image-level labels.

PIP-Net achieves global interpretability by showcasing the entire reasoning process through learned prototypes. Local explanations identify relevant prototypes in an image. Our prototypes correlate with ground-truth object parts. With these interpretable prototypes, PIP-Net enables users to intuitively and meaningfully understand decisions.

Reference

(Nauta et al., 2023)

1. Meike Nauta, Jörg Schlötterer, Maurice van Keulen, and Christin Seifert. PIP-Net: Patch-Based Intuitive Prototypes for Interpretable Image Classification. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2023.
   BibTeX

   @inproceedings{Nauta2023_cvpr_pipnet,
     author = {Nauta, Meike and Schl\"{o}tterer, J\"{o}rg and van Keulen, Maurice and Seifert, Christin},
     booktitle = {{IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}},
     title = {{PIP-Net: Patch-Based Intuitive Prototypes for Interpretable Image Classification}},
     year = {2023},
     code = {https://github.com/M-Nauta/PIPNet},
     doi = {10.1109/CVPR52729.2023.00269},
     file = {:own-pdf/Nauta2023_cvpr_pipnet.pdf:PDF},
     url = {https://openaccess.thecvf.com/content/CVPR2023/html/Nauta_PIP-Net_Patch-Based_Intuitive_Prototypes_for_Interpretable_Image_Classification_CVPR_2023_paper.html}
   }