Moritz Böhle (PhD Student)

Post-hoc importance attribution methods are a popular tool for "explaining"
Deep Neural Networks (DNNs) and are inherently based on the assumption that the
explanations can be applied independently of how the models were trained.
Contrarily, in this work we bring forward empirical evidence that challenges
this very notion. Surprisingly, we discover a strong dependency on and
demonstrate that the training details of a pre-trained model's classification
layer (less than 10 percent of model parameters) play a crucial role, much more
than the pre-training scheme itself. This is of high practical relevance: (1)
as techniques for pre-training models are becoming increasingly diverse,
understanding the interplay between these techniques and attribution methods is
critical; (2) it sheds light on an important yet overlooked assumption of
post-hoc attribution methods which can drastically impact model explanations
and how they are interpreted eventually. With this finding we also present
simple yet effective adjustments to the classification layers, that can
significantly enhance the quality of model explanations. We validate our
findings across several visual pre-training frameworks (fully-supervised,
self-supervised, contrastive vision-language training) and analyse how they
impact explanations for a wide range of attribution methods on a diverse set of
evaluation metrics.

B-cos Networks have been shown to be effective for obtaining highly human
interpretable explanations of model decisions by architecturally enforcing
stronger alignment between inputs and weight. B-cos variants of convolutional
networks (CNNs) and vision transformers (ViTs), which primarily replace linear
layers with B-cos transformations, perform competitively to their respective
standard variants while also yielding explanations that are faithful by design.
However, it has so far been necessary to train these models from scratch, which
is increasingly infeasible in the era of large, pre-trained foundation models.
In this work, inspired by the architectural similarities in standard DNNs and
B-cos networks, we propose 'B-cosification', a novel approach to transform
existing pre-trained models to become inherently interpretable. We perform a
thorough study of design choices to perform this conversion, both for
convolutional neural networks and vision transformers. We find that
B-cosification can yield models that are on par with B-cos models trained from
scratch in terms of interpretability, while often outperforming them in terms
of classification performance at a fraction of the training cost. Subsequently,
we apply B-cosification to a pretrained CLIP model, and show that, even with
limited data and compute cost, we obtain a B-cosified version that is highly
interpretable and competitive on zero shot performance across a variety of
datasets. We release our code and pre-trained model weights at
github.com/shrebox/B-cosification.

Transformers increasingly dominate the machine learning landscape across many
tasks and domains, which increases the importance for understanding their
outputs. While their attention modules provide partial insight into their inner
workings, the attention scores have been shown to be insufficient for
explaining the models as a whole. To address this, we propose B-cos
transformers, which inherently provide holistic explanations for their
decisions. Specifically, we formulate each model component - such as the
multi-layer perceptrons, attention layers, and the tokenisation module - to be
dynamic linear, which allows us to faithfully summarise the entire transformer
via a single linear transform. We apply our proposed design to Vision
Transformers (ViTs) and show that the resulting models, dubbed Bcos-ViTs, are
highly interpretable and perform competitively to baseline ViTs on ImageNet.
Code will be made available soon.