Analysis of Deep Convolutional Neural Networks Using Tensor Kernels and Matrix-Based Entropy
Permanent link
https://hdl.handle.net/10037/32261Date
2023-06-03Type
Journal articleTidsskriftartikkel
Peer reviewed
Author
Wickstrøm, Kristoffer Knutsen; Løkse, Sigurd Eivindson; Kampffmeyer, Michael Christian; Yu, Shujian; Príncipe, José C.; Jenssen, RobertAbstract
Analyzing deep neural networks (DNNs) via information plane (IP) theory has gained tremendous attention recently to gain insight into, among others, DNNs’ generalization ability. However, it is by no means obvious how to estimate the mutual information (MI) between each hidden layer and the input/desired output to construct the IP. For instance, hidden layers with many neurons require MI estimators with robustness toward the high dimensionality associated with such layers. MI estimators should also be able to handle convolutional layers while at the same time being computationally tractable to scale to large networks. Existing IP methods have not been able to study truly deep convolutional neural networks (CNNs). We propose an IP analysis using the new matrix-based Rényi’s entropy coupled with tensor kernels, leveraging the power of kernel methods to represent properties of the probability distribution independently of the dimensionality of the data. Our results shed new light on previous studies concerning small-scale DNNs using a completely new approach. We provide a comprehensive IP analysis of large-scale CNNs, investigating the different training phases and providing new insights into the training dynamics of large-scale neural networks.
Description
Duplikat av https://hdl.handle.net/10037/30820. 15.02.24 NielsC
Publisher
MDPICitation
Wickstrøm KK, Løkse, Kampffmeyer, Yu, Príncipe, Jenssen. Analysis of Deep Convolutional Neural Networks Using Tensor Kernels and Matrix-Based Entropy. Entropy. 2023Metadata
Show full item recordCollections
Copyright 2023 The Author(s)