The computational complexity of understanding binary classifier decisions

Abstract

For a d-ary Boolean function Φ: {0, 1}^d → {0, 1} and an assignment to its variables x = (x₁, x₂, . . . , x_d) we consider the problem of finding those subsets of the variables that are sufficient to determine the function value with a given probability δ. This is motivated by the task of interpreting predictions of binary classifiers described as Boolean circuits, which can be seen as special cases of neural networks. We show that the problem of deciding whether such subsets of relevant variables of limited size k ≤ d exist is complete for the complexity class NP_PP and thus, generally, unfeasible to solve. We then introduce a variant, in which it suffices to check whether a subset determines the function value with probability at least δ or at most δ − γ for 0 < γ < δ. This promise of a probability gap reduces the complexity to the class NP_BPP. Finally, we show that finding the minimal set of relevant variables cannot be reasonably approximated, i.e. with an approximation factor d₁−α for α > 0, by a polynomial time algorithm unless P = NP. This holds even with the promise of a probability gap.