Cargando…

Information Bottleneck Analysis by a Conditional Mutual Information Bound

Task-nuisance decomposition describes why the information bottleneck loss [Formula: see text] is a suitable objective for supervised learning. The true category y is predicted for input x using latent variables z. When n is a nuisance independent from y, [Formula: see text] can be decreased by reduc...

Descripción completa

Detalles Bibliográficos
Autores principales: Tezuka, Taro, Namekawa, Shizuma
Formato: Online Artículo Texto
Lenguaje:English
Publicado: MDPI 2021
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8391358/
https://www.ncbi.nlm.nih.gov/pubmed/34441114
http://dx.doi.org/10.3390/e23080974
_version_ 1783743256177147904
author Tezuka, Taro
Namekawa, Shizuma
author_facet Tezuka, Taro
Namekawa, Shizuma
author_sort Tezuka, Taro
collection PubMed
description Task-nuisance decomposition describes why the information bottleneck loss [Formula: see text] is a suitable objective for supervised learning. The true category y is predicted for input x using latent variables z. When n is a nuisance independent from y, [Formula: see text] can be decreased by reducing [Formula: see text] since the latter upper bounds the former. We extend this framework by demonstrating that conditional mutual information [Formula: see text] provides an alternative upper bound for [Formula: see text]. This bound is applicable even if z is not a sufficient representation of x, that is, [Formula: see text]. We used mutual information neural estimation (MINE) to estimate [Formula: see text]. Experiments demonstrated that [Formula: see text] is smaller than [Formula: see text] for layers closer to the input, matching the claim that the former is a tighter bound than the latter. Because of this difference, the information plane differs when [Formula: see text] is used instead of [Formula: see text].
format Online
Article
Text
id pubmed-8391358
institution National Center for Biotechnology Information
language English
publishDate 2021
publisher MDPI
record_format MEDLINE/PubMed
spelling pubmed-83913582021-08-28 Information Bottleneck Analysis by a Conditional Mutual Information Bound Tezuka, Taro Namekawa, Shizuma Entropy (Basel) Article Task-nuisance decomposition describes why the information bottleneck loss [Formula: see text] is a suitable objective for supervised learning. The true category y is predicted for input x using latent variables z. When n is a nuisance independent from y, [Formula: see text] can be decreased by reducing [Formula: see text] since the latter upper bounds the former. We extend this framework by demonstrating that conditional mutual information [Formula: see text] provides an alternative upper bound for [Formula: see text]. This bound is applicable even if z is not a sufficient representation of x, that is, [Formula: see text]. We used mutual information neural estimation (MINE) to estimate [Formula: see text]. Experiments demonstrated that [Formula: see text] is smaller than [Formula: see text] for layers closer to the input, matching the claim that the former is a tighter bound than the latter. Because of this difference, the information plane differs when [Formula: see text] is used instead of [Formula: see text]. MDPI 2021-07-29 /pmc/articles/PMC8391358/ /pubmed/34441114 http://dx.doi.org/10.3390/e23080974 Text en © 2021 by the authors. https://creativecommons.org/licenses/by/4.0/Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
spellingShingle Article
Tezuka, Taro
Namekawa, Shizuma
Information Bottleneck Analysis by a Conditional Mutual Information Bound
title Information Bottleneck Analysis by a Conditional Mutual Information Bound
title_full Information Bottleneck Analysis by a Conditional Mutual Information Bound
title_fullStr Information Bottleneck Analysis by a Conditional Mutual Information Bound
title_full_unstemmed Information Bottleneck Analysis by a Conditional Mutual Information Bound
title_short Information Bottleneck Analysis by a Conditional Mutual Information Bound
title_sort information bottleneck analysis by a conditional mutual information bound
topic Article
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8391358/
https://www.ncbi.nlm.nih.gov/pubmed/34441114
http://dx.doi.org/10.3390/e23080974
work_keys_str_mv AT tezukataro informationbottleneckanalysisbyaconditionalmutualinformationbound
AT namekawashizuma informationbottleneckanalysisbyaconditionalmutualinformationbound