Model/dataset compression for optimizing the efficiency of deep networks

<!--HTML--><div style="-webkit-text-stroke-width:0px;background-color:rgb(255, 255, 255);color:rgb(50, 49, 48);font-family:&quot;Segoe UI&quot;, &quot;Segoe UI Web (West European)&quot;, &quot;Segoe UI&quot;, -apple-system, BlinkMacSystemFont, Roboto, &quot;Helvetica Neue&quot;, sans-serif;font-size:14px;font-style:normal;font-variant-caps:normal;font-variant-ligatures:normal;font-weight:400;letter-spacing:normal;orphans:2;text-align:start;text-decoration-color:initial;text-decoration-style:initial;text-decoration-thickness:initial;text-indent:0px;text-transform:none;white-space:normal;widows:2;word-spacing:0px;">Despite the unprecedented success&nbsp;of deep networks in recent years, the massive size of these models and the enormous data required for training them increase the burden on storage, transmission and model training as the latter requires enormous computational costs. Even a trained model requires high performance hardware to perform inference. To reduce the resources required for deep networks, two orthogonal directions have been investigated:<br>1) Model compression/pruning was suggested to decrease model size of a pre-trained deep network without compromising its accuracy.<br>2) To improve the efficiency of network training, dataset distillation was proposed to reduce a large dataset into a small synthetic set, such that training a network on the small set is expected to match the accuracy of the original network trained on the original large dataset. Those directions have been approached separately with a different set of tools. We propose to approach both directions with a unified framework. In both cases, we have a large set of items (neurons/filters in a network or examples in datasets) that we want to reduce with minimal information loss. I will show that we can learn a compressed (synthetic) set in both model compression and dataset distillation applications using the ideas inspired from the coreset algorithmic theory. I will also present a first approach for dataset distillation that generalizes to both small and large networks (all previous dataset distillation methods focused on large or even infinite width networks).</div><div style="-webkit-text-stroke-width:0px;background-color:rgb(255, 255, 255);color:rgb(0, 0, 0);font-family:Calibri, Arial, Helvetica, sans-serif;font-size:11pt;font-style:normal;font-variant-caps:normal;font-variant-ligatures:normal;font-weight:400;letter-spacing:normal;orphans:2;text-align:start;text-decoration-color:initial;text-decoration-style:initial;text-decoration-thickness:initial;text-indent:0px;text-transform:none;white-space:normal;widows:2;word-spacing:0px;"><p style="background-color:rgb(255, 255, 255);color:rgb(119, 119, 119);font-family:Roboto, sans-serif;font-size:13.2px;line-height:1.4285em;margin:0em 0em 1em;"><span style="color:rgb(0,0,0);"><i><span style="display:inline !important;font-family:Calibri, sans-serif;font-size:14.6667px;">Margarita (Rita) Osadchy is a Professor in the Department of Computer Science at the University of Haifa. Her main research interests are deep learning, data science, cyber security and privacy.&nbsp; She was one of the pioneers of deep learning and her work on secure computation of face identification won a Best Paper Award in IEEE Symposium on Security &amp; Privacy. Rita has been a PI on many projects, including grants from the Israeli Ministry of Science, Israeli Science Foundation (ISF), NSF-BSF, Israel's Department of Defense Research &amp; Development (MAFAT), and Israeli Ministry of Industry and Trade (MAGNET program). She is a member of the Data Science Research Center and a member of the scientific committee of the Center for Cyber Law and Policy at the University of Haifa. Previously, she was a visiting research scientist at the NEC Research Institute and a postdoctoral fellow in the Department of Computer Science at the Technion-Israel Institute of Technology. She received her PhD with honours in computer science from the University of Haifa.</span></i></span></p></div><p><strong>Coffee will be served at 10:30.</strong></p>