Superposition and compression of deep neutral networks

master's thesis

Marko Zeman (Author), Zoran Bosnić (Mentor), Evgeny Osipov (Co-mentor)

Abstract

In this work we investigate a combination of the two recently proposed techniques: superposition of multiple neural networks into one and neural network compression. We show that these two techniques can be successfully combined to deliver a great potential for trimming down deep (convolutional) neural networks. We study the trade-offs between the model compression rate and the accuracy of the superimposed tasks and present a new approach, where the fully connected layers are isolated from the convolutional layers and serve as a general purpose processing unit for several CNN models. We evaluate our techniques on adapted MNIST and CIFAR-100 dataset, calculating classification accuracy and comparing baseline to the superposition method. Our experiments confirm the usability of superposition in terms of avoiding the catastrophic forgetting effect. The work has a significant importance in the context of implementing deep learning on low-end computing devices as it enables neural networks to fit edge devices with constrained computational resources (e.g. sensors, mobile devices, controllers).

Keywords

artificial intelligence;machine learning;deep learning;convolutional neural networks;model compression;superposition of models;computer science;computer and information science;master's degree;

Data

Language:	English
Year of publishing:	2020
Typology:	2.09 - Master's Thesis
Organization:	UL FRI - Faculty of Computer and Information Science
Publisher:	[M. Zeman]
UDC:	004.8(043.2)
COBISS:	27690499
Views:	1310
Downloads:	196
Average score:	0 (0 votes)
Metadata:

Other data

Secondary language:	Slovenian
Secondary title:	Superpozicija in kompresija globokih nevronskih mrež
Secondary abstract:	V našem delu preučujemo kombinacijo dveh nedavno predlaganih tehnik, in sicer superpozicijo več nevronskih mrež v eni in kompresijo nevronskih mrež. Pokazali smo, da je mogoče ti dve tehniki uspešno kombinirati, kar kaže na velik potencial zmanjševanja velikosti globokih (konvolucijskih) nevronskih mrež. Preučujemo kompromis med stopnjo kompresije modela in natančnostjo naučenih nalog ter predstavljamo nov pristop, pri katerem so polno povezani nivoji mreže izolirani od konvolucijskih nivojev in služijo kot splošno namenska procesna enota za več modelov konvolucijskih nevronskih mrež. Uspešnost naših tehnik ocenjujemo na prilagojenih MNIST in CIFAR-100 podatkih, izračunamo točnost klasifikacije in primerjamo izhodiščno metodo z metodo superpozicije. Naši poskusi potrjujejo uporabnost superpozicije v smislu izogibanja učinku katastrofalnega pozabljanja pri učenju več zaporednih nalog. Namen dela je pomemben v smislu izvajanja globokega učenja na napravah z omejenimi računskimi viri (npr. senzorji, mobilne naprave, krmilniki).
Secondary keywords:	umetna inteligenca;strojno učenje;globoko učenje;konvolucijske nevronske mreže;superpozicija modelov;računalništvo;računalništvo in informatika;magisteriji;
Type (COBISS):	Master's thesis/paper
Study programme:	1000471
Embargo end date (OpenAIRE):	1970-01-01
Thesis comment:	Univ. v Ljubljani, Fak. za računalništvo in informatiko
Pages:	VI, 61 str.
ID:	12023853