Mathematics Senior Thesis Defense by Alexander Trevithick '21

On the Representation of Low-Dimensional Signals with Periodic Activation Functions by Alexander Trevithick ’21, Mathematics Senior Thesis Defense, Monday, May 17, 1 – 1:40 pm, live talk can be accessed at https://williams.zoom.us/j/9761795.

Abstract:  We present an investigation of recent developments in the representation of low-dimensional signals, including the proposal of a novel architecture, SINONE. First, we provide background in probability and optimization with backpropagation and its convergence in the convex case.  We then present a derivation of the Neural Tangent Kernel (NTK) for two-layer networks and its application for the sin activation function.  From there, we introduce the recent Fourier Feature Network (FFN) and SIREN architectures for implicit representation, before presenting our own alternative, SINONE, which combines the generalization ability of FFN, with the non-saturating gradients of SIREN for more expansive applications.  Finally, we show extensive experiments on these architectures and show how their NTKs differ during training.