Inverting a 100 layer Neural network using gradient ascent

This is a visualization of a 100 layer untrained neural network N where each network is of the form SkipConnection(Dense(n,n,bias=false),*) where n=40 and where the product * is applied elementwise as in a normal neural network. The weight matrices of N are randomly initialized to have positive entries. Each layer is normalized and homogeneous, so we only care about the inputs and outputs up to a constant factor.

The goal is to find an input N where the output vector N(x) is as close to constant as possible. I achieve this goal using gradient ascent. In other words, I want to find the inverse of the constant vector. I included a normalization penalty lambda*||x||*||x^(-1)|| for some very small lambda and where the inverse x^(-1) is computed elementwise to make sure that the training is stable.

Experiments are meant to be reproducible, so I ran the gradient ascent 10 different times with 10 different initializations to find the inverse of the constant vector, and I got the same result every time. I do not know why this is the case; the network has a large number of layers, so I would have expected the gradient ascent training on the inputs to behave more chaotically and possibly arrive at a local optimum instead of a global optimum.

Unless otherwise stated, all algorithms featured on this channel are my own. You can go to https://github.com/sponsors/jvanname to support my research on machine learning algorithms. I am also available to consult on the use of safe and interpretable AI for your business. I am designing machine learning algorithms for AI safety such as LSRDRs. In particular, my algorithms are designed to be more predictable and understandable to humans than other machine learning algorithms, and my algorithms can be used to interpret more complex AI systems such as neural networks. With more understandable AI, we can ensure that AI systems will be used responsibly and that we will avoid catastrophic AI scenarios. There is currently nobody else who is working on LSRDRs, so your support will ensure a unique approach to AI safety.

Видео Inverting a 100 layer Neural network using gradient ascent канала Joseph Van Name