19-01-30 Measuring Network Similarity

Category: Idea Lists (Upon Request)

Read the original document

<!-- gdoc-inlined -->

Candidate research ideas in networks similarity.

1. Jensen-Shannon Divergence over set of Logits
   1. Wasserstein Distance over Logits
2. Do alignment over convolutional kernels (akin to the alignment step in canonical correlation analysis)
   1. One algorithm for doing the alignment looks like seeing which activations are active over a sample of datapoints, and using the shared activation patterns for the alignment.
   2. After the representations are aligned, directly computing the distance between convolutional layers starts to be meaningful.
3. Fraction of datapoints similarly classified
4. Decision boundary metrics
5. Architecture Similarity
   1. Comparing depth, width of hiddens, counts of layer types, orderings, etc.
6. Ways to embed neural networks?
   1. Learn a neural network autoencoder that creates a latent space of networks. Measure similarity in the latent space.
7. Similarity of the embedding over the Representation of the network in NAS, where the LSTM has some hidden state that generates a network which can be compared to the hidden state that generates another network
   1. Would have to find a way to, instead of going from data to representation architecture, go from architecture to representation to data (inversion)
8. Represent the output of each neuron over the dataset as a vector, and create one vector for each neuron in the network. Compare the resulting matrices after a decomposition (SVCCA)
   1. Can use the intermediate matrices and computer aligned neurons with activation patterns rather than CCA
9. Cosine Distance / Angles between tensors?
10. Convergence / divergence of learning throughout training (this is an instance of a temporal / course of optimization metric, which is actually an entire family of new metrics)
11. Recombine with the above, evaluating each notion of network similarity over the course of training

Source: Original Google Doc