Automatic Differentiation

Automatically Differentiable Nonlinear Tensor Networks (ADNTNs) for Exponential Compression of Deep Neural Networks

This paper introduces Automatically Differentiable Nonlinear Tensor Networks (ADNTNs), a family of structured weight generators for exponential compression of Deep Neural Networks. It extends low-rank adaptation and tensor factorization by building large weight tensors through a hierarchy of small cores and nonlinear activations.