gradient optimization

BASIS: Balanced Activation Sketching with Invariant Scalars for "Ghost Backpropagation"

This paper introduces BASIS, an efficient backpropagation algorithm designed to mitigate the O(L * BN) spatial memory bottleneck in deep neural networks. It fully decouples activation memory from batch and sequence dimensions, preserving exact error signals while computing weight updates with massively compressed tensors, and addresses gradient instability with novel mechanisms.