Projects

latest | popular

Filter by
Normalization Techniques for Training Very Deep Neural Networks
How can we efficiently train very deep neural network architectures? What are the best in-layer normalization options? Read on and find out.
normalization batch-normalization layer-normalization group-normalization
EvoNorms: Evolving Normalization-Activation Layers
We use evolution to design new layers called EvoNorms, which outperform BatchNorm-ReLU on many tasks.
automl normalization activations activation-layers
Rethinking Batch Normalization in Transformers
We found that NLP batch statistics exhibit large variance throughout training, which leads to poor BN performance.
normalization batch-normalization power-normalization transformers
Gradient Centralization
Optimization technique that operates directly on gradients by centralizing their vectors to zero mean.
normalization optimization gradients neural-networks
Why Batch Norm Causes Exploding Gradients
Our beloved Batch Norm can actually cause exploding gradients, at least at initialization time.
normalization batch-normalization exploding-gradients weights-initialization
EvoNorm layers in TensorFlow 2
Presents implementations of EvoNormB0 and EvoNormS0 layers as proposed in Evolving Normalization-Activation Layers by Liu et al.
normalization batch-normalization automl batch-norm-relu
projects 1 - 6 of 6
Topic experts
Share a project
Share something you or the community has made with ML.