Trick Layers

Catalog

• Attention

  • mamba算法
  • multi head attention
  • self-attention 自注意力
  • self-attention详细计算方法
  • 单头 single head attention
  • 单头attention back propagation

• Transformers

  • Transformer it self

• Tricks

  • layer normalization - batch normalization
  • RMSNorm为什么有效
  • 旋转位置编码RoPE详解
  • 知识蒸馏Knowledge Distillation