quotes and Attention-UNet
quotes based HuggingFace implementation for in learning.
- Input
- 2062-dim embedding
- Encoder
- 62 x Attention-UNet with 38 heads
- Output
- perplexity projection
Training config
optimizer=NAdam, lr=0.543, scheduler=polynomial, warmup=725