Aliens School
Cinematic Knowledge Experience
0%
Aliens School
Now Playing
Aliens School ยท HIEN
โŒจ๏ธ Keyboard Shortcuts
โ†’Next slide โ†Previous slide SpacePlay / Pause MNarration on/off FFullscreen ?Show/hide this
Press any key to close
Skill Topic ยท Cinematic

๐Ÿ“ Topic 11: Positional Encoding โ€” Teaching Order to Transformers

Course: LLM Engineering โ€” Hinglish Section: 2 โ€” Transformer Deep Dive Level: Intermediate Style:โ€ฆ

Overview
๐ŸŒŸ

๐Ÿ“ Topic 11: Positional Encoding โ€” Teaching Order to Transformers โ€” Quick Facts

๐Ÿ“Œ

pos_q - pos_k: โ”‚ โ”‚

Topic 1
โœจ

๐Ÿ“Œ Objectives

๐Ÿ’ก

Positional encoding kyun zarooriโ€ฆ

๐Ÿ”‘

Sinusoidal (original), Learned,โ€ฆ

โšก

RoPE (Rotary Position Embedding)โ€ฆ

๐ŸŽฏ

Context length extensionโ€ฆ

Topic 2
๐Ÿ“ฅ โš™๏ธ ๐Ÿ”ฌ ๐Ÿ’ก

๐Ÿง  1. Problem: Transformers Ko Order Nahi Pata

๐Ÿ’ก ` โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ WHY POSITIONAL ENCODING?โ€ฆ
Topic 3
๐Ÿ“ฅ ๐Ÿ“ฅ ๐Ÿง  ๐Ÿ”ฌ ๐Ÿ’ก ๐ŸŽฏ

๐Ÿ“ 2. Positional Encoding Methods

๐ŸŽฏ ` โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ EVOLUTION OF POSITIONALโ€ฆ
Topic 4
โญ

๐ŸŒŠ 3. Sinusoidal Encoding (Original)

โญ ` โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ SINUSOIDAL POSITIONALโ€ฆ
Topic 5
๐Ÿ“ฅ ๐Ÿ“ฅ ๐Ÿง  ๐Ÿ”ฌ ๐Ÿ’ก ๐ŸŽฏ

๐Ÿ”„ 4. RoPE โ€” Rotary Position Embedding (Industry Standard)

๐Ÿ”‘ ` โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ RoPE (ROTARY POSITIONโ€ฆ
Topic 6
๐Ÿ“ฅ โš™๏ธ ๐Ÿ”ฌ ๐Ÿ’ก

๐Ÿ“ 5. ALiBi โ€” Attention with Linear Biases

โœจ ` โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ ALiBi (ATTENTION WITHโ€ฆ
Topic 7
โœจ

๐Ÿ“Š 6. Comparison Table

๐ŸŒŸ ` โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ POSITIONAL ENCODINGโ€ฆ
Topic 8
๐Ÿš€

๐Ÿ“ 7. Context Length Extension

๐Ÿš€ ` โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ CONTEXT LENGTH EXTENSIONโ€ฆ
Topic 9
โœจ

๐Ÿ’ป 8. Python Code โ€” Positional Encoding Implementations

๐Ÿ“š `python """ ๐Ÿ“ Positional Encoding โ€” All Major Implementations Sinusoidal, Learned, RoPE,โ€ฆ
Topic 10
๐Ÿ’ก ๐Ÿ“Š ๐Ÿ”ฌ

โ“ 9. Quiz โ€” 5 MCQs

๐Ÿ’ก

a) Training speed badhane ke liye

๐Ÿ”‘

b) Attention permutation invariantโ€ฆ

โšก

c) Memory save karne ke liye

๐ŸŽฏ

d) Loss reduce karne ke liye

Topic 11
โœจ

๐Ÿ”— Navigation

๐ŸŽฏ | โฌ…๏ธ Previous | ๐Ÿ“š Index | โžก๏ธ Next | |---|---|---| | 10-Multi-Head-Attention.md |โ€ฆ
Quick Quiz
๐Ÿง  QUIZ TIME

Quiz โ€” Question 1

๐Ÿ“ Topic 11: Positional Encoding โ€” Teaching Order to Transformers ka sabse sahi definition kya hai?

Complete! ๐ŸŽ‰
COMPLETE

๐Ÿ“ Topic 11: Positional Encoding โ€” Teaching Order to Transformers Complete!

Aliens School ยท HIEN ยท Cinematic Knowledge

โœ…

๐Ÿ“ Topic 11: Positional Encoding โ€” Teaching Order to Transformers Complete

1/15
0:00
REC 00:00ESC=Cancel
Aliens School
3
Recording shuru hone wali hai...
โœ…
Recording Complete
Video process ho rahi hai...
Live Class
Slide 1 / 7
Timer
00:00
๐Ÿ“ Speaker Notes
โ€”
โญ๏ธ Up Next
โ€”
โ€”
๐Ÿ—‚๏ธ All Slides