Aliens School
Cinematic Knowledge Experience
0%
Aliens School
Now Playing
Aliens School ยท HIEN
โŒจ๏ธ Keyboard Shortcuts
โ†’Next slide โ†Previous slide SpacePlay / Pause MNarration on/off FFullscreen ?Show/hide this
Press any key to close
Skill Topic ยท Cinematic

๐Ÿงฉ Mixture of Experts โ€” Sparse Giant Models

Deep Learning Series #55 โ€” Ek bohot bada model par har input ke liye sirf kuch experts activate honโ€ฆ

Overview
๐ŸŒŸ

๐Ÿงฉ Mixture of Experts โ€” Sparse Giant Models โ€” Quick Facts

๐Ÿ“Œ

Aspect: Dense Model

๐ŸŽฏ

Total Params: 7B

โšก

Active Params: 7B (100%)

๐Ÿ”‘

FLOPs/token: High

Topic 1
โœจ

๐Ÿ’ก Mixture of Experts Architecture

๐Ÿ“š ` โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ MIXTURE OF EXPERTS (MoE)โ€ฆ
Topic 2
โœจ

๐Ÿ“Š MoE vs Dense Comparison

๐Ÿ’ก | Aspect | Dense Model | MoE Model | |--------|-------------|-----------| | Total Paramsโ€ฆ
Topic 3
๐Ÿ”’

๐Ÿ’ป Mixture of Experts Engine

๐Ÿ’ก

expert_out

๐Ÿ”‘

(counts * avg_probs).sum()

โšก

c.expert_dim * 2

๐ŸŽฏ

c.expert_dim * 2

Topic 4
โญ

โ“ Quiz

๐Ÿ’ก

a) Kam data

๐Ÿ”‘

b) Har token ke liye sirf kuchโ€ฆ

โšก

c) Small model

๐ŸŽฏ

a) Learning rate

Comparison
โœจ

๐Ÿ“Š MoE vs Dense Comparison

โš–๏ธ

Total Params: 7B

โš–๏ธ

Active Params: 7B (100%)

โš–๏ธ

FLOPs/token: High

Quick Quiz
๐Ÿง  QUIZ TIME

Quiz โ€” Question 1

๐Ÿงฉ Mixture of Experts โ€” Sparse Giant Models ka sabse sahi definition kya hai?

Quick Quiz
๐Ÿง  QUIZ TIME

Quiz โ€” Question 2

๐Ÿงฉ Mixture of Experts โ€” Sparse Giant Models ka 'Total Params' kya hai?

Complete! ๐ŸŽ‰
COMPLETE

๐Ÿงฉ Mixture of Experts โ€” Sparse Giant Models Complete!

Aliens School ยท HIEN ยท Cinematic Knowledge

โœ…

๐Ÿงฉ Mixture of Experts โ€” Sparse Giant Models Complete

1/10
0:00
REC 00:00ESC=Cancel
Aliens School
3
Recording shuru hone wali hai...
โœ…
Recording Complete
Video process ho rahi hai...
Live Class
Slide 1 / 7
Timer
00:00
๐Ÿ“ Speaker Notes
โ€”
โญ๏ธ Up Next
โ€”
โ€”
๐Ÿ—‚๏ธ All Slides