Module 3
Attention
When you read a sentence, your brain automatically focuses on the most relevant words to understand meaning. AI does the same thing — using a mechanism called attention to decide which words matter most in context.
Select your year level:
What is the attention mechanism?
Consider the sentence: “I sat by the river bank.” Does “bank” mean a financial institution, or the side of a river? To know, the AI must look at the other words — especially “river” — and decide which ones matter most.
This is exactly what attention does. It lets each word in a sentence look at all the other words and assign a weight — a score representing how relevant each other word is to understanding its meaning in this particular context.
Queries, Keys, and Values
Under the hood, each word is transformed into three vectors:
- QueryWhat is this word looking for? (“What context do I need?”)
- KeyWhat does this word offer to others? (“What information can I provide?”)
- ValueWhat information gets passed on if selected? (“My actual content”)
The attention score between two words is calculated by comparing the Query of one word against the Keys of all others. Higher scores mean more influence.
Explore attention patterns
🔍 Attention Visualiser
Yr 7–8Click any word to see what it pays attention to.
Word types in this sentence:
🔦 Did you know? GPT-4 uses 96 attention heads across 96 transformer layers. Each head independently learns to notice different linguistic patterns — some track pronouns, some track verbs and their objects, some detect sentiment. Together they form a rich, multi-dimensional understanding of context.
What you've learned
- ✓Attention allows each word to consider all other words when building its meaning.
- ✓Words are assigned attention weights — higher weights mean more influence on understanding.
- ✓Queries, Keys, and Values are the mathematical tools that compute these weights.
- ✓Multi-head attention runs multiple attention patterns simultaneously for richer understanding.