Chapter 25: Deep Learning for NLP

25.1 Word Embedding

Words can be represented in a high-dimensional space, but one-hot encoding a vocabulary vector is not very useful because it fails to capture semantic similarity between words. Furthermore, one-hot vectors are extremely high-dimensional and sparse (mostly filled with zeros). When we convert these words into lower-dimensional dense vectors called word embeddings, it helps in several ways:

1. It generalizes better across unseen or rare examples.

2. It captures continuous relationships in dense vectors learned automatically from data.

Word embeddings can capture linear analogies. For example, if $A$ is related to $B$, what relates to $C$ in the same way? The difference vector $(B - A)$ captures the relationship. Adding that vector to $C$ predicts the target word vector $D$:

$$D = C + (B - A)$$

An embedding maps words into a continuous vector space where words with similar meanings or contexts are grouped close together.

Example: Part-of-Speech (POS) Classification

A classic example is POS tagging: given a sentence, classify each word into its respective part of speech. Manually annotating or rule-coding every variation is tedious and brittle, so the model learns representations automatically:

1. Window Size ($W$): Choose an odd context window width (e.g., $W = 5$: the target word plus 2 words to the left and 2 to the right).

2. Vocabulary ($V$): Build and sort a vocabulary of unique tokens.

3. Embedding Dimension ($d$): Choose an embedding dimension size $d$.

4. Embedding Matrix ($E$): Create an embedding matrix $E$ of dimension $\vert{}V\vert{} \times d$, where each row corresponds to the embedding vector of a specific word. Initialize $E$ randomly.

5. Feed-Forward Network: Build a neural network with hidden layers and weight matrices to map the input window to the POS label of the center word.

6. Input Representation: Concatenate the embeddings of the $W$ words in the window into an input vector $X$ of length $W \times d$.

7. Training: Train the model using gradient descent and backpropagation to update both the hidden layer weights and the embedding matrix $E$ until classification error is minimized.

25.2 Neural Networks for NLP

A fixed-window feed-forward classifier only sees local context (e.g., 2 words back and 2 words forward). Natural language context, however, can appear anywhere: at the start, middle, or end of a sentence. A fixed window fails to capture long-distance dependencies, which is where Recurrent Neural Networks (RNNs) come in.

25.2.1 Recurrent Neural Networks (RNNs)

Instead of taking a concatenated window of tokens at once, an RNN processes sequential tokens step by step. Each word $x_t$ is looked up in the embedding matrix and passed into a hidden state $h_t$:

$$h_t = \text{RNN}(h_{t-1}, x_t)$$

The output layer then applies a softmax over classes (for POS tagging or next-word prediction).

Advantages of an RNN:

• Shared Parameters: The transition weights are identical across all time steps, keeping the parameter count independent of sequence length.

• Position Invariance / Symmetry: It processes tokens using the same recurrence relation regardless of where they appear in the sentence.

• Variable-Length Context: In theory, the hidden state retains a running summary of all prior words.
  

Limitations:

In practice, an RNN has limited memory capacity. Due to vanishing and exploding gradients, information from words early in a long sequence tends to decay or get distorted by the time it reaches later steps.

Text Generation:

Once trained as a language model, an RNN can generate text autoregressively: feed an initial word $x_1$, compute the softmax distribution $y_1$, sample the next word $x_2$ from this distribution, feed $x_2$ back as input, and repeat.

25.2.2 Bidirectional RNNs

A standard left-to-right RNN only conditions on past words, missing future context that appears to the right of the target word. A Bidirectional RNN (BiRNN) solves this by using two separate hidden layers:

• One running forward (left-to-right).

• One running backward (right-to-left).

The final hidden state at step $t$ is formed by concatenating the two directions: $[\overrightarrow{h_t}; \overleftarrow{h_t}]$, capturing both preceding and subsequent context.

25.2.3 LSTMs for NLP Tasks

Standard RNNs suffer heavily from the vanishing gradient problem, causing early context to be forgotten or distorted over long sequences. The Long Short-Term Memory (LSTM) network addresses this by introducing a dedicated memory cell ($c_t$) along with gating mechanisms (forget gate, input gate, output gate):

• The cell state acts as an internal highway, carrying linear information forward across time steps.

• The gates learn what to remember, what to discard, and what to expose to the hidden state.

• This allows the model to preserve long-range dependencies far more effectively than a standard RNN.

25.3 Sequence-to-Sequence Models

Sequence-to-sequence (Seq2Seq) architectures are designed for tasks where the input and output lengths can differ, such as machine translation. Machine translation cannot rely on simple 1-to-1 word alignment because word order differs across languages and words carry multiple meanings based on context.

A Seq2Seq model handles this using two main components:

1. Encoder RNN: Reads the source sentence token by token and compresses it into a final hidden vector (the "thought vector" or context vector).

2. Decoder RNN: Uses that final hidden state as its initial hidden state and generates target words step by step until an end-of-sequence token is emitted.
   

Shortcomings of Classic Seq2Seq:

1. Bottleneck Problem: Compressing an entire sentence into a single fixed-size vector loses critical information, especially for long sentences.

2. Nearby Context Bias: Recurrent decoders still struggle to retain distant source signals.

3. Sequential Bottleneck: Step-by-step computation cannot be parallelized across time.

25.3.1 Attention Mechanism

To solve the fixed-vector bottleneck, the attention mechanism lets the decoder look directly at all encoder hidden states ($s_1, s_2, \dots, s_n$), not just the final one.
At each decoder step $i$:

1. Raw Attention Scores ($e_{ij}$ or $r_{ij}$): Measure how well the previous decoder state $h_{i-1}$ matches each source encoder state $s_j$ (e.g., using a dot product):

   $$r_{ij} = h_{i-1} \cdot s_j$$

2. Attention Weights ($\alpha_{ij}$): Normalize scores across all source positions using softmax so they form a probability distribution summing to 1:

   $$\alpha_{ij} = \frac{\exp(r_{ij})}{\sum_{k} \exp(r_{ik})}$$

3. Context Vector ($c_i$): Compute a weighted sum of the source hidden states:

   $$c_i = \sum_{j} \alpha_{ij} s_j$$

4. Decoder Update: Concatenate the context vector $c_i$ with the target input $x_i$ to update the decoder state:

   $$h_i = \text{RNN}(h_{i-1}, [x_i; c_i])$$

Why Softmax?

• It is differentiable, allowing end-to-end backpropagation through the entire network.

• It dynamically selects relevant source tokens while suppressing irrelevant ones.

• It captures uncertainty by spreading probability mass over multiple plausible words.

25.3.2 Decoding

During inference, the decoder generates words autoregressively. Greedy Decoding: At each step, pick the token with the highest predicted softmax probability and feed it as input to the next step. While fast, it can get stuck in sub-optimal generation paths since an early mistake cannot be revised.

25.4 The Transformer Architecture

The Transformer replaces recurrence entirely with attention mechanisms, enabling full parallelization across sequence length during training.

25.4.1 Self-Attention

In Seq2Seq attention, target tokens attend back to source tokens. In self-attention, every token within the same sequence attends to all other tokens in that sequence, capturing dependencies regardless of distance. To prevent simple dot-product self-attention from collapsing into a biased identity lookup, the model projects each input vector into three distinct spaces using learned projection matrices:

• Query ($Q$): What the current token is looking for.

• Key ($K$): What other tokens offer to match against.

• Value ($V$): The actual content representation to extract.

The scaled dot-product attention formula is:

$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$

• The scaling factor $\frac{1}{\sqrt{d_k}}$ prevents dot products from growing excessively large, keeping softmax gradients stable.

• Multi-Head Attention: Instead of computing a single attention distribution, queries, keys, and values are linearly projected into $h$ distinct subspaces. Each head attends to different contextual relationships, and their outputs are concatenated and linearly projected back to the model dimension.

25.4.2 From Self-Attention to Transformer

A standard Transformer layer consists of:

1. Multi-Head Self-Attention.

2. Residual connections around each sublayer followed by Layer Normalization: $\text{LayerNorm}(x + \text{Sublayer}(x))$.

3. Position-wise Feed-Forward Networks (FFN) with non-linear activations (such as ReLU or GELU).

Because the Transformer contains no recurrent steps, it is order-agnostic. To give the model a sense of word order, Positional Embeddings are added directly to the word embeddings before the first layer:

$$\text{Input}_t = \text{WordEmbedding}_t + \text{PositionalEmbedding}_t$$

• Encoder: Stacks self-attention and FFN layers to encode bidirectional context (useful for classification tasks).

• Decoder: Stacks masked self-attention (preventing tokens from attending to future tokens), cross-attention (attending over encoder outputs), and FFN layers (used for generation tasks).

25.5 Pre-training and Transfer Learning

Pre-training uses large amounts of unlabeled text to train a general language representation. Through transfer learning, that pre-trained model can be fine-tuned on smaller, task-specific labeled datasets.

25.5.1 Pre-trained Word Embeddings (GloVe)

Unsupervised pre-trained embeddings differ from supervised task-specific embeddings because they leverage global corpus statistics without human labels.

GloVe (Global Vectors for Word Representation):

GloVe is based on matrix factorization of word co-occurrence counts within a sliding window. It models the ratio of co-occurrence probabilities:

• Let $P(k \mid w)$ be the probability of word $k$ occurring near word $w$.

• Ratios like $\frac{P(k \mid \text{ice})}{P(k \mid \text{steam})}$ reveal semantic distinctions:

  • If $k = \text{solid}$, the ratio is large.

  • If $k = \text{gas}$, the ratio is very small.

  • If $k = \text{water}$ or $k = \text{fashion}$, the ratio is close to 1.

• The dot product of two word vectors is trained to approximate the log of their co-occurrence probability:

$$e_i \cdot \tilde{e}_j + b_i + \tilde{b}_j \approx \log(X_{ij})$$

25.5.2 Pre-trained Contextual Representations

Static word embeddings (Word2Vec, GloVe) assign a single vector to each word form. They cannot differentiate distinct meanings (e.g., "rose" as a flower vs. "rose" as the past tense of "rise"). Contextual representation models process the token alongside its surrounding sentence, producing an embedding that dynamically shifts based on context.

25.5.3 Masked Language Models (MLM)

Unidirectional language models only attend to preceding tokens (left-to-right). A Masked Language Model (MLM) (such as BERT) masks a percentage of input tokens at random (e.g., replacing them with [MASK]) and trains a bidirectional Transformer to predict the masked identity based on both left and right context simultaneously. Because the sentence itself provides the ground-truth target, MLM requires no manual data annotation and produces deep bidirectional contextual representations.

25.6 State of the Art

• NLP's ImageNet Moment: Pre-trained transfer learning (Word2Vec in 2013, GloVe in 2014, and Transformer-based models later on) allowed researchers to adapt large general-purpose models to specific tasks without training from scratch.

• RoBERTa: An optimized, robustly trained variant of BERT that improved performance on reading comprehension and question-answering benchmarks.

• GPT-2: A 1.5-billion-parameter causal Transformer trained on 40 GB of web text, demonstrating strong zero-shot task transfer (translation, summarization, general question answering) without task-specific fine-tuning.

• Aristo: An ensemble system combining multiple solvers (information retrieval, rule-based reasoning, textual entailment) capable of passing standardized science multiple-choice exams, though limited in dealing with open-ended diagrams or free-form essays.

• T5 (Text-to-Text Transfer Transformer): Formulates all NLP tasks as text-to-text problems using a unified encoder-decoder architecture pre-trained on the 750 GB C4 (Colossal Clean Crawled Corpus) dataset.

Open Questions in NLP

• Does simply adding more textual data continuously scale performance, or are there diminishing returns?

• How can models effectively integrate multimodal and structured inputs (relational databases, sensor data, images, video)?

• How can models cost-effectively scale beyond limited context lengths to process book-length or repository-scale documents?

• Can models be augmented with explicit symbolic parsing and formal semantic representations, rather than relying purely on statistical text data?

Summary

• Word Embeddings: Provide dense, distributed representations of words learned in an unsupervised manner from raw text.

• RNNs: Handle variable-length sequences by sharing parameters across time steps, but suffer from limited long-term memory.

• Seq2Seq Models: Pair an encoder with a decoder to map one sequence to another, forming the basis of neural machine translation.

• Attention & Transformers: Attention mechanisms eliminate recurrent bottlenecks, allowing direct modelling of relationships across any sequence distance with parallel training.

• Transfer Learning & Pre-training: Pre-training on massive unlabeled corpora followed by downstream fine-tuning defines modern NLP pipelines.

Notable Research Papers / Systems Considered in This Chapter

• Word2Vec

• GloVe

• FastText

• T5

Follow me on LinkedIn