CS546 Learning Notes: From TagLM and ELMo to Seq2Seq Decoding — Reviewing Contextual Embeddings

1. Text classification: CNNs and RNNs differ only in the middle step

Slides 3–8 show how CNNs and RNNs do text classification. I first sorted out the pipeline they share:

1. Turn the sentence into a matrix, one embedding per row.
2. Extract features with a CNN or an RNN, giving a vector at each position.
3. Squash those vectors into one fixed-length sentence vector and feed it to a softmax.

The two models differ only in step 2.

CNN: the filter slides down the sentence

The example on slides 5–6 is "I like this movie very much !": 7 words, 5 dimensions each, so a $7\times5$ matrix. The yellow $2\times5$ matrix is a filter. Height 2 means it looks at two adjacent words at a time (a bigram); width equals the embedding size, so it can only slide along the sentence.

I computed the first value by hand. With the filter over "I like", multiply element-wise and sum:

Slide down one row to cover "like this" and you get 0.53. Slide all the way down and you get $7-2+1=6$ values: that's the feature map on the right.

The slides stop there, but the standard next step is max pooling over the feature map: each filter keeps a single number, meaning "how strongly this n-gram pattern shows up anywhere in the sentence." Use many filters of different heights, concatenate their outputs, and you have the sentence vector. Max pooling also makes the output length independent of sentence length.

RNN: take the last hidden state, or pool over all of them

• Slide 7, right (Lee & Dernoncourt 2016): pool over all $\mathbf h_t$ to get the sentence vector.
• Slide 8 (Ravuri & Stolcke 2015) shows two variants. On the left, a plain RNN outputs a class at every step and the last one counts. On the right, the LSTM outputs a class once, after reading </s>.

2. Multi-task sequence tagging: shared lower layers, separate output heads

Slide 21 (Tafforeau et al. 2016) shows the French sentence "oui il est ouvert". Each word gets two outputs: one row is part-of-speech (adv, cln, v, vppart), the other is semantic frame labels (Agreement, B_A#Closure, null, Closure).

The model has two parts:

• Shared lower layers: embeddings plus a bidirectional RNN, so each word's representation carries context from both sides.
• One output head per task: each task has its own softmax, since the two label sets are completely different.

The goal is on the slide: use a task with lots of data (POS tagging) to help a task with little data (semantic tagging). Training usually alternates batches between tasks. For a batch from one task, only that head's loss is computed, but the gradient still updates the shared layers. So a sentence doesn't need both label sets.

Compared with section 1: text classification gives one label per sentence and needs pooling; sequence tagging gives one label per word, so a classifier sits directly on each position's hidden state.

3. TagLM: train an LM on unlabeled text, then concatenate its vectors into a tagger

Slides 25–27 cover TagLM (Peters et al. 2017), which the slides call "Pre-ELMo". It has three steps.

Step 1: pretrain on unlabeled data. On one side, ordinary static word embeddings. On the other, a bidirectional LM made of a forward LSTM language model and a backward one, trained separately on about 800M words from the 1B Word Benchmark. The LM is frozen afterwards.

Step 2: give each word two representations. Take "York" in "New York is located …":

• Word embedding: the static vector for York, the same in every sentence.
• LM embedding: run the whole sentence through the frozen bidirectional LM and take the top-layer hidden states of the forward and backward LMs at York's position, concatenated:

At this position the forward LM has already seen "New" and the backward LM has already seen "is located …", so this vector can say "this York is part of the place name New York."

Step 3: the sequence tagger. A two-layer bidirectional RNN. The LM embedding is concatenated onto the first layer's output before it goes into the second layer, and a CRF on top outputs BIOES tags: New → B-LOC, York → E-LOC. This step is trained on the CoNLL 2003 labeled data, only about 200K tokens.

My summary at the time was: "first train an embedding model and an RNN separately; the embeddings give word vectors, the RNN gives context, then it outputs tags." Mostly right, but two corrections:

• The RNN LM doesn't produce a separate "context." It produces one context-aware vector per word.
• There's also a tagger in between (BiRNN + CRF) that has to be trained on labeled data. That's what outputs the tags.

4. Two follow-ups: what is a CRF, and why does semi-supervised still use labels?

CRF

If each word gets its own softmax, you can end up with a sequence where every tag looks fine on its own but the whole is invalid, like O directly followed by E-LOC. A CRF scores the entire tag sequence:

Training maximizes the probability of the correct sequence; prediction uses Viterbi to find the highest-scoring sequence.

Semi-supervised doesn't mean no labels

At first I assumed semi-supervised meant not using labels. It actually means using a lot of unlabeled data plus a little labeled data:

In TagLM, the LM part uses only raw text: "the next word" is itself the training target, which is what we'd now call self-supervised. The NER part needs human labels, otherwise the model can't know what LOC or PER means. The point of TagLM is that unlabeled data lets you get good results from only a small amount of labeled data.

5. So what is ELMo, really?

I stared at the figure on slide 29 for a while without getting it. What finally clicked: ELMo is a tool that computes word vectors per sentence. The same word gets a different vector in different sentences.

The problem it solves is word2vec's: one fixed vector per word. The "bank" in "deposit money at the bank" and in "sat on the river bank" get the same vector, so the model can't tell the two senses apart.

The figure shows training, read from the bottom up:

Only the forward half is drawn. There's also a backward LM with the same structure that reads right to left. After training, the orange softmax layer is thrown away and only the lower layers are used to produce vectors.

My second summary was: "what it outputs in the end is still a word vector, just a better one than the original word embedding." That's right. Two additions:

• It isn't a lookup table. Every sentence has to be run through the Bi-LM to get the vectors.
• It's usually concatenated, not substituted: $[\mathbf x_k;\text{ELMo}_k]$. The original static embedding stays.

6. The Bi-LM objective: forward and backward log-likelihoods summed

Slides 30–31 first write two factorizations:

Both are just the chain rule in different orders, and both equal the same sentence probability.

The training objective adds the two directions' log-likelihoods and maximizes them jointly:

I asked whether this is the joint maximum likelihood of forward and backward, and yes, it is. One detail that's easy to miss:

The yellow highlights in the slide's figure mark exactly the two shared parts.

Also worth remembering: "bidirectional" here means two unidirectional models predicting separately, with their losses added. When the forward LSTM predicts $t_k$ it only sees the left; the backward one only sees the right. Information from both sides meets only later, when the hidden states are concatenated. BERT later called this shallowly bidirectional and switched to a masked LM so every layer sees both sides at once.

7. ELMo's weighting formula: one weight per layer, learned by the task

Slide 32 is ELMo's actual definition:

Here $L=2$, so three layers:

Layer 0 duplicates $\mathbf x_k$ so its dimension matches the two layers above.

• $s_j$: one scalar per layer, passed through softmax so $s_j\ge0$ and $\sum_j s_j=1$. It says which layer this task relies on more.
• $\gamma$: one scalar that rescales the whole vector to suit the task model.
• Optional layer normalization: normalize each layer's values before weighting.

"Task-specific" means the Bi-LM is frozen and shared by all tasks; each downstream task learns only 4 extra numbers, $s_0,s_1,s_2,\gamma$, trained along with the task model.

Why use every layer? The paper found lower layers lean toward syntax (useful for POS tagging) and higher layers toward semantics (useful for word sense disambiguation). Rather than choose a layer by hand, let the task learn it.

Here's how I'd draw the whole thing:

                     Task model (e.g. NER)
                            ↑
                    [ x_k ; ELMo_k ]
                            ↑
          ELMo_k = γ · ( s0·h_k0 + s1·h_k1 + s2·h_k2 )     ← s, γ learned per task
               ↑              ↑              ↑
  ┌─────────── Frozen Bi-LM ─────────────────────────────┐
  │   h_k2 = [ →LSTM2 ; ←LSTM2 ]                          │
  │   h_k1 = [ →LSTM1 ; ←LSTM1 ]                          │
  │   h_k0 = [ x_k ; x_k ]      ← char CNN, no context     │
  └───────────────────────────────────────────────────────┘
                            ↑
                           t_k
   (the pretraining softmax output layer is discarded)

8. I said "ELMo just concatenates a context embedding onto TagLM" — not quite

At this point I summarized ELMo as giving TagLM's supervised part an extra context representation. The problem: TagLM already concatenated a context vector. ELMo changed two things on top of that:

A more accurate version: TagLM introduced "pretrain a Bi-LM, freeze it, concatenate its context vectors into a supervised model"; ELMo switched to a multi-layer weighted mix and generalized it into contextual word vectors any task can plug in.

9. Encoder: squash the whole sentence into a context vector

The encoder on slide 47 is just an ordinary RNN:

The simplest $f$ is $\tanh(W\mathbf x_t+U\mathbf h_{t-1}+\mathbf b)$; in practice it's usually an LSTM or GRU, with $\mathbf h_0$ set to zeros. The last hidden state after reading "omg the house is on fire <eos>" is the context vector $\phi$.

This is exactly "classify from the last hidden state" from slide 8. The only difference is that $\phi$ feeds another RNN instead of a classifier.

The weakness is obvious too: however long the sentence, it all has to fit in one fixed-length vector. That's the information bottleneck, and attention is what later addresses it.

10. Decoder: take $\phi$ and generate one word at a time

Slides 48–49. At step $t'$ the decoder:

1. Updates its hidden state: $\mathbf s_{t'}=g(\mathbf y_{t'-1},\phi,\mathbf s_{t'-1})$. The first input is <bos>, and the initial state is often just $\mathbf s_0=\phi$. The slide stresses that the encoder and decoder have the same number of layers and hidden units, which is what makes that direct assignment possible.
2. Scores every word in the vocabulary with a dense layer: $\mathbf o_{t'}=W_o\mathbf s_{t'}+\mathbf b_o\in\mathbb R^{|V|}$.
3. Applies softmax to get $P(y_{t'}\mid y_1,\dots,y_{t'-1},\phi)$.
4. Picks a word and feeds it in as the next step's input.

The training loss is the sum of per-step cross entropy:

11. I mixed up "training vs. inference" with "encoder vs. decoder"

In the previous section I mentioned teacher forcing during training and feeding the model's own outputs back during inference, and I took that to mean "the encoder is for training and the decoder is for inference." That's wrong. The encoder and decoder are two halves of one model, and both are used in both phases.

The only difference between the two phases is where the decoder's previous word comes from.

12. Greedy search: best at each step is not best overall

Greedy search on slide 50 takes the most probable word at each step:

It stops at <eos> or when the length hits the limit $T$.

But what we actually want is the most probable whole sequence:

A two-step example convinced me:

Once greedy picks A, it can't go back. And exhaustive search over all sequences is $O(|Y|^{T'})$, far too many, so we need a compromise.

13. Beam search: re-select the top k at every step, globally

Looking at the $k=2$ figures on slides 55–57, I thought beam search picks two paths only at the first step and then each path runs greedy. That's also wrong.

At every step it does this:

1. Each of the $k$ kept candidates expands by all $|V|$ words, giving $k\times|V|$ candidate sequences.
2. All of them are ranked together by sequence probability $P(y_1,\dots,y_{t'})$.
3. The global top $k$ are kept.

In the figure, step 2 keeps AB and CE, one from each branch, which is why it looks like per-branch greedy. But the top 2 could easily come from the same branch:

Here AB and AD survive and the whole C branch is dropped.

How it ends: when a candidate produces <eos>, it moves to a "finished" list and the rest keep expanding. Once enough are finished or the max length is reached, the best finished sequence wins. In practice you compare summed log probabilities; since each extra word adds a negative number, short sequences have a built-in advantage, so scores are often length-normalized.

$k=1$ is greedy. Larger $k$ gets closer to exhaustive search, and slower.

14. Hierarchical RNNs: words make utterances, utterances make a dialogue

The two formulas on slide 67:

• Sentence encoding: $x_t$ is the $t$-th utterance in the dialogue. A word-level RNN reads it and produces the utterance vector $m_t$. Every utterance goes through the same $\text{RNN}_1$.
• Context encoding: $f$ is another RNN whose inputs are utterance vectors instead of words. $c_t$ summarizes utterances 1 through $t-1$, i.e. the state of the conversation before utterance $t$.

Watch the indices: $c_t$ uses $m_{t-1}$ and doesn't include the current utterance. It's used together with the current utterance to generate a response, playing the role of $\phi$ in plain S2S.

Why not concatenate the whole dialogue into one long sequence? It's too long: the RNN forgets early content and can't see turn boundaries. With the hierarchy, the lower level handles a few dozen words per utterance and the upper level handles a dozen or so turns, so both paths stay short.

15. VHRED: first sample "what to say," then say it word by word

Slides 73–74. HRED's responses tend to be bland, like "I don't know" no matter what you say. The reason is that its only randomness comes from picking words one at a time; nothing is dedicated to deciding "what topic and tone should this reply have."

VHRED adds a latent variable $z_n$ for each response. Generation has two steps.

Prior: sampled from at inference time. It only sees the dialogue history:

Then $c$ and $z_n$ together form the decoder's initial state, and the reply is generated word by word.

Posterior: used only during training. The dashed line in the figure shows it also sees the real reply $x_n$:

The training objective is the ELBO:

The first term teaches the decoder to use $z$ to reconstruct the real reply. The second pulls the history-only prior toward the posterior that saw the answer. Sampling isn't differentiable, so the reparameterization $z=\mu+\sigma\odot\epsilon$ handles that.

At inference, sampling different $z$ for the same history yields different replies. That's what the slide means by "injecting stochastic noise." Long-term memory still lives in the utterance-level RNN's $c$; $z$ only governs the current reply.

In practice training often hits KL collapse: the decoder is strong enough to ignore $z$. KL annealing is a common fix.

16. Nucleus sampling: sample randomly, but cut off the long tail

The left figure on slide 76 compares beam search with human text. Under beam search almost every word has probability near 1, and the text repeats the same sentence over and over; in human text, the model's probability for each word goes up and down.

Sampling from the full vocabulary doesn't work either. Tens of thousands of low-probability words add up, and drawing one derails the sentence.

Nucleus (top-p) sampling:

1. Sort words by probability, highest first.
2. Accumulate from the top and take the smallest set $V^{(p)}$ whose total reaches $p$:

1. Renormalize within $V^{(p)}$ and sample one word.

With $p=0.9$: fine 0.50, good 0.25, okay 0.10, great 0.06. The running total hits 0.91, so we stop there and drop every other word.

The difference from top-k is that the candidate set changes size on its own. When the model is confident, one or two words may be enough; when it's uncertain, the set grows. $p$ is usually 0.9–0.95, often combined with temperature.

17. A quiz question: clipping can't fix vanishing gradients

Which of the following aims to help with the vanishing gradients problem of RNNs? (A) Gradient clipping (B) Gating mechanisms (C) Weight decay (D) A and B (E) None

My first instinct was D, but then I suspected clipping only handles explosion. The answer is indeed B.

Going backward, the gradient gets multiplied by a Jacobian at every step:

• Clipping: when $\|\mathbf g\|>\theta$, scale the gradient down to $\theta$. It makes large gradients smaller; it never makes small ones larger.
• Gating: in an LSTM, $\mathbf c_t=\mathbf f_t\odot\mathbf c_{t-1}+\mathbf i_t\odot\tilde{\mathbf c}_t$, so $\partial\mathbf c_t/\partial\mathbf c_{t-1}=\text{diag}(\mathbf f_t)$. With the forget gate near 1, the gradient barely decays.
• Weight decay: pulls weights toward 0; if anything, that makes vanishing gradients worse.

18. What I most need to remember

• In text classification, CNNs and RNNs differ only in the feature extraction step; both end by squashing everything into one sentence vector.
• Multi-task learning: shared lower layers, one output head per task, so a data-rich task helps a data-poor one.
• TagLM: pretrain a Bi-LM, freeze it, concatenate its top-layer context vectors into the tagger. Semi-supervised means unlabeled plus labeled, not no labels.
• ELMo computes word vectors per sentence. $\text{ELMo}_k=\gamma\sum_j s_j\mathbf h_{k,j}$; the Bi-LM is frozen and each task learns only 4 numbers.
• The Bi-LM's "bidirectional" is two unidirectional losses added; the two sides meet only when concatenated.
• The encoder squashes the sentence into $\phi$; the decoder generates word by word from $\phi$. Training and inference differ only in whether the decoder's previous word is the reference or its own output.
• Beam search picks the global top k among $k\times|V|$ candidates at every step; $k=1$ is greedy.
• HRED has two levels: words → utterance vector → dialogue state. VHRED adds a latent $z$ to decide what to say before saying it word by word.
• Nucleus sampling: sample randomly, but only among the smallest set of words whose total probability reaches $p$.
• Clipping handles explosion; gating handles vanishing.

Sources and slide mapping

This post is based on my Lecture 6 review conversation and Lecture6_ContextualEmbeddings_ModelArchitectures_Decoding.pdf, reorganized by the questions I actually asked.

• CNNs / RNNs for text classification: PDF slides 3–8; the CNN example is from the joshuakim.io illustration cited in the slides.
• Multi-task sequence tagging: slide 21 (Tafforeau et al., Interspeech 2016).
• TagLM: slides 25–28 (Peters et al., ACL 2017).
• ELMo: slides 29–38 (Peters et al., NAACL 2018).
• Encoder-decoder, greedy and exhaustive search: slides 44–52.
• Beam search: slides 54–58.
• Hierarchical RNNs: slides 67–68 (Serban et al., AAAI 2016).
• VHRED: slides 73–74 (Serban et al., 2016).
• Nucleus sampling: slides 75–77 (Holtzman et al., ICLR 2020).
• Vanishing gradients, clipping, and LSTMs: see my earlier post and the Lecture 5 notes.