1 00:00:00,588 --> 00:00:02,233 Here's something that works. 2 00:00:02,233 --> 00:00:05,500 Take a language model, and train it on text written by a 3 00:00:05,500 --> 00:00:07,443 completely different model. 4 00:00:07,443 --> 00:00:10,057 Different weights, different architecture, 5 00:00:10,057 --> 00:00:12,642 and no correction of any kind. 6 00:00:12,642 --> 00:00:15,530 It's called distillation, and it's stable. 7 00:00:15,902 --> 00:00:19,466 Now take that same model, and train it by reward, 8 00:00:19,466 --> 00:00:21,953 on attempts written by a copy of itself. 9 00:00:21,953 --> 00:00:24,349 Same weights, same code. 10 00:00:24,349 --> 00:00:26,981 The only difference is that the copy doing the writing 11 00:00:26,981 --> 00:00:30,750 stores its numbers more roughly, to run faster and save 12 00:00:30,750 --> 00:00:31,580 memory. 13 00:00:31,970 --> 00:00:35,812 This is a 30 billion parameter model learning math, 14 00:00:35,812 --> 00:00:38,563 with that rounding turned up on purpose. 15 00:00:38,563 --> 00:00:42,750 In under two hundred steps, its accuracy falls to zero. 16 00:00:44,019 --> 00:00:46,305 A completely different model as a teacher: 17 00:00:46,305 --> 00:00:49,526 fine. A slightly blurry copy of yourself: 18 00:00:49,526 --> 00:00:52,820 fatal. You'd expect it to be the other way around. 19 00:00:53,160 --> 00:00:56,360 The reason is one hidden term in the math of how these 20 00:00:56,360 --> 00:00:58,360 models learn from reward. 21 00:00:58,360 --> 00:01:00,870 The paper that found it calls it drift. 22 00:01:02,810 --> 00:01:04,973 It's a short paper from this September, 23 00:01:04,973 --> 00:01:08,910 and it also removes drift, with a single subtraction. 24 00:01:08,910 --> 00:01:11,690 We'll build up how learning from reward works, 25 00:01:11,690 --> 00:01:15,130 find where drift hides, see why a copy of yourself makes 26 00:01:15,130 --> 00:01:17,720 it run away, and test the fix. 27 00:01:21,645 --> 00:01:24,640 Start with how reinforcement learning trains one of these 28 00:01:24,640 --> 00:01:26,397 models. Give it a puzzle: 29 00:01:26,397 --> 00:01:29,680 use these four numbers to make 65. 30 00:01:29,680 --> 00:01:33,280 Let it write out several attempts, one token at a time. 31 00:01:33,280 --> 00:01:35,103 Check each final answer. 32 00:01:35,103 --> 00:01:37,596 Right or wrong: that's the reward. 33 00:01:37,596 --> 00:01:39,999 Now go back through every attempt. 34 00:01:39,999 --> 00:01:42,880 Every token in a winning attempt gets nudged to be a 35 00:01:42,880 --> 00:01:44,861 little more likely next time. 36 00:01:44,861 --> 00:01:48,513 Every token in a losing attempt, a little less likely. 37 00:01:48,513 --> 00:01:51,920 Repeat thousands of times, and the model gets better at 38 00:01:51,920 --> 00:01:52,830 the puzzle. 39 00:01:53,557 --> 00:01:56,767 Zoom in on one moment in one attempt. 40 00:01:56,767 --> 00:01:58,850 The model has written part of an answer, 41 00:01:58,850 --> 00:02:02,210 and for the next token, it holds a probability for every 42 00:02:02,210 --> 00:02:04,857 token it knows. It picks one at random, 43 00:02:04,857 --> 00:02:06,842 according to those probabilities. 44 00:02:06,842 --> 00:02:10,074 Now, somewhere in the model's billions of weights, 45 00:02:10,074 --> 00:02:12,290 there's a direction you could move them all that would 46 00:02:12,290 --> 00:02:15,410 make that exact token a bit more likely. 47 00:02:15,410 --> 00:02:18,515 That direction has a wonderfully understated name: 48 00:02:18,515 --> 00:02:19,680 the score. 49 00:02:20,020 --> 00:02:21,837 So here's the whole method. 50 00:02:21,837 --> 00:02:25,460 For every token in every attempt, take its score, 51 00:02:25,460 --> 00:02:27,780 multiply it by the attempt's reward, 52 00:02:27,780 --> 00:02:30,397 and add it all up. That's the update. 53 00:02:30,397 --> 00:02:33,380 In reinforcement learning, a model's way of choosing is 54 00:02:33,380 --> 00:02:36,980 called its policy, so this is the policy gradient. 55 00:02:36,980 --> 00:02:39,220 It dates from 1992. 56 00:02:39,220 --> 00:02:41,943 The algorithms that train today's reasoning models, 57 00:02:41,943 --> 00:02:46,370 with names like PPO and GRPO, are all built on it. 58 00:02:47,044 --> 00:02:49,190 To see what the score actually does, 59 00:02:49,190 --> 00:02:52,251 shrink the vocabulary to just three tokens: 60 00:02:52,251 --> 00:02:55,030 plus, minus and times. 61 00:02:55,030 --> 00:02:58,390 Now any next-token distribution is just a point in a 62 00:02:58,390 --> 00:03:01,738 triangle. At a corner, the model is certain: 63 00:03:01,738 --> 00:03:04,069 one hundred percent plus. 64 00:03:04,069 --> 00:03:06,389 In the middle, it can't decide. 65 00:03:06,389 --> 00:03:08,853 Call the model's current point p. 66 00:03:08,853 --> 00:03:13,350 Here, it says plus half the time, minus thirty percent, 67 00:03:13,350 --> 00:03:14,980 and times twenty. 68 00:03:15,320 --> 00:03:19,049 Now the score. Say the model just wrote plus. 69 00:03:19,049 --> 00:03:23,000 The most extreme version of "more plus" is one hundred 70 00:03:23,000 --> 00:03:25,388 percent plus: the plus corner. 71 00:03:25,388 --> 00:03:29,000 So on this triangle, the score for a token is an arrow 72 00:03:29,000 --> 00:03:31,555 from p to that token's corner: 73 00:03:31,555 --> 00:03:34,120 from what the model predicted, to what actually got 74 00:03:34,120 --> 00:03:38,040 written. If the attempt wins, take a small step toward 75 00:03:38,040 --> 00:03:41,430 that corner. If it loses, step the other way. 76 00:03:41,770 --> 00:03:44,166 That's all a single update is: 77 00:03:44,166 --> 00:03:46,331 an arrow toward the token that was written, 78 00:03:46,331 --> 00:03:48,250 scaled by the reward. 79 00:03:48,250 --> 00:03:51,050 The real model moves billions of weights, 80 00:03:51,050 --> 00:03:54,806 not a dot. But at any one point in an attempt, 81 00:03:54,806 --> 00:03:58,245 every arrow is turned into weight changes the same way, 82 00:03:58,245 --> 00:04:01,530 so whatever adds up or cancels here does the same in the 83 00:04:01,530 --> 00:04:03,857 weights. Hold on to this picture. 84 00:04:03,857 --> 00:04:07,080 Nearly every idea in the paper can be drawn on it. 85 00:04:10,660 --> 00:04:13,379 Now the catch, and it's the whole film. 86 00:04:13,379 --> 00:04:15,856 Running this takes two copies of the model. 87 00:04:15,856 --> 00:04:17,577 One writes the attempts. 88 00:04:17,577 --> 00:04:20,900 That's the sampler, running in software built for fast 89 00:04:20,900 --> 00:04:23,940 text generation. The other goes back over what was 90 00:04:23,940 --> 00:04:26,185 written, and computes the scores. 91 00:04:26,185 --> 00:04:27,490 That's the trainer. 92 00:04:27,830 --> 00:04:31,550 In theory they're the same model, with the same weights. 93 00:04:31,550 --> 00:04:33,845 Call the trainer's probabilities p, 94 00:04:33,845 --> 00:04:37,586 in gold, and the sampler's q, in blue. 95 00:04:37,586 --> 00:04:40,470 When the model that writes is exactly the model that 96 00:04:40,470 --> 00:04:43,106 learns, that's called on-policy. 97 00:04:43,106 --> 00:04:45,430 When they differ, it's off-policy. 98 00:04:45,430 --> 00:04:49,852 And in practice, p and q almost never match exactly. 99 00:04:49,852 --> 00:04:51,270 The gap has a name: 100 00:04:51,270 --> 00:04:53,860 the training-inference mismatch. 101 00:04:54,544 --> 00:04:55,865 Why don't they match? 102 00:04:55,865 --> 00:04:58,027 Partly because computers round. 103 00:04:58,027 --> 00:05:00,680 Here's the coarsest number format in this story: 104 00:05:00,680 --> 00:05:04,359 four bits. These are all the positive numbers it can say. 105 00:05:04,359 --> 00:05:06,423 A half plus a half is one. 106 00:05:06,423 --> 00:05:10,040 Plus two: three. Now add the same numbers in another 107 00:05:10,040 --> 00:05:13,399 order. Two plus a half would be two and a half, 108 00:05:13,399 --> 00:05:16,200 but that's not on the list, so it rounds to two. 109 00:05:16,200 --> 00:05:17,543 Add the other half: 110 00:05:17,543 --> 00:05:21,720 two again. Three, or two, from the same numbers. 111 00:05:21,720 --> 00:05:25,080 When every step rounds, the order of addition changes the 112 00:05:25,080 --> 00:05:26,070 answer. 113 00:05:26,457 --> 00:05:30,490 Real formats have far more values, so the errors are far 114 00:05:30,490 --> 00:05:33,130 smaller, but they're the same kind. 115 00:05:33,130 --> 00:05:36,170 And the two copies add things up in different orders: 116 00:05:36,170 --> 00:05:38,656 the sampler writes one token at a time, 117 00:05:38,656 --> 00:05:41,608 while the trainer reads the whole attempt at once. 118 00:05:41,608 --> 00:05:45,610 Forcing them to agree costs speed, so it's rarely done all 119 00:05:45,610 --> 00:05:46,440 the way. 120 00:05:47,180 --> 00:05:49,891 And sometimes the rounding is on purpose. 121 00:05:49,891 --> 00:05:52,943 Writing the attempts is the expensive part of training, 122 00:05:52,943 --> 00:05:56,380 so a popular speed trick is to let the sampler use rougher 123 00:05:56,380 --> 00:05:59,647 numbers. Here's what that does to a number line. 124 00:05:59,647 --> 00:06:02,182 Eight bits gives you a couple of hundred levels. 125 00:06:02,182 --> 00:06:04,780 Four bits, you can count them, like our friends from 126 00:06:04,780 --> 00:06:08,140 before. And the model's memory of everything it has read 127 00:06:08,140 --> 00:06:12,140 so far, called the KV cache, can be rounded harder than 128 00:06:12,140 --> 00:06:15,130 the weights. It's a lot of memory to save. 129 00:06:15,790 --> 00:06:18,560 There's one more way for the copies to disagree, 130 00:06:18,560 --> 00:06:20,477 even with perfect arithmetic: 131 00:06:20,477 --> 00:06:24,350 timing. To keep every chip busy, the sampler and the 132 00:06:24,350 --> 00:06:26,264 trainer don't wait for each other. 133 00:06:26,264 --> 00:06:29,550 So while the trainer takes a step, the sampler is still 134 00:06:29,550 --> 00:06:31,270 writing with older weights. 135 00:06:31,270 --> 00:06:35,150 One batch, or even one attempt, can come from several 136 00:06:35,150 --> 00:06:36,669 versions of the model. 137 00:06:36,669 --> 00:06:39,723 For AI agents whose tasks run for hours, 138 00:06:39,723 --> 00:06:40,940 it's worse. 139 00:06:41,333 --> 00:06:45,037 So the mismatch isn't a bug you can simply fix, 140 00:06:45,037 --> 00:06:47,760 not without giving up the speed that makes these systems 141 00:06:47,760 --> 00:06:51,390 practical. You need a way of learning that survives it. 142 00:06:54,969 --> 00:06:56,469 So what goes wrong? 143 00:06:56,469 --> 00:06:59,337 Go back to the strange fact from the opening: 144 00:06:59,337 --> 00:07:03,370 training on a totally different model's text is stable. 145 00:07:03,370 --> 00:07:06,010 And that kind of training is a special case of what we 146 00:07:06,010 --> 00:07:10,541 just built. Give winners a reward of one and losers zero, 147 00:07:10,541 --> 00:07:12,010 and the losers drop out: 148 00:07:12,010 --> 00:07:13,893 you're training on the winners. 149 00:07:13,893 --> 00:07:17,290 If the sampler also never changes, that's exactly 150 00:07:17,290 --> 00:07:19,800 distillation, from a fixed teacher. 151 00:07:20,141 --> 00:07:24,060 So only two things separate that stable setup from real 152 00:07:24,060 --> 00:07:25,661 reinforcement learning. 153 00:07:25,661 --> 00:07:29,260 The rewards: real systems use negative ones too. 154 00:07:29,260 --> 00:07:32,460 And the sampler: in real training, it gets the trainer's 155 00:07:32,460 --> 00:07:34,498 new weights after every step. 156 00:07:34,498 --> 00:07:37,178 So the authors turned each knob separately, 157 00:07:37,178 --> 00:07:39,578 on a small model doing the numbers puzzle, 158 00:07:39,578 --> 00:07:42,860 with the sampler's weights nudged so the copies disagree a 159 00:07:42,860 --> 00:07:45,237 little. Three reward schemes: 160 00:07:45,237 --> 00:07:48,300 one or zero; plus or minus one; 161 00:07:48,300 --> 00:07:51,665 and the one most real systems use, group-centered, 162 00:07:51,665 --> 00:07:54,460 where each attempt is scored against the average of its 163 00:07:54,460 --> 00:07:55,370 group. 164 00:07:55,950 --> 00:07:58,273 Each line is one training run. 165 00:07:58,273 --> 00:08:00,262 Left to right is training steps; 166 00:08:00,262 --> 00:08:04,270 up means the model solves more puzzles it has never seen. 167 00:08:04,270 --> 00:08:07,634 First, the sampler frozen, like distillation. 168 00:08:07,634 --> 00:08:11,307 One-or-zero trains fine, just as distillation would. 169 00:08:11,307 --> 00:08:14,270 The two schemes with negative rewards die almost 170 00:08:14,270 --> 00:08:17,710 instantly. Now let the sampler follow the trainer, 171 00:08:17,710 --> 00:08:19,228 as in real training. 172 00:08:19,228 --> 00:08:21,823 Group-centered becomes the best on the chart. 173 00:08:21,823 --> 00:08:24,030 Plus-or-minus-one sags. 174 00:08:24,030 --> 00:08:27,283 And one-or-zero, the only one that worked a moment ago, 175 00:08:27,283 --> 00:08:28,780 now collapses. 176 00:08:29,795 --> 00:08:32,278 The order has completely reversed. 177 00:08:32,278 --> 00:08:35,240 So the authors propose there are two separate problems 178 00:08:35,240 --> 00:08:39,000 here. Negative rewards failing with a frozen sampler has a 179 00:08:39,000 --> 00:08:40,680 simpler explanation: 180 00:08:40,680 --> 00:08:43,800 being pushed away from a fixed target never stops, 181 00:08:43,800 --> 00:08:47,576 because a probability can always be pushed closer to zero. 182 00:08:47,576 --> 00:08:50,358 That's a separate problem, so set it aside. 183 00:08:50,358 --> 00:08:52,466 The one we want is the other: 184 00:08:52,466 --> 00:08:55,295 something that makes training on the winners fall apart, 185 00:08:55,295 --> 00:08:57,510 but only once the sampler moves. 186 00:08:58,169 --> 00:09:01,025 So look closely at the setup that broke: 187 00:09:01,025 --> 00:09:04,568 one-or-zero rewards, sampler refreshed. 188 00:09:04,568 --> 00:09:06,810 The trainer is learning from the sampler's winning 189 00:09:06,810 --> 00:09:09,043 attempts. That's distillation. 190 00:09:09,043 --> 00:09:10,808 The sampler is the teacher. 191 00:09:10,808 --> 00:09:14,511 But this teacher is a slightly blurry copy of the student. 192 00:09:14,511 --> 00:09:17,850 And after every step, the student's new weights are copied 193 00:09:17,850 --> 00:09:19,480 back into the teacher. 194 00:09:19,869 --> 00:09:22,226 Compared with the distillation that works, 195 00:09:22,226 --> 00:09:24,376 only one thing has changed. 196 00:09:24,376 --> 00:09:26,410 The teacher moves with the student. 197 00:09:31,069 --> 00:09:32,893 Here's the cleanest way to see it. 198 00:09:32,893 --> 00:09:35,863 Give every attempt a reward of plus one. 199 00:09:35,863 --> 00:09:38,270 Right, wrong, gibberish: 200 00:09:38,270 --> 00:09:40,989 plus one. There's nothing to learn from that, 201 00:09:40,989 --> 00:09:43,864 so a correct training method should do nothing at all. 202 00:09:43,864 --> 00:09:46,380 On average, the model shouldn't move. 203 00:09:46,719 --> 00:09:49,839 Back to the triangle, with p at fifty, 204 00:09:49,839 --> 00:09:54,000 thirty, twenty. Let the model write ten tokens from its 205 00:09:54,000 --> 00:09:58,160 own probabilities: say, exactly five pluses, 206 00:09:58,160 --> 00:10:00,880 three minuses and two times. 207 00:10:00,880 --> 00:10:04,240 Every one gets the same reward, so every one becomes the 208 00:10:04,240 --> 00:10:05,868 same kind of nudge: 209 00:10:05,868 --> 00:10:08,720 an arrow from p toward its own corner. 210 00:10:08,720 --> 00:10:10,948 Now average the ten arrows. 211 00:10:10,948 --> 00:10:14,320 Shrink each to a tenth, and lay them end to end. 212 00:10:14,320 --> 00:10:16,750 You land right back where you started. 213 00:10:17,131 --> 00:10:21,090 Here's why. The average arrow points from p to the average 214 00:10:21,090 --> 00:10:22,690 of the corners you drew: 215 00:10:22,690 --> 00:10:25,649 five tenths plus, three tenths minus, 216 00:10:25,649 --> 00:10:28,551 two tenths times. That's p itself. 217 00:10:28,551 --> 00:10:32,161 You drew from p, so on average you land on p. 218 00:10:32,161 --> 00:10:35,158 So here's the rule for everything that follows: 219 00:10:35,158 --> 00:10:38,601 the overall pull is the arrow from where the arrows start, 220 00:10:38,601 --> 00:10:40,434 to where their average lands. 221 00:10:40,434 --> 00:10:42,508 If you write the attempts yourself, 222 00:10:42,508 --> 00:10:46,130 those are the same place, and on average the nudges cancel 223 00:10:46,130 --> 00:10:49,730 exactly, for any model, not just this toy. 224 00:10:49,730 --> 00:10:53,010 That's why a reward that's the same for everyone teaches 225 00:10:53,010 --> 00:10:53,840 nothing. 226 00:10:54,500 --> 00:10:56,218 Now add the mismatch. 227 00:10:56,218 --> 00:10:59,689 The sampler writes the tokens, so they come from q: 228 00:10:59,689 --> 00:11:02,580 here, about thirty-one percent plus, 229 00:11:02,580 --> 00:11:05,929 a quarter minus, and forty-four percent times. 230 00:11:05,929 --> 00:11:08,180 But the trainer computes the scores, 231 00:11:08,180 --> 00:11:11,220 so every arrow still starts from p. 232 00:11:11,220 --> 00:11:14,260 Draw a few hundred tokens, and keep a running average of 233 00:11:14,260 --> 00:11:15,939 where their arrows land. 234 00:11:15,939 --> 00:11:17,617 It doesn't settle on p. 235 00:11:17,617 --> 00:11:21,207 It settles on q, because that's the mix you drew from. 236 00:11:21,207 --> 00:11:26,456 So by our rule, the overall pull is an arrow from p to q. 237 00:11:26,456 --> 00:11:30,100 Even with nothing to learn, every step quietly pulls the 238 00:11:30,100 --> 00:11:31,859 trainer toward the sampler. 239 00:11:31,859 --> 00:11:34,470 The paper calls this pull drift. 240 00:11:34,470 --> 00:11:36,605 And look at what kind of pull it is. 241 00:11:36,605 --> 00:11:40,230 "Move your probabilities toward someone else's" is exactly 242 00:11:40,230 --> 00:11:41,918 what distillation does. 243 00:11:41,918 --> 00:11:45,657 So hidden inside every reinforcement learning step, 244 00:11:45,657 --> 00:11:47,511 there's a little bit of distillation, 245 00:11:47,511 --> 00:11:49,426 with the sampler as the teacher. 246 00:11:49,426 --> 00:11:52,857 If p and q match, the arrow has zero length, 247 00:11:52,857 --> 00:11:54,469 and the pull vanishes. 248 00:11:54,469 --> 00:11:58,660 That's why, on-policy, nobody ever had to think about it. 249 00:11:59,320 --> 00:12:01,387 Now give it real rewards. 250 00:12:01,387 --> 00:12:04,280 Take one point partway through an attempt, 251 00:12:04,280 --> 00:12:06,347 and the ways it could go on from there. 252 00:12:06,347 --> 00:12:10,839 Say four of them score one, one, one and zero. 253 00:12:10,839 --> 00:12:12,839 The average is three quarters. 254 00:12:12,839 --> 00:12:15,480 So split each reward into two parts: 255 00:12:15,480 --> 00:12:19,240 the average, three quarters, plus how much better or worse 256 00:12:19,240 --> 00:12:22,083 that one did: a quarter, a quarter, 257 00:12:22,083 --> 00:12:24,710 a quarter, and minus three quarters. 258 00:12:25,050 --> 00:12:27,918 The average part is the same for every token, 259 00:12:27,918 --> 00:12:30,002 just like our constant reward. 260 00:12:30,002 --> 00:12:33,445 So it produces exactly the pull we just found: 261 00:12:33,445 --> 00:12:37,841 the arrow from p to q, scaled by the average reward. 262 00:12:37,841 --> 00:12:40,810 That's the drift. The other part is different. 263 00:12:40,810 --> 00:12:43,770 Tokens that did better than average get a step toward 264 00:12:43,770 --> 00:12:47,535 their corner. Tokens that did worse get a step away. 265 00:12:47,535 --> 00:12:50,890 This is the only part that knows which token led to which 266 00:12:50,890 --> 00:12:53,607 reward. It's the learning signal: 267 00:12:53,607 --> 00:12:55,400 the part you actually want. 268 00:12:55,740 --> 00:12:58,620 On screen is the same split, written the way the paper 269 00:12:58,620 --> 00:13:02,300 writes it: the average update equals the average reward 270 00:13:02,300 --> 00:13:05,900 times the average arrow, plus a term statisticians call a 271 00:13:05,900 --> 00:13:09,193 covariance. Drift, plus signal. 272 00:13:09,193 --> 00:13:12,060 On-policy, the average arrow is zero, 273 00:13:12,060 --> 00:13:15,262 so drift disappears and only signal is left. 274 00:13:15,262 --> 00:13:18,539 Off-policy, drift is there at every point: 275 00:13:18,539 --> 00:13:22,060 a pull toward the sampler, or a push away when the average 276 00:13:22,060 --> 00:13:23,740 reward is negative. 277 00:13:23,740 --> 00:13:26,288 Its size depends only on that average, 278 00:13:26,288 --> 00:13:28,330 and not at all on what worked. 279 00:13:28,990 --> 00:13:33,309 So why would a gentle pull toward the sampler be fatal? 280 00:13:33,309 --> 00:13:36,498 Picture someone trying to look like a portrait on a wall. 281 00:13:36,498 --> 00:13:41,070 They look, adjust, get closer, and eventually they match, 282 00:13:41,070 --> 00:13:44,658 and stop. That's distillation from a fixed teacher: 283 00:13:44,658 --> 00:13:47,900 it settles, even if the portrait is of someone else. 284 00:13:48,283 --> 00:13:51,360 Now replace the portrait with a funhouse mirror that makes 285 00:13:51,360 --> 00:13:53,553 you look one percent taller. 286 00:13:53,553 --> 00:13:56,159 They look, and stretch to match. 287 00:13:56,159 --> 00:13:59,120 But the mirror is showing them, so now the reflection is 288 00:13:59,120 --> 00:14:01,242 one percent taller than that. 289 00:14:01,242 --> 00:14:02,682 They stretch again. 290 00:14:02,682 --> 00:14:05,742 There's nothing to settle on, because the target moves 291 00:14:05,742 --> 00:14:07,390 every time they do. 292 00:14:07,729 --> 00:14:11,170 That's the loop. The sampler is a slightly distorted copy 293 00:14:11,170 --> 00:14:13,891 of the trainer. Drift pulls the trainer toward it, 294 00:14:13,891 --> 00:14:16,610 the new weights are copied back into the sampler, 295 00:14:16,610 --> 00:14:18,403 and the distortion is applied again, 296 00:14:18,403 --> 00:14:22,815 on top. And rounding isn't random noise that averages out. 297 00:14:22,815 --> 00:14:25,272 Round the same weights the same way, 298 00:14:25,272 --> 00:14:27,846 and you get the same error, every time. 299 00:14:27,846 --> 00:14:30,291 So the distortion points the same way, 300 00:14:30,291 --> 00:14:33,200 step after step, just like the mirror. 301 00:14:33,594 --> 00:14:36,165 Here it is, computed on our triangle. 302 00:14:36,165 --> 00:14:39,776 With a fixed teacher, p slides to q and stops. 303 00:14:39,776 --> 00:14:42,740 With a teacher that's always the same fixed distortion of 304 00:14:42,740 --> 00:14:46,420 p, p chases it, and the two run off together into a 305 00:14:46,420 --> 00:14:50,337 corner: a model that only ever says one thing. 306 00:14:50,337 --> 00:14:54,120 With a fixed teacher, the error is a distance you close. 307 00:14:54,120 --> 00:14:57,650 With a moving one, it's a speed that never stops. 308 00:14:58,310 --> 00:15:01,590 The paper describes that constant-reward setup as a 309 00:15:01,590 --> 00:15:02,776 thought experiment. 310 00:15:02,776 --> 00:15:05,670 It's cheap to try, so we ran it ourselves, 311 00:15:05,670 --> 00:15:09,510 on a tiny model: a network of seventy thousand parameters 312 00:15:09,510 --> 00:15:11,590 that adds two-digit numbers. 313 00:15:11,590 --> 00:15:14,390 It starts out getting about half its sums right. 314 00:15:14,390 --> 00:15:17,270 Then every attempt gets a reward of plus one, 315 00:15:17,270 --> 00:15:19,960 no matter what, so with a perfect method, 316 00:15:19,960 --> 00:15:21,429 nothing should happen. 317 00:15:21,429 --> 00:15:24,660 We trained three copies, side by side. 318 00:15:25,000 --> 00:15:27,160 First, no mismatch at all. 319 00:15:27,160 --> 00:15:30,760 Its weights barely move, and its accuracy only wanders a 320 00:15:30,760 --> 00:15:33,477 little, from the randomness of sampling. 321 00:15:33,477 --> 00:15:36,038 Second, a tiny mismatch: 322 00:15:36,038 --> 00:15:39,240 the teacher is a barely scrambled copy that follows the 323 00:15:39,240 --> 00:15:43,076 student. Within fifty steps, it's below ten percent. 324 00:15:43,076 --> 00:15:44,470 Here's what it writes. 325 00:15:48,010 --> 00:15:50,143 Third, a fixed teacher: 326 00:15:50,143 --> 00:15:53,372 a frozen copy, scrambled five times harder, 327 00:15:53,372 --> 00:15:56,010 for about twenty times the mismatch. 328 00:15:56,010 --> 00:15:59,121 That teacher only gets about twenty-eight percent right. 329 00:15:59,121 --> 00:16:01,203 The student slides down to its level, 330 00:16:01,203 --> 00:16:04,652 and stops there: exactly as bad as its teacher, 331 00:16:04,652 --> 00:16:05,960 and no worse. 332 00:16:06,300 --> 00:16:09,083 Now measure how far the weights travel. 333 00:16:09,083 --> 00:16:11,435 No mismatch: almost nowhere. 334 00:16:11,435 --> 00:16:15,405 The fixed teacher: a short way, and slowing down. 335 00:16:15,405 --> 00:16:16,535 The moving teacher: 336 00:16:16,535 --> 00:16:19,820 eight times further, and still moving about six times 337 00:16:19,820 --> 00:16:21,500 faster when we stopped. 338 00:16:21,500 --> 00:16:24,097 The size of the mismatch isn't what matters. 339 00:16:24,097 --> 00:16:26,970 Whether the teacher moves, is. 340 00:16:27,617 --> 00:16:29,308 Wait, you might say. 341 00:16:29,308 --> 00:16:32,848 Real systems don't hand out plus one for everything. 342 00:16:32,848 --> 00:16:34,986 They use group-centered rewards: 343 00:16:34,986 --> 00:16:37,859 each attempt is scored against its group's average, 344 00:16:37,859 --> 00:16:40,110 so the rewards add up to zero. 345 00:16:40,110 --> 00:16:43,027 An average reward of zero means no drift, 346 00:16:43,027 --> 00:16:43,660 right? 347 00:16:44,012 --> 00:16:48,160 Not quite. Remember, drift is scaled by the average reward 348 00:16:48,160 --> 00:16:50,082 from each point onward. 349 00:16:50,082 --> 00:16:52,397 Take a point where the attempt is on track: 350 00:16:52,397 --> 00:16:56,000 say three of every four ways it can go on from here end up 351 00:16:56,000 --> 00:16:59,514 winning. From here, the average is above zero, 352 00:16:59,514 --> 00:17:02,000 so at this point, the trainer is pulled toward the 353 00:17:02,000 --> 00:17:05,658 sampler. Now a point just after a mistake. 354 00:17:05,658 --> 00:17:10,000 Most ways on from here lose, so the average is below zero, 355 00:17:10,000 --> 00:17:12,559 and the trainer is pushed away from the sampler. 356 00:17:12,559 --> 00:17:15,140 And that happens to every token after the mistake, 357 00:17:15,140 --> 00:17:17,920 whether or not it had anything to do with the answer. 358 00:17:17,920 --> 00:17:20,960 These pulls and pushes happen at different points, 359 00:17:20,960 --> 00:17:24,080 with different p's and q's, so they don't cancel each 360 00:17:24,080 --> 00:17:27,041 other out. Group centering does shrink drift, 361 00:17:27,041 --> 00:17:29,433 which fits with it winning that earlier test. 362 00:17:29,433 --> 00:17:30,830 It doesn't remove it. 363 00:17:34,489 --> 00:17:36,374 So now we know what's wrong. 364 00:17:36,374 --> 00:17:37,667 How do you cancel it? 365 00:17:37,667 --> 00:17:40,840 There's a classic answer, and a new one. 366 00:17:41,218 --> 00:17:43,739 The classic answer comes from statistics. 367 00:17:43,739 --> 00:17:45,980 It's called importance sampling. 368 00:17:45,980 --> 00:17:49,660 Your tokens came from q, but you wanted them from p. 369 00:17:49,660 --> 00:17:52,380 So you weight each one by how much likelier the trainer 370 00:17:52,380 --> 00:17:54,062 finds it than the sampler: 371 00:17:54,062 --> 00:17:58,220 p over q. On the triangle, tokens the sampler draws too 372 00:17:58,220 --> 00:18:00,458 often get their arrows shrunk. 373 00:18:00,458 --> 00:18:03,740 Tokens it draws too rarely get their arrows stretched. 374 00:18:03,740 --> 00:18:07,071 On average, that exactly undoes the mismatch. 375 00:18:07,071 --> 00:18:10,810 The average lands back on p, and drift is gone. 376 00:18:11,213 --> 00:18:12,473 Here's the trouble. 377 00:18:12,473 --> 00:18:15,390 Take a token the sampler picks only four times in a 378 00:18:15,390 --> 00:18:18,270 hundred, but the trainer would pick twenty times in a 379 00:18:18,270 --> 00:18:21,390 hundred. When it finally shows up, its arrow gets 380 00:18:21,390 --> 00:18:23,384 multiplied by five. 381 00:18:23,384 --> 00:18:26,830 Most updates count about once, and then a rare one hits 382 00:18:26,830 --> 00:18:28,670 five times as hard. 383 00:18:28,670 --> 00:18:32,539 The average is right, but each update gets much noisier, 384 00:18:32,539 --> 00:18:35,022 and the more the sampler under-picks a token, 385 00:18:35,022 --> 00:18:39,070 the worse it gets. So in practice, almost everybody caps 386 00:18:39,070 --> 00:18:39,580 it. 387 00:18:40,269 --> 00:18:43,890 Here's every possible multiplier, on one line. 388 00:18:43,890 --> 00:18:47,404 One popular version, truncated importance sampling, 389 00:18:47,404 --> 00:18:51,450 caps it at two. Another, masked importance sampling, 390 00:18:51,450 --> 00:18:54,961 throws out any token whose multiplier falls below a half 391 00:18:54,961 --> 00:18:58,328 or above five. And most of the well-known methods, 392 00:18:58,328 --> 00:19:01,680 PPO among them, turn out to be variations on the same 393 00:19:01,680 --> 00:19:04,959 idea. They differ in where they draw the line, 394 00:19:04,959 --> 00:19:06,590 and what they do past it. 395 00:19:06,941 --> 00:19:10,290 But every cap brings back a little of what it was meant to 396 00:19:10,290 --> 00:19:14,209 remove. When a multiplier can't stretch all the way, 397 00:19:14,209 --> 00:19:17,330 the average doesn't land all the way back on p, 398 00:19:17,330 --> 00:19:19,920 and some drift leaks back in. 399 00:19:20,500 --> 00:19:23,819 Now the paper's answer, and it starts with a detour. 400 00:19:23,819 --> 00:19:27,940 Take the arrow for the plus token, from p to its corner. 401 00:19:27,940 --> 00:19:31,854 Instead of going straight there, stop off at q first: 402 00:19:31,854 --> 00:19:34,660 p to q, then q on to the corner. 403 00:19:34,660 --> 00:19:36,850 Same start, same finish. 404 00:19:37,190 --> 00:19:41,110 Do that for all three tokens, and look at the first legs. 405 00:19:41,110 --> 00:19:43,588 They're all the same arrow, p to q, 406 00:19:43,588 --> 00:19:45,508 whichever token was drawn. 407 00:19:45,508 --> 00:19:49,030 The second legs all start at q, and the tokens were drawn 408 00:19:49,030 --> 00:19:53,110 from q. So by our rule, if every token got the same 409 00:19:53,110 --> 00:19:55,590 reward, they'd land right where they start, 410 00:19:55,590 --> 00:19:58,297 and cancel out. So all of the drift, 411 00:19:58,297 --> 00:20:02,212 that pull from p to q, was hiding in the first leg: 412 00:20:02,212 --> 00:20:04,900 the one that's the same for every token. 413 00:20:05,267 --> 00:20:08,440 And you can know that leg exactly, because it's just the 414 00:20:08,440 --> 00:20:10,445 difference between two lists. 415 00:20:10,445 --> 00:20:13,640 Both copies already work out a full list of probabilities 416 00:20:13,640 --> 00:20:15,000 for the next token; 417 00:20:15,000 --> 00:20:17,400 the sampler just sends its list along. 418 00:20:17,400 --> 00:20:20,840 The trainer says fifty, thirty, twenty. 419 00:20:20,840 --> 00:20:23,883 The sampler says thirty-one, twenty-five, 420 00:20:23,883 --> 00:20:26,840 forty-four. Going from one to the other: 421 00:20:26,840 --> 00:20:30,919 down nineteen, down five, up twenty-four. 422 00:20:30,919 --> 00:20:34,678 That's the leg. No sampling, no estimating: 423 00:20:34,678 --> 00:20:37,590 it's known before the next token is even drawn. 424 00:20:37,929 --> 00:20:42,167 So subtract that one known leg from every token's arrow, 425 00:20:42,167 --> 00:20:44,392 before the reward multiplies it. 426 00:20:44,392 --> 00:20:47,131 What's left of each arrow is its second leg, 427 00:20:47,131 --> 00:20:50,088 starting at q. That's score centering. 428 00:20:50,088 --> 00:20:54,138 Drift is zero, at every point, mismatch or not. 429 00:20:54,138 --> 00:20:56,216 And nothing useful is lost. 430 00:20:56,216 --> 00:20:59,020 The leg we dropped was the same for every token, 431 00:20:59,020 --> 00:21:01,738 so it never said which token did better. 432 00:21:01,738 --> 00:21:03,340 That's in the second legs: 433 00:21:03,340 --> 00:21:06,700 they point to different corners, so when better tokens get 434 00:21:06,700 --> 00:21:09,179 bigger rewards, they don't cancel. 435 00:21:09,179 --> 00:21:12,460 That's the signal. And if p and q already match, 436 00:21:12,460 --> 00:21:16,060 the first leg has zero length, so score centering does 437 00:21:16,060 --> 00:21:19,819 nothing at all. No multipliers, no caps, 438 00:21:19,819 --> 00:21:21,130 nothing to tune. 439 00:21:21,790 --> 00:21:24,196 So there are two ways to cancel drift, 440 00:21:24,196 --> 00:21:27,128 and here's why one is so much steadier. 441 00:21:27,128 --> 00:21:31,150 Importance sampling changes how much each token counts. 442 00:21:31,150 --> 00:21:34,270 Take the rare token from before, the one the sampler picks 443 00:21:34,270 --> 00:21:36,442 only four times in a hundred: 444 00:21:36,442 --> 00:21:39,701 it counts five times as much whenever it shows up. 445 00:21:39,701 --> 00:21:42,910 So each update swings, depending on whether that rare 446 00:21:42,910 --> 00:21:45,228 token happened to be drawn this time. 447 00:21:45,228 --> 00:21:47,593 That swinging is the noise. 448 00:21:47,593 --> 00:21:50,030 It's why people cap the multipliers, 449 00:21:50,030 --> 00:21:52,540 and the caps let some drift back in. 450 00:21:52,865 --> 00:21:56,318 Score centering never changes how much a token counts. 451 00:21:56,318 --> 00:21:58,963 It drops the same leg from every arrow, 452 00:21:58,963 --> 00:22:00,881 whichever token was drawn. 453 00:22:00,881 --> 00:22:04,240 No token gets blown up, so there's nothing to cap, 454 00:22:04,240 --> 00:22:06,590 and drift is cancelled completely. 455 00:22:06,984 --> 00:22:10,782 It's like correcting a survey that polled the wrong crowd. 456 00:22:10,782 --> 00:22:14,130 You can reweight each person by how under-represented they 457 00:22:14,130 --> 00:22:17,490 are, but then one rare respondent can swing the whole 458 00:22:17,490 --> 00:22:21,810 result. Or, if you know exactly how that crowd leans, 459 00:22:21,810 --> 00:22:24,610 you can subtract the lean from everyone's answer. 460 00:22:24,610 --> 00:22:25,970 And you can even do both: 461 00:22:25,970 --> 00:22:29,570 reweight with capped multipliers, then subtract whatever 462 00:22:29,570 --> 00:22:30,720 lean is left. 463 00:22:31,442 --> 00:22:35,140 There's a catch. Computing it exactly means the sampler 464 00:22:35,140 --> 00:22:38,980 hands over its probability for every possible next token, 465 00:22:38,980 --> 00:22:41,700 at every token of every attempt. 466 00:22:41,700 --> 00:22:45,699 These models know a hundred and fifty-two thousand tokens. 467 00:22:45,699 --> 00:22:49,329 Store all of that, for a big batch of long attempts, 468 00:22:49,329 --> 00:22:51,090 and it's twenty terabytes. 469 00:22:51,985 --> 00:22:56,570 So the paper keeps only the sampler's top 128 guesses at 470 00:22:56,570 --> 00:23:00,249 each token. Those hold almost all of its probability: 471 00:23:00,249 --> 00:23:03,210 over ninety-nine point nine percent on average, 472 00:23:03,210 --> 00:23:05,768 in all but one of the paper's settings. 473 00:23:05,768 --> 00:23:08,970 For the rest, it borrows the trainer's own probabilities, 474 00:23:08,970 --> 00:23:12,650 rescaled to fit. The whole thing is a few lines of code, 475 00:23:12,650 --> 00:23:15,850 and in the paper's runs it added at most about one percent 476 00:23:15,850 --> 00:23:17,080 to training time. 477 00:23:20,740 --> 00:23:23,459 Now look again at what centering did. 478 00:23:23,459 --> 00:23:27,300 Before, every arrow started at p, the trainer's own 479 00:23:27,300 --> 00:23:31,941 probabilities. After centering, every arrow starts at q. 480 00:23:31,941 --> 00:23:35,700 So which way each token gets pushed no longer depends on 481 00:23:35,700 --> 00:23:37,890 the trainer's probabilities at all. 482 00:23:38,230 --> 00:23:42,070 In words: a normal update says, make this token more 483 00:23:42,070 --> 00:23:45,350 likely than I, the trainer, expected. 484 00:23:45,350 --> 00:23:48,230 A centered update says, make it more likely than the 485 00:23:48,230 --> 00:23:49,860 sampler expected. 486 00:23:50,279 --> 00:23:53,386 It's as if the sampler graded its own attempts, 487 00:23:53,386 --> 00:23:56,440 by perfectly ordinary on-policy learning, 488 00:23:56,440 --> 00:23:58,689 and handed the result to the trainer. 489 00:23:58,689 --> 00:24:02,039 In the simplest models, that's exactly what happens. 490 00:24:02,039 --> 00:24:05,640 In deep networks it's close, as long as the two copies are 491 00:24:05,640 --> 00:24:06,550 close. 492 00:24:07,238 --> 00:24:10,084 So there are two ways to read this fix. 493 00:24:10,084 --> 00:24:12,230 One: it removes drift. 494 00:24:12,230 --> 00:24:16,364 True. Two: it doesn't try to undo the mismatch. 495 00:24:16,364 --> 00:24:19,450 It learns from the sampler's side, and simply stops the 496 00:24:19,450 --> 00:24:21,850 trainer from being dragged toward it. 497 00:24:21,850 --> 00:24:24,813 In survey terms, it subtracts the lean, 498 00:24:24,813 --> 00:24:27,315 but it's still listening to the sampler's crowd. 499 00:24:27,315 --> 00:24:30,250 Importance sampling aims for the trainer's own update, 500 00:24:30,250 --> 00:24:32,210 and pays for it in noise. 501 00:24:32,210 --> 00:24:35,495 Centering is steady, but it's the sampler's update, 502 00:24:35,495 --> 00:24:37,160 not quite the trainer's. 503 00:24:37,570 --> 00:24:41,144 While the sampler is a near copy, that's a good trade: 504 00:24:41,144 --> 00:24:45,420 arrows from q point almost the same way as arrows from p. 505 00:24:45,420 --> 00:24:47,816 But when the sampler falls far behind, 506 00:24:47,816 --> 00:24:50,700 the two sets of arrows point quite differently. 507 00:24:50,700 --> 00:24:53,735 Then you're learning from an old model's point of view, 508 00:24:53,735 --> 00:24:56,300 and centering alone shouldn't be enough. 509 00:24:56,300 --> 00:24:58,831 The paper says the same in its own terms: 510 00:24:58,831 --> 00:25:01,980 after centering, the remaining signal is measured from the 511 00:25:01,980 --> 00:25:04,570 sampler's side, not the trainer's. 512 00:25:04,958 --> 00:25:08,179 So here's a prediction to check against the tests. 513 00:25:08,179 --> 00:25:10,589 Rounding keeps the two copies close, 514 00:25:10,589 --> 00:25:12,989 so centering alone should hold. 515 00:25:12,989 --> 00:25:15,150 A stale sampler lets them move apart, 516 00:25:15,150 --> 00:25:17,580 so centering alone should struggle. 517 00:25:21,256 --> 00:25:25,000 To compare fairly, the authors strip every method down to 518 00:25:25,000 --> 00:25:28,280 the same basic recipe, so that only the correction 519 00:25:28,280 --> 00:25:29,350 changes. 520 00:25:29,690 --> 00:25:33,610 Then there's time. Drift builds up step by step, 521 00:25:33,610 --> 00:25:36,490 and under the mild mismatch of a normal setup, 522 00:25:36,490 --> 00:25:39,370 it can take a very long time to cause a collapse, 523 00:25:39,370 --> 00:25:42,406 longer than they could afford to run over and over. 524 00:25:42,406 --> 00:25:44,735 So they turn the mismatch up on purpose: 525 00:25:44,735 --> 00:25:48,528 severe-and-short, as a stand-in for mild-and-long, 526 00:25:48,528 --> 00:25:50,890 like testing a part by overloading it. 527 00:25:50,890 --> 00:25:54,010 They think that's also why earlier reports disagree about 528 00:25:54,010 --> 00:25:56,723 how dangerous a small mismatch is: 529 00:25:56,723 --> 00:26:00,360 it depends on how severe it is, and how long you watch. 530 00:26:01,056 --> 00:26:03,822 Every chart from here on works the same way. 531 00:26:03,822 --> 00:26:06,010 Left to right, training steps. 532 00:26:06,010 --> 00:26:09,020 Up, how often the model gets the answer right. 533 00:26:09,020 --> 00:26:11,500 A healthy run climbs and stays up. 534 00:26:11,500 --> 00:26:13,179 A collapse looks like this: 535 00:26:13,179 --> 00:26:15,290 a cliff, down to the floor. 536 00:26:15,630 --> 00:26:18,882 To keep them readable, four methods get colors. 537 00:26:18,882 --> 00:26:22,510 Plain policy gradient, with no correction at all. 538 00:26:22,510 --> 00:26:25,782 Truncated importance sampling, the capped classic. 539 00:26:25,782 --> 00:26:29,390 Score centering. And score centering stacked on top of 540 00:26:29,390 --> 00:26:30,827 importance sampling. 541 00:26:30,827 --> 00:26:33,580 Every other published method stays in grey. 542 00:26:34,240 --> 00:26:37,736 First test: the bluntest kind of mismatch. 543 00:26:37,736 --> 00:26:40,160 Take a small model doing the numbers puzzle, 544 00:26:40,160 --> 00:26:43,037 and add random noise to the sampler's weights, 545 00:26:43,037 --> 00:26:45,203 the same noise every time. 546 00:26:45,203 --> 00:26:48,000 Since the sampler is refreshed from the trainer at every 547 00:26:48,000 --> 00:26:51,520 step, it's always the trainer plus the same distortion: 548 00:26:51,520 --> 00:26:53,541 a funhouse mirror with a dial. 549 00:26:53,541 --> 00:26:55,589 Small noise. Double it. 550 00:26:55,589 --> 00:26:56,910 Five times. 551 00:26:57,250 --> 00:27:01,010 Plain policy gradient, in cream, struggles even at the 552 00:27:01,010 --> 00:27:04,681 smallest setting. Now watch truncated importance sampling, 553 00:27:04,681 --> 00:27:08,214 in pink. At the smallest noise, it's fine. 554 00:27:08,214 --> 00:27:11,650 Double the noise, and it collapses around step one-eighty. 555 00:27:11,650 --> 00:27:14,282 Five times, around step forty. 556 00:27:14,282 --> 00:27:16,939 More mismatch, earlier collapse. 557 00:27:16,939 --> 00:27:18,496 That's drift building up. 558 00:27:18,496 --> 00:27:21,414 And at the largest noise, only score centering, 559 00:27:21,414 --> 00:27:24,130 alone or stacked with importance sampling, 560 00:27:24,130 --> 00:27:25,360 keeps training. 561 00:27:26,020 --> 00:27:29,057 Next, the two realistic kinds of damage, 562 00:27:29,057 --> 00:27:31,038 both turned up on purpose. 563 00:27:31,038 --> 00:27:35,380 Rounding: the sampler runs entirely in eight-bit integers. 564 00:27:35,380 --> 00:27:38,538 The copies stay close, just slightly distorted, 565 00:27:38,538 --> 00:27:40,819 so centering alone should hold. 566 00:27:40,819 --> 00:27:44,175 And it does: score centering, alone or stacked, 567 00:27:44,175 --> 00:27:45,588 is on top. 568 00:27:45,588 --> 00:27:48,790 Staleness: the sampler gets fresh weights only every 569 00:27:48,790 --> 00:27:52,550 sixty-four steps. In between, it stands still while the 570 00:27:52,550 --> 00:27:53,751 trainer moves away; 571 00:27:53,751 --> 00:27:55,429 then it jumps to catch up. 572 00:27:55,429 --> 00:27:57,873 So it's still a copy that follows the student, 573 00:27:57,873 --> 00:28:01,589 just in jumps. And between jumps the two move apart, 574 00:28:01,589 --> 00:28:04,939 exactly the case where centering alone should struggle. 575 00:28:04,939 --> 00:28:09,333 It does: on its own, it climbs, then slides back down. 576 00:28:09,333 --> 00:28:12,146 Importance sampling alone does even worse. 577 00:28:12,146 --> 00:28:15,031 But stack the two, and it's the best on the chart: 578 00:28:15,031 --> 00:28:17,910 the multipliers pull the aim back toward the trainer's own 579 00:28:17,910 --> 00:28:21,190 side, and centering removes the drift their caps let 580 00:28:21,190 --> 00:28:24,469 through. So: for rounding, centering. 581 00:28:24,469 --> 00:28:26,617 For a sampler that falls far behind, 582 00:28:26,617 --> 00:28:27,540 both. 583 00:28:28,200 --> 00:28:32,360 Finally, scale: back to the 30 billion parameter model 584 00:28:32,360 --> 00:28:35,160 from the opening, learning math, with the sampler's 585 00:28:35,160 --> 00:28:37,319 numbers rounded three ways. 586 00:28:37,319 --> 00:28:40,996 Mild rounding: even plain policy gradient does fine, 587 00:28:40,996 --> 00:28:43,480 at about fifty-eight percent, level with the best 588 00:28:43,480 --> 00:28:47,239 corrections. Round the sampler's memory to four bits, 589 00:28:47,239 --> 00:28:49,471 and there's our collapse from the opening: 590 00:28:49,471 --> 00:28:53,275 plain policy gradient is gone within two hundred steps. 591 00:28:53,275 --> 00:28:56,040 Score centering holds at fifty-two percent, 592 00:28:56,040 --> 00:28:59,160 and truncated importance sampling at fifty-one. 593 00:28:59,160 --> 00:29:00,919 The harshest rounding of all: 594 00:29:00,919 --> 00:29:03,559 score centering climbs to thirty percent. 595 00:29:03,559 --> 00:29:06,498 Truncated importance sampling reaches twelve. 596 00:29:06,498 --> 00:29:09,000 Every other method ends around five, 597 00:29:09,000 --> 00:29:11,560 or below. The harsher the mismatch, 598 00:29:11,560 --> 00:29:12,870 the wider the gap. 599 00:29:13,594 --> 00:29:17,930 And that shortcut, keeping only the top 128 guesses? 600 00:29:17,930 --> 00:29:20,748 Against the full list, in every setting, 601 00:29:20,748 --> 00:29:23,720 it's on par. Even with only thirty-two. 602 00:29:27,389 --> 00:29:31,059 Now the fine print, and the authors state it themselves. 603 00:29:31,059 --> 00:29:33,860 The headline results come from deliberately severe 604 00:29:33,860 --> 00:29:37,462 mismatch, on short runs with short attempts. 605 00:29:37,462 --> 00:29:40,903 That's a stand-in for long runs with milder mismatch, 606 00:29:40,903 --> 00:29:42,443 not a measurement of them. 607 00:29:42,443 --> 00:29:45,940 And at thirty billion parameters, many methods were only 608 00:29:45,940 --> 00:29:47,270 run once. 609 00:29:47,270 --> 00:29:50,476 Their advice: if the mismatch comes from rounding, 610 00:29:50,476 --> 00:29:52,225 use score centering. 611 00:29:52,225 --> 00:29:54,950 If the sampler can fall far behind the trainer, 612 00:29:54,950 --> 00:29:57,190 stack it with importance sampling. 613 00:29:57,190 --> 00:30:00,711 And still, refresh the sampler as often as you can, 614 00:30:00,711 --> 00:30:03,235 and round both copies the same way. 615 00:30:03,235 --> 00:30:05,589 Centering doesn't make the mismatch good. 616 00:30:05,589 --> 00:30:07,460 It makes it survivable. 617 00:30:08,196 --> 00:30:09,794 It has traveled fast. 618 00:30:09,794 --> 00:30:13,240 According to its code repository, score centering is now 619 00:30:13,240 --> 00:30:16,280 implemented in most of the major frameworks for this kind 620 00:30:16,280 --> 00:30:19,952 of training. In one trainer, it's already the default, 621 00:30:19,952 --> 00:30:22,150 and another says it soon will be. 622 00:30:22,855 --> 00:30:25,125 So, back to where we started. 623 00:30:25,125 --> 00:30:27,514 A completely different model as a teacher: 624 00:30:27,514 --> 00:30:31,111 stable. A slightly blurry copy of yourself: 625 00:30:31,111 --> 00:30:33,929 fatal. Now you can say why. 626 00:30:33,929 --> 00:30:37,290 Every reinforcement learning step carries a hidden pull 627 00:30:37,290 --> 00:30:39,439 toward whoever wrote the attempts. 628 00:30:39,439 --> 00:30:43,153 With a fixed teacher, that pull closes a distance, 629 00:30:43,153 --> 00:30:45,906 and stops. With a copy of yourself, 630 00:30:45,906 --> 00:30:48,419 the teacher moves every time you do, 631 00:30:48,419 --> 00:30:50,317 and the pull never stops. 632 00:30:50,317 --> 00:30:52,567 Or, as the paper puts it: 633 00:30:52,567 --> 00:30:55,690 what matters is not the size of the mismatch, 634 00:30:55,690 --> 00:30:59,480 but whether the teacher is fixed, or tracks the student. 635 00:31:01,205 --> 00:31:05,620 One hidden term, and one subtraction to take it away. 636 00:31:05,620 --> 00:31:07,700 It doesn't remove the mismatch. 637 00:31:07,700 --> 00:31:11,010 It stops the mismatch from feeding on itself.