large models are too good at memorization which can limit true understanding
most text on the internet is not nicely written wall street journal articles, but instead mostly junk/noise. to absorb the signal, you need very large models, say hundreds of billions to ~trillion parameters
one liability that occurs once models get very large is that they can become rote memorizers, which is in some ways is at odds with true understanding.
this makes intuitive sense to me; thinking about how you can learn a math formula & plug values in to calculate some output, but still not really understand the why behind the formula or be able to derive it yourself. or how the answer to most basic multiplication questions are stored as lookup tables that you’ve memorized and therefore do not have to perform computation to recite 9x9, for example
if you train on some nonsensical/pattern-less text, say the SHA-256 hash of your name, for even a single iteration or two, the model can regurgitate/memorize it. humans are, in comparison, much worse at memorization, and this forces us to compress information into generalizable patterns
so, making memory more scarce within a model will force it to understand instead of memorize, but how does one actually handicap a model’s ability to memorize?
(1) increase the signal on the data you are training on
by (a) doing something along the lines of really scrubbing and cleaning up and dedup-ing the pre-training dataset & training on that
OR (b) distilling from another model that already has done the hard work of compressing + filtering the raw internet down to more genrealizable structures
andrej has a vision for a personalized small langauge models (couple billion params) that have been deliberately stripped of encyclopedic knowledge and replaces that with lots and lots of tool calls.
akin to us humans looking something up when it doesn’t know something
in-context learning vs. pre-training
weights are a “hazy recollection”, while context window/ KV cache is like working memory that is directly accessible to the model
dwarkesh also presents this information density argument saying that there is roughly 0.07 bits stored per token in pretraining versus ~320kB stored per token in the KV cache, which is a 36 million fold increase