Listen now | Had so much fun chatting with my good friends Trenton Bricken and Sholto Douglas on the podcast. No way to summarize it, except: This is the best context dump out there on how LLMs are trained, what capabilities they're likely to soon have, and what exactly is going on inside them.
Two things, both about superposition: first a note about the brain, and then a note about linguistics.
FWIW, a bit over two decades ago I had extensive correspondence with the late Walter Freeman, who was one of the pioneers in the application of complexity theory to the study of the brain. He pretty much assumed that any given neuron (w/ it's 10K connections to other neurons) would participate in many perceptual or motor schemas. The fact that now and then you'd come up with neurons who had odd-ball receptive properties (e.g. a monkey's paw, or Bill Clinton) was interesting, but hardly evidence for the existence of so-called grandmother neurons (i.e. a neuron for your grandmother and, by extension, individual neurons for individual perceptual objects). As far as I can tell, the idea of neural superposition goes back decades, at least to the late 1960s when Karl Pribram and others started thinking about the brain in holographic terms.
Setting that aside, a somewhat limited form of superposition has been common in linguistics going back to the early 20th century. It's the basic idea underling the concept of distinctive features in phonetics/phonology. Speech sound is continuous, but we hear language in terms of discrete segments, called phonemes. Phonemes are analyzed in terms of distinctive features (https://en.wikipedia.org/wiki/Distinctive_feature). That is, they are analyzed in terms of the sound features that distinguish one speech sound from another in a given language. The number of distinctive features in a given language system is smaller than the number of phonemes. I don't know off hand what the range is, but the number of phonemes in a language is on the order of 10s and the number of distinctive features will be somewhat smaller for a given language. So phonemes can be identified by a superposition of distinctive features.
The numbers involved are obviously way smaller than the features and parameters in an LLM. But the principle seems to be the same.
“There's some work where they'll simulate little basic agents and see if the representations they learn map to the tools they can use and the inputs they should have. ”
Some typos from the first 2 hours: "So there's a direct path and an indirect path. and, and so the model can pick up whatever information it wants and then add that back in."
"But we have a lot more work to do on that. surprise to the Twitter guy,"
"And there's a verifier there too, right? There's the real world. You might generate a theory about the gods causing the storms, And then someone else finds cases where that isn't true."
I enjoyed it, nice looser conversation. I didn't understand everything, but hey, you learn by stretching into new territory, so I appreciate the advanced discussion! 💚 🥃
Two things, both about superposition: first a note about the brain, and then a note about linguistics.
FWIW, a bit over two decades ago I had extensive correspondence with the late Walter Freeman, who was one of the pioneers in the application of complexity theory to the study of the brain. He pretty much assumed that any given neuron (w/ it's 10K connections to other neurons) would participate in many perceptual or motor schemas. The fact that now and then you'd come up with neurons who had odd-ball receptive properties (e.g. a monkey's paw, or Bill Clinton) was interesting, but hardly evidence for the existence of so-called grandmother neurons (i.e. a neuron for your grandmother and, by extension, individual neurons for individual perceptual objects). As far as I can tell, the idea of neural superposition goes back decades, at least to the late 1960s when Karl Pribram and others started thinking about the brain in holographic terms.
Setting that aside, a somewhat limited form of superposition has been common in linguistics going back to the early 20th century. It's the basic idea underling the concept of distinctive features in phonetics/phonology. Speech sound is continuous, but we hear language in terms of discrete segments, called phonemes. Phonemes are analyzed in terms of distinctive features (https://en.wikipedia.org/wiki/Distinctive_feature). That is, they are analyzed in terms of the sound features that distinguish one speech sound from another in a given language. The number of distinctive features in a given language system is smaller than the number of phonemes. I don't know off hand what the range is, but the number of phonemes in a language is on the order of 10s and the number of distinctive features will be somewhat smaller for a given language. So phonemes can be identified by a superposition of distinctive features.
The numbers involved are obviously way smaller than the features and parameters in an LLM. But the principle seems to be the same.
Any idea which paper this is?
“There's some work where they'll simulate little basic agents and see if the representations they learn map to the tools they can use and the inputs they should have. ”
Some typos from the first 2 hours: "So there's a direct path and an indirect path. and, and so the model can pick up whatever information it wants and then add that back in."
"But we have a lot more work to do on that. surprise to the Twitter guy,"
"And there's a verifier there too, right? There's the real world. You might generate a theory about the gods causing the storms, And then someone else finds cases where that isn't true."
There's also "Valley lock-in" -> "value lock-in" (or is that a pun? ;))
I enjoyed it, nice looser conversation. I didn't understand everything, but hey, you learn by stretching into new territory, so I appreciate the advanced discussion! 💚 🥃
Hiring discussion was spot on.
Reminds me that my blog post learning RLHF in one week may still be the top Google result 🤣