codec bottleneck for multimodal ML
- claim: we lack good compression/tokenization schemes (codecs) for non-text modalities like audio, image/video, vibrations/haptics and this is a bottleneck holding back models from doing more useful things
- text is incredibly information dense per token. you can pack a lot of meaning into ~750 tokens but you. may only be able to get ~6 seconds of audio
- english is still lossy, but compared to other modalities is an extremely efficient codec for meaning
- we care about this because transformers operate over token sequences with a bounded context window