KV cache

Memory a model keeps of calculations it already did on earlier text, so it doesn't redo them for each new token. Also behind prompt caching, which makes repeated long prompts cheaper. TechCrunch's glossary calls this memory cache.

“With caching on, re-sending the same 50-page contract costs a fraction.”