Thirteen Million Sessions: The Cache That Remembers

The number is 13.5 million. That is how many GitHub Copilot sessions Microsoft dissected to study how its own infrastructure fails. Not how its models perform — how its machines waste work. The findings read like a post-mortem audit I have produced dozens of times: prompt cache misses burning compute, retry cascades flooding the gateway, GPUs idling through most of every session. The signature is always the same. The weakest component is never the intelligence layer. It is the plumbing underneath. Microsoft calls it research. I call it a vulnerability pre-mortem conducted at industrial scale. The core claim is unglamorous: AI-native workloads are structurally incompatible with the cloud infrastructure they currently run on. Caching, retry behavior, and idle-time handling are the three failure vectors. All three are engineering problems. None of them get solved by a larger model. The paper proposes no new architecture. No breakthrough in reasoning. No novel attention mechanism. This is an engineering-level intervention — and in some respects that makes it more consequential than any single model release could be. Thirteen and a half million real sessions. Not synthetic benchmarks. Not a curated test set harvested in a lab. Production telemetry from developers whose code completions are being served in real time. That scale gives the conclusions an industrial credibility that isolated benchmarks cannot match. I have spent years telling clients that the only trustworthy stress test is one drawn from live traffic. Microsoft just ran one on a planetary scale. The findings align with a shift I have tracked since the DeFi summer of 2020: the realization that system efficiency, not raw model capability, is the binding constraint on AI commercialization. Every serious inference stack — vLLM, SGLang, TensorRT-LLM — is converging on the same three levers this paper quantifies. RadixAttention. Prefix caching. Continuous batching. Microsoft is validating those approaches with production-scale evidence. There is a commercial motive behind the technical facade. GitHub Copilot passed one million users by the end of 2024. At enterprise pricing, that is an approximate annualized run rate of $228 million. Inference cost is the margin tax on that revenue. Every percentage point of cache improvement lands directly on the income statement. But the strategic target is larger than Copilot. This research is the engineering evidence behind Azure's cost narrative. Microsoft is telling the market: we can run AI workloads cheaper because we have measured exactly where the waste lives. That is a pricing weapon aimed at AWS and Google Cloud. It is also quiet leverage in the negotiation with OpenAI — the stronger its own inference stack becomes, the less singular its dependence on any single model vendor. Let me take the three vectors in order, because each one maps to a class of failure I have encountered before. First, cache efficiency. The industry already knew that prompt caching could cut inference costs by up to 70 percent. Anthropic commercialized it as an API feature. What Microsoft measured is the waste embedded in cache misses at production scale. Industry estimates suggest cache inefficiency accounts for 30 to 50 percent of total inference cost. If that holds, optimization here is the highest-ROI lever in the entire AI stack. This is where my 2017 experience intrudes. That year, I flagged an integer overflow in a token distribution contract. The developers knew it was there. They shipped anyway because the token sale deadline was immovable. Two weeks later, forty percent of the treasury drained through the exact line I had marked. The parallel is uncomfortable: I now see teams treating cache misses as a performance nuisance when they are actually a structural cost leak. The blockchain remembers; the architect forgets. In inference, the leak is visible in telemetry. A cache miss forces full recomputation of prefill and KV cache for every request. The GPU repeats identical work for identical prompts across different sessions. Microsoft's 13.5 million sessions quantified what was previously estimated: the repetition rate in code completion workloads is high enough to make caching the primary optimization surface. Code completion is structurally cache-friendly. Framework boilerplate, common library signatures, repeated import patterns — this is not novel computation. It is re-computation presented as intelligence. The second vector is retry cascades. When rate limits or timeouts trigger retries, the system enters a feedback loop. Microsoft observed an average of 1.2 retry requests per session. That sounds benign until you inspect the distribution. At peak, retry cascades can amplify API gateway traffic by 300 to 500 percent. The gateway becomes the bottleneck precisely because it is trying to protect the system from overload. I have seen this failure mode in DeFi. In 2020, I analyzed a leveraged yield protocol holding $50 million in total value locked. My risk models predicted a geometric collapse if oracle price feeds were manipulated during low-liquidity conditions. I published the analysis. The community dismissed it as bearish noise. Three days later, a $10 million flash loan exploited the exact parameter I had flagged. The mechanism was identical: a small disturbance amplified through a feedback loop until the system destabilized. In Copilot's case, the disturbance is a timeout. The amplification is retry. The collapse mode is latency chaos. The proposed mitigation is not clever. Exponential backoff with jitter. Randomized retry timing. This has been standard practice in distributed systems for decades. The noteworthy finding is not the solution. It is the confirmation that LLM serving stacks are still immature enough to require this paper at all. The third vector is idle time. Copilot sessions arrive in bursts. The average interval between requests is roughly 5.8 seconds. That is an eternity in GPU cycles. The result is an idle rate between 40 and 70 percent. Continuous batching and speculative prefill exist to absorb that gap, but Microsoft's data suggests they are not yet deployed effectively at production scale. Industry-wide inference MFU — Model FLOPs Utilization — sits in the 20 to 50 percent band. Optimization here can push MFU toward 60 to 80 percent. That is not incremental. That is nearly doubling the compute you extract from hardware you have already paid for. Here is something the paper does not state explicitly: cache and idle-time optimization are functionally equivalent to adding GPUs. Improve cache hit rate from 50 to 70 percent and effective throughput rises by roughly 66 percent without purchasing a single additional chip. In an environment where supply constraints and power limits dominate procurement timelines, software-level efficiency becomes the first line of capacity. Microsoft is building its own answer to the GPU shortage: making the GPUs it already owns do more work. All three vectors converge on one conclusion. The era of gratuitous compute is over. The infrastructure layer has become a first-class competitive differentiator, and the winner is the player with the largest production dataset to measure against. Microsoft has 13.5 million sessions. Its competitors possess smaller samples. That asymmetry compounds. The bulls are not entirely wrong here. The optimization direction is sound. Caching, retry discipline, and idle-time management are the correct targets. The paper's empirical foundation is genuinely valuable — this is among the first large-scale public datasets on real-world LLM serving behavior. That is a contribution I would not dismiss. But the paper ignores the liability embedded in its own solution. Multi-tenant caching creates a data isolation problem that the research does not address. If a developer's prompt — containing proprietary algorithms, embedded credentials, internal API patterns — is served from a shared cache, what is the exposure surface? Enterprise clients will ask a blunt question: can my code appear in another tenant's response? The GDPR implications alone are material. The paper's silence on compliance is its most revealing feature. The second blind spot is consolidation. Thirteen million sessions is not a resource any mid-size AI company can replicate. This research widens the moat around hyperscalers. The cost advantage generated by cache optimization will concentrate in organizations that already possess massive, repetitive traffic patterns. That is a competitive distortion, not a neutral efficiency gain. The infrastructure remembers. The compliance officer forgets. Microsoft has measured what most AI companies have only guessed. That alone justifies attention. But the deeper signal is strategic: the AI industry is entering an infrastructure cold war. Model capability is commoditizing. The differentiator is now cost per unit of intelligence — per token, per completion, per session. The blockchain remembers; the architect forgets. In this case, the architect is an entire industry that spent two years optimizing models while the systems underneath them bled capital. Microsoft just published the receipts. The question for everyone else is not whether to optimize. It is whether they have the data to know what to optimize first.