Magic AI Coding With Giant Context Windows | NextBigFuture.com

Magic is building frontier code models to automate software engineering and research. This AI coding assistance startup raised $320 million in a funding round led by former Google CEO Eric Schmidt. Magic’s total funding now stands at $515 million.

Magic had about 23 people (+ 8000 H100s) in August, 2024. They are hiring more Engineers and Researchers to accelerate our work and deploy upcoming models. Over time, they will scale up to tens of thousands of GB200s.

Magic raised $320 million in funding led by former Google CEO Eric Schmidt, with additional investments from Elad Gil, Nat Friedman, Daniel Gross, Jane Street, Sequoia, and Atlassian. This followed an earlier Series B funding round of $117 million in February. Magic is important in the AI coding industry because it enhances the efficiency and capabilities of software development.

Magic’s LTM-2-mini model can process information equivalent to 10 million lines of code, enhancing software development efficiency.

Magic trained their first 100M token context model: LTM-2-mini in August, 2024. 100 million tokens equals ~10 million lines of code or ~750 novels.

For each decoded token, LTM-2-mini’s sequence-dimension algorithm is roughly 1000x cheaper than the attention mechanism in Llama 3.1 405B1 for a 100M token context window.

The contrast in memory requirements is even larger – running Llama 3.1 405B with a 100M token context requires 638 H100s per user just to store a single 100M token KV cache.2 In contrast, LTM requires a small fraction of a single H100’s HBM per user for the same context.

There are currently two ways for AI models to learn things:
training, and
in-context during inference.

Training has dominated, because contexts are relatively short. But ultra-long context could change that.

Instead of relying on fuzzy memorization, the LTM (Long-Term Memory) models are trained to reason on up to 100M tokens of context given to them during inference.

While the commercial applications of these ultra-long context models are plenty, at Magic they are focused on the domain of software development.

It’s easy to imagine how much better code synthesis would be if models had all of your code, documentation, and libraries in context, including those not on the public internet.

Trained on hashes with chain of thought, the LTM architecture gets the following results: