Kimi K3 and Inkling Set New Open-Source Benchmark Marks

Moonshot AI and Thinking Machines released major open-weights models this week, each posting benchmark scores that challenge proprietary systems. Moonshot’s Kimi K3, at 2.8 trillion parameters, claims the title of the largest open-source AI model ever built, while Thinking Machines’ Inkling, at 975 billion parameters, scores 77.6% on SWE-bench Verified — outpacing NVIDIA’s Nemotron 3 at 71.9%. Both releases arrive as a separate arXiv study warns that leaderboard scores can shift dramatically based on prompt formatting alone.

Kimi K3: 2.8 Trillion Parameters, Open Weights

Moonshot AI’s Kimi K3 is the largest open-source language model publicly available, with 2.8 trillion total parameters and a 1-million-token context window. VentureBeat reported that internal benchmarks place the model neck-and-neck with top proprietary systems from Anthropic and OpenAI. Full model weights are scheduled for release on July 27, with the model already accessible via kimi.com for registered users.

The Beijing-based startup, backed by Alibaba, timed the release to precede the 2026 World Artificial Intelligence Conference in Shanghai. Kimi K3 is roughly 75% larger than DeepSeek V4 Pro, which Moonshot’s own technical documentation places at approximately 1.6 trillion parameters. The release marks a significant recovery for Moonshot AI, whose market position had eroded over the 18 months following DeepSeek’s rise, according to VentureBeat.

The model features native visual understanding alongside text processing, positioning it as a multimodal competitor. Whether its benchmark parity with closed models holds under independent evaluation remains to be seen — a caveat made more salient by new research on benchmark fragility published the same week.

Inkling: Thinking Machines’ First Open Model

Thinking Machines — the AI startup founded by former OpenAI CTO Mira Murati — released Inkling on an Apache 2.0 open-source license, making it freely usable for commercial enterprise deployments. At 975 billion parameters, the Mixture-of-Experts model processes text, images, and audio natively.

Key benchmark results from Thinking Machines’ own release documentation:

• 77.6% on SWE-bench Verified (software engineering), versus NVIDIA Nemotron 3’s 71.9%
• 91.4% on VoiceBench (voice understanding), compared to Gemini 3.1 Pro’s 94.4% on high reasoning effort

The company describes Inkling as sub-state-of-the-art relative to frontier closed models, but competitive among open-weights alternatives. A notable design choice: Thinking Machines built Inkling to “answer directly on topics that may be subject to censorship,” per the company’s announcement — a feature aimed at enterprises that require factual outputs regardless of topic sensitivity.

Inkling also includes a “controllable thinking effort” mechanism that lets operators trade reasoning depth for inference cost. Weights are available on Hugging Face and through Thinking Machines’ own API, Tinker. A lighter 276-billion-parameter variant, Inkling-Small, is in preview for latency-sensitive workloads.

Benchmark Reliability Under Scrutiny

A paper published this week on arXiv adds an important caveat to any leaderboard reading: prompt formatting alone can flip benchmark conclusions. Researchers introduced two metrics — the Format Sensitivity Index (FSI), measuring accuracy variance from wrapper choice, and the Parseability Sensitivity Index (PSI), measuring answer parseability variance.

Across 140,000 generations on OpenRouter, spanning 7 QA tasks, 5 prompt-wrapper families, and 4 instruct models ranging from 7B to 72B parameters, the study found that mean FSI varies by more than 30x across models. The primary driver is compliance failures — models that fail to parse structured output formats score lower even when their underlying reasoning is correct.

A fixed-effects regression in the paper confirmed that parseability remains a strong predictor of accuracy after controlling for task, model, and wrapper type. The authors argue that “reporting accuracy without wrapper variance and compliance is statistically fragile” — a finding that applies directly to the SWE-bench and VoiceBench scores Thinking Machines and others use for positioning.

The Singularity Gate: A Niche Benchmark for Post-Cutoff Reasoning

A separate community benchmark called the Singularity Gate tests whether frontier models can predict scientific discoveries published after their training cutoff — a measure of generalization rather than memorization. According to a Reddit post in r/singularity, Claude’s Fable 5 currently leads the leaderboard, though the post notes a meaningful degradation between versions: the original Fable 5 responded to 45% of benchmark tasks, while the latest version responded to only 39%, with a slight drop in accuracy on shared tasks.

GPT-5.6 Sol is described in the post as a noticeable improvement over GPT-5.5 on the same benchmark. The Singularity Gate is a community-run project and has not been independently audited, so its results should be treated as directional rather than definitive. Still, the benchmark targets a capability gap — post-cutoff scientific extrapolation — that standard evals like MMLU or SWE-bench do not address.

What This Means

The Kimi K3 and Inkling releases, taken together, mark a meaningful week for open-weights AI — but the arXiv formatting study is a useful corrective against over-reading any single leaderboard number. Moonshot’s 2.8-trillion-parameter model is an engineering achievement at scale; whether its claimed parity with Anthropic and OpenAI’s closed systems survives rigorous third-party evaluation is still an open question, since the benchmarks cited come from Moonshot’s own documentation.

Thinking Machines occupies a different position: Inkling’s scores are explicitly framed as sub-state-of-the-art, and the company’s differentiators — censorship resistance, controllable reasoning effort, Apache 2.0 licensing — are targeted at enterprise buyers rather than benchmark chasers. That positioning is more durable than raw leaderboard rank.

The FSI research also has direct implications for how both companies’ numbers should be interpreted. A 30x variance in model scores based on prompt formatting means that a 5-6 percentage point gap between Inkling and Nemotron 3 on SWE-bench could narrow or reverse under different evaluation conditions. Benchmark consumers — procurement teams, researchers, journalists — should demand wrapper-variance disclosures alongside headline accuracy figures.

FAQ

What is Kimi K3 and how large is it?

Kimi K3 is an open-source large language model released by Beijing-based Moonshot AI, with 2.8 trillion total parameters and a 1-million-token context window. According to VentureBeat, it is the largest open-source AI model ever released, approximately 75% larger than DeepSeek V4 Pro. Full weights are scheduled for public release on July 27.

What benchmark scores did Thinking Machines’ Inkling achieve?

Inkling scored 77.6% on SWE-bench Verified and 91.4% on VoiceBench, according to Thinking Machines’ release announcement. The SWE-bench result beats NVIDIA Nemotron 3’s 71.9%, though Inkling trails Gemini 3.1 Pro’s 94.4% on VoiceBench. The company describes these as competitive for open-weights models but below closed frontier systems.

Why do AI benchmark scores sometimes disagree across sources?

A study published on arXiv found that prompt-wrapper formatting can cause model accuracy to vary by more than 30x across models on the same tasks. Compliance failures — where a model doesn’t produce parseable output — are the dominant cause, meaning the same model can score very differently depending on how the evaluation prompt is structured.