Qwen-4 Architecture: New Achievements in AI
The Qwen-4 model promises significant improvements over previous versions due to new technologies. Research shows that the new architecture delivers high performance and resource savings.
Qwen-4 Architecture: New Achievements in AI
Model Scope has launched the countdown to the release of Qwen3.8-Flash-Next. Its release will take place in a day, and this will be the first model built on the next-generation architecture, which will also be the basis for Qwen-4.
Key Features of Qwen-3.8 Flash-Next
This model features MoE 125B + 51B N-gram table (with only 6B active), outperforming Opus 4.6 Max on 8 out of 9 benchmarks! It also surpasses Qwen3.8-27B and DeepSeek-V4-Flash, while training costs were 9 times cheaper than Qwen3.7-Plus.
Key Innovations in Architecture
N-gram embeddings: This is a table indexed not by a single token but by short sequences. The model remembers typical local combinations of tokens directly as vectors, reducing token computations and increasing overall model capacity.
Gated DeltaNet + Qwen Sparse Attention: The model selects relevant context blocks and works only with them. Gated DeltaNet is a linear attention mechanism with gates that control what the model remembers and what it forgets. Result: on 1M context, prefill became 7.6 times faster, and decode became 4.9 times faster.
Gated Residual connections: Gates in residual connections determine how much information to pass on.
Muon optimizer: This new optimizer provides a 9-fold savings in training costs.
Conclusion
As a result, training is cheaper, there are more parameters, and fewer computations per token. This is another achievement of Chinese open-source.
Why it matters
AnalysisThis news is significant as the new Qwen-4 architecture could fundamentally change approaches to AI model development, reducing training costs and enhancing performance.
Discuss in community
Share your questions and insights with developers
Related in Ecosystem
Practical guides, core terminology, and agent skills
GPT-6 Astra reportedly edits another AI model's weights without retraining
GPT-6 Astra is claimed to modify only 0.016% of Qwen3.5-0.8B's weights without gradients or retraining, improving reasoning, math results, and instruction following.
GPT Image Generative Models Prompting Guide
OpenAI's gpt-image generative models are designed to produce production-quality visuals and fully controllable creative workflows. They are suitable for both professional design ta...
Claude Sonnet 4.6 vs GPT-5.5 vs Gemini 3.1 Pro: 2026 model comparison
Every few weeks, a new model comes out, and every time the same questions arise: should I switch? what changed what to pay for?
Binance publishes investigation into $6.3 billion theft
Binance’s investigation links the theft to a modified Homebrew download, a fake interview and abuse of an internal AI tool.