Yao Shunyu appeared on Reddit to answer questions for two hours: The most difficult problem in training after Hunyuan is overthinking
AGI-Signal
2026.11.21 10:53
· From Beijing
Full text 3036 words
The competition for open source large models is shifting from list results to real reputation among the developer community.
In the early morning of November 21st, Beijing time, Yao Shunyu, chief AI scientist at Tencent and head of the Hunyuan large model, and researchers from the Hunyuan team appeared in the r/LocalLLaMA community on Reddit and conducted a two-hour AMA (Ask Me Anything). This is the first time the Hunyuan team has publicly answered questions in the developer community. It has received more than 300 messages, covering topics ranging from architecture choices to security evaluation.
Less than a month ago, Hunyuan released two models one after another. On November 4, Hy4 ranked among the world's strongest open source models for the first time, ranking second among Chinese models on the intelligence index list of the third-party evaluation agency Artificial Analysis, and ranking first among open source models on the Agent Arena list. Hy4 flash, released more than a week later, is positioned as "the ultimate cost-effective model for daily work", with a single task cost only one-twentieth of Hy4.
Nine months, two releases
From the perspective of time rhythm:
Only nine months have passed since the reconstruction in February.
The division of labor between Hy4 and Hy4 flash corresponds to two routes. The former expands model size and data scale and explores the upper limit of intelligence, while the latter compresses costs and makes it affordable for daily tasks.
The team confirmed in the Q&A that the scale of the next-generation Hy model "will definitely be further significantly expanded." A developer asked about the fact that companies such as ByteDance and Kimi are promoting ultra-large-scale models with 3T or even 5T parameters, while Hunyuan is still at the 700B level. The team's reply was that Hy3 and Hy4 have shown that smaller models that are solidly built can often outperform larger models at lower costs and learn more in the process.
At the other end of the product line, Hy4 mini is already on the roadmap, 35B-A3B Flash-Lite is confirmed to be in progress, smaller open source image models are under consideration, and there is new news about video models. For on-device visual understanding, the team’s judgment is that the A3B model with approximately 3 billion activated parameters can already undertake the task of understanding screenshots and charts and conducting inferences.
There is also a place for cyclic MoE and continuous learning on the roadmap. The response to the former is "of course it is possible, please stay tuned", while the latter is being actively studied. The training infrastructure supporting research and development is also being iterated, and Hyra 3.0 is already under development.
Trade-offs precede parameters
When asked why they ultimately chose MLA instead of GQA, the team revealed a history of repeated trade-offs. The computational intensity of MLA is significantly higher than that of GQA. When performing speculative decoding on the inference GPU, the bottleneck will fall on the computing power. However, under the same KV cache size, the model quality of MLA is better than that of GQA. After the introduction of DSA, the computational intensity of the decoding stage is reduced, and the balance of trade-offs is tilted.
Several other designs of Hy4 flash continue the same set of trade-offs. SWA reduces the KV cache to a quarter of its original size, and service costs are significantly reduced. Coupled with the design of offloading 64B Engram parameters to the host memory, models with 12B activated parameters can be run smoothly with configurations above 128GB.
A developer listed a set of sparsity comparisons. Hy4 flash without Engram has a total parameter of 156B, an activation of 12B, and an activation ratio of 7.7%. Qwen3.8-Flash is 4.8%, and DSv4.1 is as low as 2.9%. How important is higher activation density to intelligence? The team's answer is based on the scale ladder experiment: when the amount of activation parameters is fixed, increasing sparsity can improve computing efficiency. However, keeping the total parameters unchanged and simply reducing the activation parameters will significantly damage the model quality, because the amount of activation parameters essentially determines the computational budget.
In the team’s opinion, the too sparse MoE architecture is not friendly to reasoning and takes up a lot of video memory but cannot fully utilize the computing power. Under the same throughput, it is possible to run models with the same total parameters and more activation parameters. As for the end side, it is a different set of ideas. The architecture needs to be designed more closely with the GPU hardware of the edge device.
In the answer to the quantization scheme, the down projection is more sensitive to quantization due to outliers in the activation value, and will be allocated more bits. The bit width is selected based on the Hessian sensitivity. After quantization, the router will be trained together with QAT to maintain stability, and the team prefers full-parameter QAT because the model quality is better. This set of answers is the most detailed part of the project in the entire Q&A.
The hardest thing is overthinking
When asked about the most difficult behavior to improve in the post-training phase, the reply was only four words: overthinking. Hy4 pursues the upper limit of intelligence, and the long thinking chain is a known problem. The solution has the color of a closed-loop product, distilling Hy4's long thinking chain into the cost-effective Hy4 flash, and then ensuring a shorter thinking chain for the latter through reinforcement learning.
Some developers asked questions on the grounds of writing, saying that if a model explained things more easily, it would be an important reason for them to switch from Qwen 3.8 Flash Next. The team responded that creative writing and readability are very important and are being improved as key directions, adding that "sometimes, a good example is more helpful than five paragraphs." They also admitted to paying attention to the observation that the writing ability of large models has generally deteriorated, and believed that it may be related to the training objectives. They hope that the models can write "more like humans and less routine expressions."
Regarding whether Hy4 can operate autonomously and develop next-generation models, the response stated that it is still early days for recursive self-improvement capabilities, but several internal benchmarks have been developed to evaluate key capabilities. When asked to rank Hy4 according to the Levels of AGI, team members gave a self-assessment of 2.5.
Interspersed between technical issues are memories of the research and development process. The expansion from Hy3 to Hy4 was described as the most difficult moment. The data ablation experiment did not meet expectations, and the beginning of training was full of uncertainty. "I was quite nervous when the training just started." The team's recollection of the final result is that "it felt like putting all the chips on the table and actually winning."
Explaining the restrictions clearly is also a kind of open source
In order to responsibly open source, Hunyuan filters network security data mainly focused on vulnerability exploitation and offensive attacks during training, cooperates with training-level denial-of-answer protection, and conducts red team assessments. Some developers questioned that this would weaken defenders, limit reverse engineering, and hurt malware analysis and forensics. The evaluation conclusion given by the team is that Hy4 has strong vulnerability discovery capabilities but relatively weak vulnerability exploitation capabilities. Excessive rejection is recognized as still an important direction for improvement.
The adaptation of llama.cpp is in progress and will be released to the community soon. Cooperation with vllm and sgl is also in-depth. Hy4 and Hy4 flash have been widely used in OpenCode, Claude Code and Cline. The answer is specific to the project and tool name. During the Q&A, developers also interjected actual testing. Hy4 flash became the first model with perfect scores on its internal agent programming and PR review benchmarks, and the completion speed was faster than Qwen 3.8 Flash Next, GLM 5.3 Flash and DeepSeek V4.1 Flash.
As for why the results of common benchmarks such as GPQA Diamond and SWE Bench are not announced, the team’s explanation is that these benchmarks were launched earlier and the scores have been saturated. What they show are newer and more difficult benchmarks such as Agents’ Last Exam and ProgramBench. The principle of evaluation authenticity has changed from an internal creed to an external stance.
In more than 300 comments, developers are not only concerned about running scores, but also are repeatedly asked about real task performance, usage costs and ecological adaptation. The competitive dimension of the open source model is wider than what the list shows. The priority presented by the Hunyuan team in two hours was to lay a solid foundation first, make the model more usable in real tasks, and then continue to explore the boundaries of capabilities.
The unanswered questions, overthinking, creative writing, and recursive self-improvement that are acknowledged firsthand form the roadmap for the next iteration. Nine months ago, Hunyuan rebuilt its pre-training and reinforcement learning infrastructure. Nine months later, two models and a two-hour Q&A constituted the staged results after reconstruction. Lay the foundation first, then explore the boundaries.
Leave the rest to the next version. (This article was first published on Titanium Media APP, author|AGI-Signal, editor|Qin Conghui)
Respect for the original work, if you are qualified, you should be appreciated.
According to the real-name requirement of the Cybersecurity Law, please bind your mobile phone number before posting a comment.
Express
11:05
U.S. cargo Dragon spacecraft breaks away from the space station and sets off for return to Earth
10:31
U.S. presidential envoy discusses U.S.-Iran mediation work with senior Qatari officials
10:30
Quantum computer "high-speed memory" is realized on a real machine for the first time in the world
10:27
Agency: Demand has cooled, and prices for TVs, monitors and laptop panels may remain flat across the board in November.
10:25
Ming-Chi Kuo: Google is expected to adjust the shipment proportion of TPU 8t and 8i series in 2027
10:22
The phased adjustment has not changed the background, and many institutions are optimistic about the valuation and resilience of A-shares.
10:21
It is reported on the Internet that the trunk of Leipao A10 was crushed by several boxes of beer. Leapao responded.
10:19
International gold prices continue to decline
10:16
Federal Reserve official Paulson: Supports prudent interest rate hikes and can continue to raise interest rates if necessary
10:15
If the sale to SoftBank cannot be completed, ABB is reported to be considering spinning off its robotics business.
10:12
Anthropic IPO received approximately US$115 billion to US$120 billion in subscription demand from retail investors
10:10
The price of gold jewelry continues to fall, with Lao Fengxiang falling by 34 yuan in a single day
Food delivery company Wonder is considering selling Grubhub
10:09
Databricks is reported to be negotiating a new round of financing, with a valuation expected to reach US$225 billion
10:08
CrowdStrike reportedly in talks to acquire cybersecurity company Zscaler
10:06
Apollo and Goldman Sachs complete pricing of $44 billion SpaceX GPU financing
‘Big Short’ investor Michael Burry reduces holdings of Nvidia put options, locking in some gains
10:04
"Grand Theft Auto 6" retail sales hit $3.79 billion on first day, setting entertainment industry record
09:59
Iran's Foreign Minister: The "two-day plan" is still on the table and is willing to discuss it with the United States
09:53
Trump: China-Russia-US presidents’ meeting “unprecedented” and “of great significance”
Scan to download App



