With the same video, what can the AI still determine after the original audio is turned off?

artifactNovember 21, 2026release AiFun
35 0

There are two envelopes in the frame. The narrator says, “Send the blue one,” but doesn’t specify where it is; the camera shows the left and right positions, but doesn’t provide any instructions. True audio-visual comprehension isn’t about simply repeating these two pieces of information separately, but rather about matching the “blue” in the audio with the position in the visual. WeMM-Video, released by Tencent WeChat’s Visual Team on November 19, 2026, provides a reproducible model family for this type of task. The most pertinent question is not whether it carries the label of “understanding video,” but rather: which types of problems are genuinely solvable because of the original audio, and which answers are still merely guesses by the model?

How do I import the original audio into the model?

Section 2 of the Technical ReportThe results show that the 2B, 4B, and 9B versions are based on Qwen3.5 at their respective scales, while the flagship 35B-A3B is based on Qwen3.6-35B-A3B. The development team retained the foundation’s visual encoding pipeline and added a new audio processing pipeline consisting of a Whisper-style encoder and a Q-Former. Audio is resampled to 16 kHz and divided into 30-second chunks; local windows of approximately 0.1 seconds are compressed into a single audio token. When processing videos with audio, the model receives audio and visual tokens in a time-interleaved manner, supplemented by text timestamps.

This differs from “frame-by-frame analysis plus full-text transcription”: the model can directly process the original audio track and is not limited to the words derived from the transcription; therefore, in principle, it can utilize intonation, non-verbal sounds, or the sequence in which sounds occur. However, the encoding time window does not equate to event localization accuracy; video sampling may omit brief movements, and audio clues may be lost due to compression or noise. The output remains text, and the time information in the report does not guarantee clip-level localization, let alone the ability to search the entire video library.

Three scores are more useful than a single overall ranking.

Table 1 of the ReportThe results from the corresponding table are as follows. The numbers are presented as they appear in the original table; they cannot be directly averaged across different rows to produce a single “video comprehension score.”

Collection of Problems Flagship Instruct Flagship Thinking Gemini 3.0 Flash
Video-MME (with audio) 84.2 86.4 88.6
AV-SpeakerBench 66.5 72.6 75.9
JointAVBench 67.4 65.9 68.4

Thinking improved by 6.1 points on the speaker-related dataset but decreased by 1.5 points on JointAVBench. If the task primarily involves identifying “who verbally confirmed what and when,” it is worth including Thinking in the experiment; however, if the judgment itself is simple or latency-sensitive, it should not be enabled by default. The report also notes that when public metrics are available for comparison models, those results are cited; otherwise, the authors conduct their own evaluations. The comparison here is based on the results presented in the report, not on conclusions drawn from uniformly rerunning all models; differences in cross-table scores between audio and silent modes cannot be directly attributed to the audio encoder.

How can I tell that it wasn't just a lucky break?

First, create four controlled versions using the same 10-second clip without subtitles. Videos A and B have identical frames; the only difference is that the instruction at the 3-second mark is changed from “Send red” to “Send blue.” The muted version retains the visuals but removes the instructions; the audio-only version retains the instructions but lacks visual cues. Next, film a set of footage where the red and blue elements are swapped left and right to prevent the model from guessing correctly based on positional preferences. Do not include the answer in the filenames; start a new session for each test, keeping the checkpoint, sampling frame rate, maximum frame count, and prompt fixed, and record the inference patterns.Official Deployment ExampleDistinguishing between frame-only video Compared to those with original audio video_audio; The API path can also be set explicitly use_audio.

Your score isn't determined solely by how many questions you answer correctly. First,Flip in Pairs: The original sound has changed—do A and B change as expected? Second,Ownership of Evidence: Can you specify, respectively, that sound provides color and images provide location? Third,Don't speculate when there's no evidence: Can both "silent" and "audio-only" clearly indicate what is missing? When testing real-world business footage, separately list cases involving overlapping voices, background noise, fast-moving action, and long videos, and note whether the errors relate to listening, viewing, synchronization, or inference. If specific timestamps are required, set an acceptable margin of error in advance and verify using the original footage.

For training videos and process reviews, I first use a model to identify segments worth reviewing, and then have a person verify the specific timestamps and exact quotes. If, within the same domain’s training data, the audio track does not provide a consistent performance gain over the “video + transcript” baseline—or if the gain is outweighed by latency and false positives—there is no need to deploy both input streams to the production system.

© Copyright notes

Related articles

No comments

none
No comments...