On 2026-11-03/04, DeepSeek V4.1 Flash 1017 was tested on five providers through AIHubMix with the APB Full suite: the same questions, parameters and request scripts each time. This APB Profile covers the Baidu run only, section by section.
The numbers below are scores only, not a ranking or a provider recommendation. Q 38.00, F 88.39, P 37.77, K 68.85 and R 100 come first; the overall, 56.48, is a summary index with no confidence interval. The scoring rules are those of AIHubMix Provider Benchmark Standard v2.0.
Scores across five providers
Bold marks this post's provider. Providers are listed in a fixed order, not a ranking. K is a sub-score of P at 20%. The overall is a summary index with no confidence interval. The official channel and Alibaba Cloud differ by 0.37, a difference that is not distinguishable.
At a glance
- Q: 38.00. BFCL 86.40%, HLE 9.36%, AA-LCR 32.80%. The combined result is 4.46 points below the official channel, with a 95% interval of −5.78 to −3.16 that excludes 0. HLE recorded 864 request failures and AA-LCR 100, almost all in the first window, counted as wrong answers.
- F: 88.39. 148 of 174 counted checks passed. The cache-read field and the thinking-token count were not returned; over-context and orphan tool-result cases returned 200.
- P: 37.77. 50 of 90 ten-concurrent probes returned
no_available_channel, so more than half were invalid and TTFA and TPOT were scored at their penalty values. Single-concurrency probes were valid 78 times out of 90. - K: 68.85. 8k hot requests hit 5 of 9 times, with 3 request failures; the 32k prefix hit 8 of 9. Hits are reported in 1,024-token blocks: 7,168 tokens for 8k and 31,744 for 32k.
- R: 100. All 300 dedicated requests succeeded on the first attempt. Channel failures during answering and speed probes are counted under Q and P.
Test setup
Requests used the AIHubMix quick start route with a provider-specific ID, and the parameters follow DeepSeek's API documentation; other provider IDs are listed on the models page. This is not a direct connection to Baidu's own account.
| Field | Value |
|---|---|
| Model | DeepSeek V4.1 Flash 1017 |
| Requested model ID | baidu-deepseek-v4.1-flash-1017 |
| Returned model name | deepseek-v4.1-flash-1017 |
| Access route | Via AIHubMix with the provider-specific model ID; not a direct connection to the provider's own account |
| Parameters | Non-thinking mode, temperature 1.0, max_tokens 16,384, timeout 300 s, no automatic retries |
| Thinking off | Every request used reasoning_effort=none |
| Time | 2026-11-03 18:07 – 11-04 16:33 UTC+8; all five providers shared one Mac mini, network and account |
| Samples | 4,250 answer attempts (BFCL 500 × 3, HLE 250 × 10, AA-LCR 25 × 10); 60 function cases × 3 windows; 90 single-concurrency and 90 ten-concurrent speed probes; 300 stability requests; 30 cache probes (three groups each for 8k and 32k prefixes) |
Q — answer quality: 38.00
Answer quality uses three task types: 500 BFCL tasks answered three times each, 250 HLE tasks answered ten times each, and 25 AA-LCR tasks answered ten times each. BFCL is scored by automatic rules; HLE and AA-LCR are judged correct or incorrect by the judge model deepseek-flash. All five providers received the same task set, and the three types are combined into one accuracy figure. Ten responses were removed because the task could not be judged, leaving 4,240 scored attempts.
| Task | Items | Correct | Accuracy | Failures |
|---|---|---|---|---|
| BFCL tool calls | 500 × 3 | 1,296/1,500 | 86.40% | 0 |
| HLE hard questions, judge-scored | 250 × 10 | 233/2,490 | 9.36% | 875 |
| AA-LCR long-doc reasoning | 25 × 10 | 82/250 | 32.80% | 100 |
| Combined · 95% interval 36.23%–39.58% | — | 1,611/4,240 | 38.00% | 975 |
Service failures, whether a failed request or truncated output, stay in the denominator and count as wrong answers. HLE had 864 request failures (863 in the first window, 1 in the second) and 11 outputs truncated at the max_tokens limit. AA-LCR had 100 request failures, all in the first window.
BFCL is answered three times per task; the table shows the three rounds combined.
| BFCL category | Passed | Pass rate |
|---|---|---|
| simple_python: one function, correct call and arguments | 336/360 | 93.33% |
| multiple: choose the right function from several | 269/300 | 89.67% |
| parallel: call one function several times in parallel | 268/300 | 89.33% |
| parallel_multiple: call several different functions in parallel | 206/240 | 85.83% |
| irrelevance: no function applies, do not call | 113/150 | 75.33% |
| ahm_multi_turn: three-stage tool chain | 104/150 | 69.33% |
Compared with the other four
The difference is in percentage points against the official channel, with the 95% interval shown as whiskers. An interval that crosses 0 is treated as indistinguishable. The HLE and AA-LCR judge, deepseek-flash, belongs to the same DeepSeek family as the model under test and was not calibrated, so its judgments may be lenient.
F — function support: 88.39
Function cases test whether the interface behaves as the protocol specifies, not whether the model answers well. Sixty cases in six groups were run once in each of three windows, for 180 checks, judged against the format the provider documents. Checks that were not applicable, invalid or unobserved are excluded from the denominator. Baidu counted 174 checks and passed 148, for a weighted score of 88.39.
The group score is weighted by each group's share, not a simple 148/174.
| Case group | Weight | Passed | Excluded | Group score |
|---|---|---|---|---|
| Streaming and message protocol | 20% | 27/27 | 3 | 100.00 |
| Tool interface | 20% | 30/30 | — | 100.00 |
| Output constraints and parameters | 20% | 29/30 | — | 96.67 |
| Context and state | 15% | 26/30 | — | 86.67 |
| Error and recovery | 15% | 21/30 | — | 70.00 |
| Metering and cache protocol | 10% | 15/27 | 3 | 55.56 |
| Total, weighted | 100% | 148/174 | 6 | 88.39 |
| Failed case | Windows failed | Reason |
|---|---|---|
| Over context (context and state) | 3/3 | Should return a parameter error; returned 200 |
| Malformed messages (error and recovery) | 3/3 | Returned 400 with an empty body, no structured error |
| Orphan tool result (error and recovery) | 3/3 | Should return a parameter error; returned 200 |
| Malformed stream_options (error and recovery) | 3/3 | Returned 400 with an empty body, no structured error |
| Cache read field (metering and cache protocol) | 3/3 | No cache-read field returned; the reading cannot be confirmed |
| Thinking-token metering (metering and cache protocol) | 3/3 | No thinking-token count returned |
| Tool definitions counted in input (metering and cache protocol) | 3/3 | With tool definitions, reported input tokens were not higher than without |
| Input count consistent before and after cache (metering and cache protocol) | 3/3 | No cache-read field returned; the reading cannot be confirmed |
| Sampling diversity (output constraints and parameters) | 1/3 | Repeated outputs at temperature 0 and 1 differed too little, below the pre-registered standard |
| Isolation between requests (context and state) | 1/3 | The first step did not reply only STORED as required, so the isolation check could not proceed |
These checks are excluded from the denominator: neither passed nor failed.
| Status | Case | Count | Reason |
|---|---|---|---|
| Unobserved | Streaming stop sequence | 3 | The request for this step was not sent after the previous step, so it cannot be judged |
| Not applicable | Explicit cache_control | 3 | Pre-registered as unsupported by this interface |
Malformed messages and malformed stream_options failed for all five providers: both returned 400 with an empty body, leaving no structured error for the client. A failure shared by all five is more likely to sit in the AIHubMix gateway layer and needs separate investigation. Thinking-token metering also failed for all five, for different reasons: ByteDance and Fireworks reported thinking tokens even with thinking off, while the other three returned no thinking-token count.
Compared with the other four
P — response speed: 37.77
Each request carried about 10,000 input tokens and asked for a body of about 1,100 output tokens. Across three windows, 30 single-concurrency and 30 ten-concurrent probes were sent in each, for 180 requests.
| Metric | Result |
|---|---|
| First-token TTFA | Single median 2.26 s (valid-request median 2.05 s); ten-concurrent more than half invalid, median counted as infinity (valid-request median 1.86 s) |
| Generation speed TPOT | Single 3.48 ms/token; ten-concurrent 50.00 ms/token (more than half invalid, penalty value) |
| Goodput (valid throughput) | Ten-concurrent 1,329.2 token/s |
| Valid requests | Single 78/90 (first window 18/30); ten-concurrent 40/90 (first window 0/30, third window 10/30) |
| Invalid reason | 62 probes returned 400 no_available_channel |
| Component score | TTFA 0.00, TPOT 0.00, throughput 100.00; response score 30.00 |
| P composition | Response score 30.00 at 80%, cache K 68.85 at 20%, total 37.77 |
TPOT is estimated from streamed chunks, not measured per token on the hardware. Goodput is valid output tokens at ten-concurrent load divided by batch wall-clock time.
In the ten-concurrent tier, 50 of the 90 probes returned no_available_channel. With more than half invalid, the median first-token time is treated as infinity and both TTFA and TPOT take their penalty values. The single-concurrency tier stayed valid for 78 of 90 probes.
All five providers shared one machine, network and account. The three windows were scheduled provider by provider, so collection periods do not fully overlap; small gaps should not be over-read.
K — cache: 68.85
Each group sends five requests. The first carries a fixed prefix of about 8k or 32k tokens and seeds the cache. Requests 2–4 keep the prefix and change only the question at the end, at gaps of 5 s, 30 s and 120 s. The fifth uses a new prefix as a cold control. Three windows × two lengths make 30 requests. K is the cache tokens read by hot requests divided by the tokens that should be reusable under the rules, with each length weighted half. K is 20% of P.
The table shows the cache-read token count reported by the server; "not returned" means the field was absent from the response.
| Prefix | Window | Seed | +5 s | +30 s | +120 s | Cold control |
|---|---|---|---|---|---|---|
| 8k | 1 | Failed | Failed | Failed | Not returned | Not returned |
| 8k | 2 | Not returned | 7,168 | 7,168 | Failed | Failed |
| 8k | 3 | Not returned | 7,168 | 7,168 | 7,168 | Not returned |
| 32k | 1 | Not returned | Not returned | 31,744 | 31,744 | Not returned |
| 32k | 2 | Not returned | 31,744 | 31,744 | 31,744 | Not returned |
| 32k | 3 | Not returned | 31,744 | 31,744 | 31,744 | Not returned |
The 8k prefix hit 5 of 9 hot requests, reading 35,840 of 72,252 tokens that should have been reusable, for 49.60. The 32k prefix hit 8 of 9, reading 253,952 of 288,261, for 88.10. K is 68.85 with the two lengths weighted equally. Five 8k requests failed overall: window 1's seed, 5 s and 30 s requests, and window 2's 120 s and cold-control requests, all with no_available_channel. That leaves three failed hot requests in the 8k tier.
On a hit, the reported reading is counted in 1,024-token blocks: 7,168 tokens for the 8k prefix and 31,744 for the 32k prefix. On a miss, the usage block omits prompt_tokens_details; the table records "not returned" and the value is counted as 0.
Compared with the other four
R — service stability: 100
The 300 requests fall into four types: short text, long input, tool call and multi-turn history. They were sent at single concurrency across three windows. Only whether the first response was valid and complete is checked; answers are not graded and nothing is retried. Failures during answering and speed probes are counted under Q and P, not again under R.
| Request type | First-attempt success |
|---|---|
| Short text | 120/120 |
| 10k long input | 90/90 |
| Tool call | 60/60 |
| Multi-turn history | 30/30 |
| Total | 300/300, 95% CI 98.78%–100% |
All 300 requests returned a valid, complete response on the first attempt. R 100 covers only these 300 dedicated requests; it does not mean the whole path had no failures. With 300 of 300 successful, the lower bound of the 95% interval is 98.78%, so the failure rate is likely below 1.22%.
Compared with the other four
R covers only these 300 dedicated requests; failures during answering and speed probes are recorded under Q and P, not again under R.
| Provider | First-attempt success | 95% interval |
|---|---|---|
| ByteDance | 300/300 | 98.78%–100% |
| Baidu | 300/300 | 98.78%–100% |
| Alibaba Cloud | 300/300 | 98.78%–100% |
| DeepSeek official | 300/300 | 98.78%–100% |
| Fireworks | 300/300 | 98.78%–100% |
FAQ
What does the overall score represent?
It is a summary index with no confidence interval, not a certification. This post reports Q, F, P, K and R and makes no provider recommendation.
Why did half the ten-concurrent probes fail?
50 of the 90 returned no_available_channel. Because more than half were invalid, the median first-token time is treated as infinity and TTFA and TPOT are scored at their penalty values.
Does R 100 mean there were no channel failures?
No. R covers 300 dedicated requests, all first-attempt successes. The 62 no_available_channel responses in the speed run and the failures in the answering run are counted under P and Q.
Why is K 68.85 when the 32k prefix hit 8 of 9?
The two lengths weigh equally. The 8k prefix hit 5 of 9 hot requests and scored 49.60, while 32k hit 8 of 9 and scored 88.10; the average is 68.85.
Is Baidu's answer quality distinguishable from the official channel?
Yes. The combined Q difference is −4.46 percentage points, with a 95% interval of −5.78 to −3.16 that excludes 0. It is the one provider result below the official channel whose interval excludes 0.
Which checks were excluded from F?
Six: three streaming stop-sequence checks that were never observed because the preceding step's request was not sent, and three explicit cache_control checks pre-registered as unsupported by this interface.
Keep reading: the DeepSeek V4.1 Flash series
- Baidu's BFCL accuracy was close to the official channel; the ByteDance post follows the same suite through its tool-call and metering results: DeepSeek V4.1 Flash on ByteDance: tool calls match, metering doesn't
- Baidu's single-concurrency TTFA of 2.26 s contrasts with a run that reached 1.34 s: DeepSeek V4.1 Flash on Alibaba Cloud: 1.34s first token, 99.37 cache
- Baidu's F 88.39 sits against a reference run at 94.00 with 15 invalid probes: DeepSeek V4.1 Flash on the official channel: 94.00 F, 15 invalid probes
- Baidu's 8k cache hits of 5/9 compare with a run at 3/9 and empty stop replies: DeepSeek V4.1 Flash on Fireworks: empty stop replies, 3/9 cache hits
