On 2026-11-03/04, the APB Full suite ran against DeepSeek V4.1 Flash 1017 on five providers through AIHubMix. The same model, questions, parameters and request scripts were used for every provider; the model is listed on the AIHubMix model page, and the five provider entries are on the models page. This APB Profile covers Alibaba Cloud only: what each section measured and what came back.
This post reports scores without ranking providers or recommending one. Q, F, P, K and R come first; K is a sub-score of P. The overall score is a summary index, not a certification, and has no confidence interval. The scoring rules are defined in AIHubMix Provider Benchmark Standard v2.0.
Scores across five providers
Bold marks this post's provider. Providers appear in a fixed order, not a ranking. K is a sub-score of P at 20%, shown separately for reference. The overall score is a summary index with no confidence interval. DeepSeek official channel and Alibaba Cloud differ by 0.37 overall and are not distinguishable.
At a glance
- Answer quality: Q 41.75. BFCL 87.00%, HLE 13.29%, AA-LCR 53.60%. Against the official channel the difference is −0.71 percentage points, with a 95% interval of −1.72 to +0.26, so the two are not distinguishable.
- Function support: F 89.78. 155 of the 177 checks in the denominator passed. An invalid tool schema returned 503 instead of a parameter error, context overflow did not return the documented error type, and an isolated tool result returned 200.
- Response speed: P 88.58. The first-content-token median was 1.34 s at single concurrency and 1.35 s at 10-concurrency, TPOT was 4.69 ms, and all 180 speed requests were valid.
- Cache: K 99.37. All 18 hot requests across the 8k and 32k prefixes hit, and no cold control read the cache.
- Service stability: R 100. All 300 dedicated requests succeeded on the first attempt.
Test setup
The run went through AIHubMix by selecting the provider model ID; the quick start documents the route. DeepSeek's pricing page lists the model's published rates.
| Item | Detail |
|---|---|
| Model | DeepSeek V4.1 Flash 1017 |
| Requested model ID | alicloud-deepseek-v4.1-flash-1017 |
| Returned model name | deepseek-v4.1-flash-1017 |
| Access route | Via AIHubMix by selecting the provider model ID, not by connecting to the provider's own account |
| Parameters | Non-thinking mode, temperature 1.0, max_tokens 16,384, timeout 300 s, no automatic retries |
| Thinking off | All requests used reasoning_effort=none |
| Time | 2026-11-03 18:07 – 11-04 16:33 UTC+8. Five providers shared one Mac mini, network and account. |
| Samples | Answer quality 4,250 question-runs (BFCL 500 × 3 rounds, HLE 250 × 10, AA-LCR 25 × 10); function cases 60 × 3 windows; speed 90 single-concurrency and 90 at 10-concurrency; stability 300; cache 30 (8k and 32k prefixes, 3 groups each) |
Q — answer quality: 41.75
Answer quality used three question sets: 500 BFCL questions at 3 rounds each, 250 HLE questions at 10 rounds each, and 25 AA-LCR questions at 10 rounds each. BFCL was scored by automatic rules; HLE and AA-LCR were judged by the judge model deepseek-flash. All five providers answered the same questions, and the three sets were combined into one accuracy. Ten responses were removed because the questions could not be scored, leaving 4,240 scored question-runs.
| Task | Questions | Correct | Accuracy | Failures |
|---|---|---|---|---|
| BFCL tool calls | 500 × 3 | 1,305/1,500 | 87.00% | 0 |
| HLE hard QA | 250 × 10 | 331/2,490 | 13.29% | 8 |
| AA-LCR long-doc reasoning | 25 × 10 | 134/250 | 53.60% | 0 |
| Combined · 95% interval 39.56%–43.85% | — | 1,770/4,240 | 41.75% | 8 |
Service failures (failed requests or truncated output) stay in the denominator and count as incorrect; they are not removed. For Alibaba Cloud, HLE output reached the max_tokens limit and was truncated 8 times.
BFCL answers each question in 3 rounds; the table shows the 3 rounds combined.
| BFCL category | Passed | Pass rate |
|---|---|---|
| simple_python: one function, correct call and arguments | 332/360 | 92.22% |
| multiple: choose the right function from several | 270/300 | 90.00% |
| parallel: call the same function several times in parallel | 273/300 | 91.00% |
| parallel_multiple: call several different functions in parallel | 212/240 | 88.33% |
| irrelevance: no function is relevant, none should be called | 113/150 | 75.33% |
| ahm_multi_turn: three-stage tool chain | 105/150 | 70.00% |
Compared with the other four
Against the official channel: the accuracy difference in percentage points, with the 95% interval shown as whiskers; an interval that crosses 0 counts as no distinction. The HLE and AA-LCR judge deepseek-flash belongs to the same DeepSeek family as the model under test and was not calibrated, so its scoring may be lenient.
F — function support: 89.78
Function cases check whether the interface behavior follows the protocol, not whether the model answers well. The 60 cases fall into six groups, each run in three windows, for 180 checks, judged against the format each provider documents. Checks that were not applicable, invalid or unobserved are excluded from the denominator. Alibaba Cloud passed 155 of the 177 checks in the denominator; group weights give 89.78.
The weighted score is computed from group weights, not a simple pass count over 177.
| Group | Weight | Passed | Excluded | Group score |
|---|---|---|---|---|
| Streaming and message protocol | 20% | 30/30 | — | 100.00 |
| Tool interface | 20% | 30/30 | — | 100.00 |
| Output constraints and parameters | 20% | 30/30 | — | 100.00 |
| Context and state | 15% | 26/30 | — | 86.67 |
| Errors and recovery | 15% | 18/30 | — | 60.00 |
| Metering and cache protocol | 10% | 21/27 | 3 | 77.78 |
| Total · weighted score | — | 155/177 | 3 | 89.78 |
A window count of 3/3 means the case failed in all three windows.
| Failed case | Windows failed | Reason |
|---|---|---|
| Context overflow (context and state) | 3/3 | Returned an error, but not the documented context-length error |
| messages format error (errors and recovery) | 3/3 | Returned 400 with an empty response body, no structured error description |
| Tool schema invalid (errors and recovery) | 3/3 | Should return a parameter error; returned 503 |
| Isolated tool result (errors and recovery) | 3/3 | Should return a parameter error; returned 200 |
| stream_options format error (errors and recovery) | 3/3 | Returned 400 with an empty response body, no structured error description |
| Reasoning token metering (metering and cache protocol) | 3/3 | No reasoning token count returned |
| Tool definitions counted as input (metering and cache protocol) | 3/3 | With tool definitions, reported input tokens were not higher than without them |
| Request isolation (context and state) | 1/3 | The first step did not reply only STORED as required, so the isolation check could not proceed |
These checks are excluded from the denominator and count as neither pass nor fail.
| Status | Case | Count | Reason |
|---|---|---|---|
| Not applicable | Explicit cache_control | 3 | Registered in advance as unsupported by this interface |
The messages format error and stream_options format error cases failed for all five providers: each returned 400 with an empty body, so the client received no structured error description. Because all five show the same behavior, it more likely sits in the AIHubMix gateway layer and needs separate investigation. Reasoning token metering also failed for all five, but for different reasons: ByteDance and Fireworks reported reasoning tokens even with thinking off, while the other three returned no reasoning token count.
Compared with the other four
P — response speed: 88.58
Each request carried about 10,000 input tokens and asked for about 1,100 tokens of output. In three windows, 30 requests were sent at single concurrency and 30 at 10-concurrency, for 180 in total. The first-content-token wait (TTFA) is measured per valid request, and the table reports medians. TPOT is estimated from streaming chunk timings, not hardware per-token timing; throughput is valid output tokens at 10-concurrency divided by the batch wall-clock time.
| Metric | Result |
|---|---|
| First content token TTFA | Single-concurrency median 1.34 s, 10-concurrency median 1.35 s |
| Generation speed TPOT | Single-concurrency 4.63 ms/token, 10-concurrency 4.69 ms/token |
| Effective throughput (goodput) | 10-concurrency 1,313.6 token/s |
| Valid requests | Single-concurrency 90/90, 10-concurrency 90/90 |
| Component scores | TTFA 59.65, TPOT 100.00, throughput 100.00; TTFA and TPOT take the lower of the two levels, weighted 35% / 35% / 30%, response score 85.88 |
| P score | Response score 85.88 at 80%, cache K 99.37 at 20%, total 88.58 |
Compared with the other four
All five providers shared the same machine, network and account. The three windows were scheduled provider by provider, so collection periods do not fully overlap and small gaps should not be over-interpreted.
K — cache: 99.37
Each group is five requests: the first carries a fixed prefix of about 8k or 32k tokens and seeds the cache; requests 2–4 keep the prefix and change only the question at the end, at 5, 30 and 120 seconds; the fifth uses a new prefix as a cold control. Three windows × two lengths make 30 requests. K is the cached tokens read on hot requests divided by the tokens that should be reusable under the rule, with each length weighted half; K is 20% of P. The matrix shows server-reported cached-token reads.
| Prefix | Window | Seed (request 1) | Hot at 5 s | Hot at 30 s | Hot at 120 s | Cold control (request 5) |
|---|---|---|---|---|---|---|
| 8k | Window 1 | 0 | 7,936 | 7,936 | 7,936 | 0 |
| 8k | Window 2 | 0 | 7,936 | 7,936 | 7,936 | 0 |
| 8k | Window 3 | 0 | 7,936 | 7,936 | 7,936 | 0 |
| 32k | Window 1 | 0 | 32,000 | 32,000 | 32,000 | 0 |
| 32k | Window 2 | 0 | 32,000 | 32,000 | 32,000 | 0 |
| 32k | Window 3 | 0 | 32,000 | 32,000 | 32,000 | 0 |
| Prefix | Hot hits | Tokens read | Should be reusable | Score |
|---|---|---|---|---|
| 8k | 9/9 | 71,424 | 72,270 | 98.83 |
| 32k | 9/9 | 288,000 | 288,279 | 99.90 |
| K (8k and 32k, half each) | 99.37 |
All 9 hot 8k requests hit, reading 7,936 tokens each; all 9 hot 32k requests hit, reading 32,000 tokens each. The seed request and the cold control read no cache, so hot and cold requests separate cleanly.
Compared with the other four
R — service stability: 100
The 300 requests covered four types — short text, 10k long input, tool calls and multi-turn history — and were sent at single concurrency across three windows. Only the first response was checked, for validity and completeness; answer correctness was not checked and no retries were made. Failures in the answer-quality and speed sections count in Q and P respectively, not again in R.
| Request type | First-attempt success |
|---|---|
| Short text | 120/120 |
| 10k long input | 90/90 |
| Tool calls | 60/60 |
| Multi-turn history | 30/30 |
| Total · 95% interval 98.78%–100% | 300/300 |
All 300 requests returned a valid, complete response on the first attempt. R 100 covers only these 300 dedicated requests and does not mean the whole chain had no failures. The lower bound of the 95% interval is 98.78%: even with 300/300, the failure rate is likely below 1.22%.
Compared with the other four
R covers only these 300 dedicated requests; failures in answer quality and speed are recorded in Q and P, not counted again in R.
| Provider | First-attempt success | 95% interval |
|---|---|---|
| ByteDance | 300/300 | 98.78%–100% |
| Baidu | 300/300 | 98.78%–100% |
| Alibaba Cloud | 300/300 | 98.78%–100% |
| DeepSeek official | 300/300 | 98.78%–100% |
| Fireworks | 300/300 | 98.78%–100% |
FAQ
Is the 0.37 overall gap between Alibaba Cloud and the official channel meaningful?
No. The two are not distinguishable on this run: the Q difference is −0.71 percentage points with a 95% interval of −1.72 to +0.26, which includes 0, and the overall score is a summary index without a confidence interval.
What does the Q score combine?
Three sets: BFCL 500 questions × 3 rounds, scored automatically; HLE 250 × 10 and AA-LCR 25 × 10, scored by the judge model deepseek-flash. Combined accuracy is 41.75% over 4,240 scored question-runs.
Why is F 89.78 when 155 of 177 checks passed?
The score is weighted by group, not a simple pass ratio. Errors and recovery scored 60.00 and metering and cache protocol scored 77.78, while the three protocol groups scored 100.00.
Why does R report 100 if other sections had failed requests?
R covers only the 300 dedicated stability requests. Failures in answer quality and speed are recorded in Q and P and are not counted again in R.
Is K 99.37 a cache hit rate?
It is the ratio of cached tokens read on hot requests to the tokens that should be reusable, with 8k and 32k each weighted half. The 8k prefix scored 98.83 and the 32k prefix scored 99.90.
Does the 1.34 s first token mean the full answer arrives in 1.34 s?
No. It is the wait for the first content token. Each request asked for about 1,100 output tokens, and TPOT at 10-concurrency was 4.69 ms per token.
Keep reading: the DeepSeek V4.1 Flash series
- Alibaba Cloud's combined Q differs from the official baseline by 0.71 percentage points, inside an interval that includes 0; the ByteDance profile follows the same comparison into tool-call metering: DeepSeek V4.1 Flash on ByteDance: tool calls match, metering doesn't.
- Alibaba Cloud returned all 180 speed requests valid; the Baidu profile follows the same harness where no available channel affected Q and P: DeepSeek V4.1 Flash on Baidu: no available channel hits Q and P.
- The official channel is the baseline for this post's answer-quality comparison; its own profile covers F 94.00 and 15 invalid single-concurrency probes: DeepSeek V4.1 Flash on the official channel: 94.00 F, 15 invalid probes.
- Alibaba Cloud hit 9/9 hot requests at both prefix lengths; the Fireworks profile covers empty stop replies and 3/9 cache hits at 8k: DeepSeek V4.1 Flash on Fireworks: empty stop replies, 3/9 cache hits.
