On 2026-11-03/04, DeepSeek V4.1 Flash 1017 was run through the APB Full suite on five providers, all reached through AIHubMix with the same questions, parameters and request scripts. This APB Profile covers Fireworks only. The model is listed as DeepSeek V4.1 Flash 1017.
Scores are Q 41.42, F 71.78, P 79.51, K 60.68 and R 100; the overall index is 64.10. This post reports scores only, with no ranking or provider recommendation. Q, F, P, K and R come first; the overall index is a summary number with no confidence interval. The standard is AIHubMix Provider Benchmark Standard v2.0.
Scores across five providers
Bold marks this post's provider. Fixed provider order, not a ranking. K is a sub-score of P at 20%. The overall index is a summary number with no confidence interval. DeepSeek official channel and Alibaba Cloud differ by 0.37 and are not distinguishable.
At a glance
- Q — answer quality: 41.42. BFCL 85.53%, HLE 14.38%, AA-LCR 46.00%. Against the official channel: −1.04 percentage points, 95% CI −2.14 to +0.05, not distinguishable. All 34 AA-LCR failures were in window 2 and counted as wrong.
- F — function support: 71.78. 122/177 counted checks passed. Both stop cases returned an empty body;
reasoning_effort=nonedid not turn thinking off; a 10k input was reported as 10,026 tokens against a native count of 10,000. - P — response speed: 79.51. Response score 84.22, cache K at 20%. TTFA median 1.63 s single, 1.64 s at 10 concurrent; TPOT 3.29 ms. 1 of 180 outputs was a single token and marked invalid.
- K — cache: 60.68. K probes used the documented session affinity. 32k hot 8/9; 8k hot 3/9.
- R — service stability: 100. 300/300 dedicated requests succeeded on the first attempt.
Test setup
| Item | Setting |
|---|---|
| Model | DeepSeek V4.1 Flash 1017 |
| Requested model ID | fireworks-deepseek-v4.1-flash-1017 |
| Returned model name | accounts/fireworks/models/deepseek-v4p1-flash-1017 |
| Access route | AIHubMix provider model ID, not the provider's own account |
| Parameters | Non-thinking, temperature 1.0, max_tokens 16,384, timeout 300 s, no retry |
| Thinking off | Native thinking.type=disabled; reasoning_effort=none does not turn thinking off. Function cases kept reasoning_effort=none. |
| Time | 2026-11-03 18:07 – 11-04 16:33 UTC+8. One Mac mini, network and account for all five. |
| Samples | Q 4,250 (BFCL 500×3, HLE 250×10, AA-LCR 25×10); F 60×3; P 90+90; R 300; K 30 |
| Extra settings | K probes carried the documented user field and x-session-affinity header for replica affinity. |
Requests used the AIHubMix quick start; model IDs are on the models page. Thinking off used native thinking.type=disabled from the model's API documentation.
Q — answer quality: 41.42
The suite ran 500 BFCL questions × 3 rounds, 250 HLE × 10 and 25 AA-LCR × 10. BFCL was auto-scored; HLE and AA-LCR were judged by deepseek-flash. All five received the same questions. Ten unscoreable answers were removed, leaving 4,240 scored instances.
| Task | Questions | Correct | Accuracy | Failures |
|---|---|---|---|---|
| BFCL tool calls | 500 × 3 | 1,283/1,500 | 85.53% | 2 |
| HLE hard QA | 250 × 10 | 358/2,490 | 14.38% | 5 |
| AA-LCR long-doc reasoning | 25 × 10 | 115/250 | 46.00% | 34 |
| Total · 95% CI 39.13%–43.59% | — | 1,756/4,240 | 41.42% | 41 |
Failures stay in the denominator as wrong: BFCL 2 (windows 2 and 3), HLE 5 truncated at max_tokens, AA-LCR 34 all in window 2.
| BFCL category | Passed | Pass rate |
|---|---|---|
| simple_python | 333/360 | 92.50% |
| multiple | 268/300 | 89.33% |
| parallel | 270/300 | 90.00% |
| parallel_multiple | 206/240 | 85.83% |
| irrelevance | 111/150 | 74.00% |
| ahm_multi_turn | 95/150 | 63.33% |
simple_python passed 333/360 (92.50%), parallel 270/300 (90.00%), ahm_multi_turn 95/150 (63.33%) and irrelevance 111/150 (74.00%).
Compared with the other four
Difference in percentage points against the official channel; 95% interval shown as whiskers. Crossing 0 means not distinguishable. The deepseek-flash judge is in the same family and uncalibrated, so may be lenient.
F — function support: 71.78
The 60 cases form six groups, each run in three windows: 180 checks judged against the provider's documented format. Inapplicable, invalid and unobserved checks stay out of the denominator. 122 of 177 counted checks passed; group weighting gives 71.78.
| Case group | Weight | Passed | Excluded | Group score |
|---|---|---|---|---|
| Streaming and messaging | 20% | 24/30 | — | 80.00 |
| Tool interface | 20% | 29/30 | — | 96.67 |
| Output constraints | 20% | 21/30 | — | 70.00 |
| Context and state | 15% | 15/30 | — | 50.00 |
| Error and recovery | 15% | 21/30 | — | 70.00 |
| Metering and cache | 10% | 12/27 | 3 | 44.44 |
| Total · weighted | — | 122/177 | 3 | 71.78 |
"Windows failed" counts the three windows in which the case did not pass.
| Failed case | Windows failed | Reason |
|---|---|---|
| Streaming stop | 3/3 | Empty body with stop; no-stop works. |
| Reasoning in body | 3/3 | reasoning_effort=none still returned reasoning. |
| Stop sequence | 3/3 | Empty body with stop; no-stop works. |
| Thinking setting | 3/3 | reasoning_effort=none still returned reasoning. |
| Sampling diversity | 3/3 | Temp 0/1 too similar |
| 10k long input | 3/3 | 10,026 reported; native 10,000. |
| Within context limit | 3/3 | 10,026 reported; native 10,000. |
| 32k long input | 3/3 | 10,026 reported; native 10,000. |
| 100k long input | 3/3 | 10,026 reported; native 10,000. |
| Beyond context | 3/3 | Should be a parameter error; returned 200. |
| Malformed messages | 3/3 | 400, empty body |
| Orphan tool result | 3/3 | Should be a parameter error; returned 200. |
| Malformed stream_options | 3/3 | 400, empty body |
| Thinking token metering | 3/3 | Thinking tokens reported after disable. |
| Native input count | 3/3 | 34 reported; native 8. |
| Tool definitions in input | 3/3 | No-tools control already mismatched. |
| Input count across cache | 3/3 | Repeated prefix count mismatched native. |
| Streaming input agreement | 3/3 | Reported 34; native 8 |
| One tool at a time | 1/3 | Upstream 500. |
Both stop cases returned an empty body while the same input without stop worked, in all three windows. reasoning_effort=none still returned reasoning in both thinking cases.
Excluded checks count as neither pass nor failure.
| Status | Case | Count | Reason |
|---|---|---|---|
| Not applicable | Explicit cache_control | 3 | Pre-registered as unsupported by this interface |
Malformed messages and stream_options failed on all five: 400 with an empty body and no structured error. The shared result more likely sits at the AIHubMix gateway layer, which needs separate investigation. Thinking token metering also failed on all five, but ByteDance and Fireworks reported thinking tokens after disable, while the other three returned no thinking count.
Compared with the other four
P — response speed: 79.51
Each request carried about 10,000 input tokens and asked for about 1,100 output tokens. In each of three windows, 30 requests were sent at single concurrency and 30 at 10 concurrency, for 180 in total.
| Metric | Result |
|---|---|
| TTFA | 1.63 s single; 1.64 s at 10 concurrent |
| TPOT | 3.24 ms single; 3.29 ms at 10 |
| Goodput | 1,504.5 token/s at 10 |
| Valid | 90/90 single; 89/90 at 10 (29/30 in window 3), one single-token output |
| Components | TTFA 54.91, TPOT 100.00, goodput 100.00; weights 35/35/30 |
| P composition | 84.22 × 80% + 60.68 × 20% = 79.51 |
179 of 180 were valid; the invalid request returned one token and counts as infinity in medians, so medians can be higher than for valid requests alone. TPOT comes from stream chunks; goodput is valid output tokens at 10 concurrency divided by batch wall time.
One machine, network and account; windows were scheduled by provider.
K — cache: 60.68
Each group is five requests: seed an 8k or 32k prefix, then the same prefix with a new final question at 5 s, 30 s and 120 s, then a new prefix as a cold control. Three windows × two lengths = 30. K is hot cache-read tokens divided by reusable tokens, lengths weighted equally; K is 20% of P.
| Prefix | Window | Seed | +5 s | +30 s | +120 s | Cold control |
|---|---|---|---|---|---|---|
| 8k | Window 1 | 0 | 7,900 | 0 | 7,900 | 0 |
| 8k | Window 2 | 0 | 7,906 | 0 | 0 | 0 |
| 8k | Window 3 | 0 | 0 | 0 | 0 | 0 |
| 32k | Window 1 | 0 | 31,902 | 31,902 | 31,902 | 0 |
| 32k | Window 2 | 0 | 31,902 | 31,902 | 31,902 | 0 |
| 32k | Window 3 | 0 | 31,900 | 31,900 | 0 | 0 |
The 8k prefix hit 3 of 9 hot requests, reading 23,706 of 72,252 reusable tokens for 32.81. The 32k prefix hit 8 of 9, reading 255,212 of 288,261 for 88.54. K is 60.68 with the two lengths weighted equally. Cold controls read 0. The two prefix lengths therefore behaved differently under the same design. K probes carried the documented user field and x-session-affinity header.
R — service stability: 100
300 requests covered short text, long input, tool calling and multi-turn history at single concurrency across three windows. Only first-response validity and completeness were checked; answers were not graded and nothing was retried. Q and P failures are not recounted here.
| Request type | First-attempt success |
|---|---|
| Short text | 120/120 |
| 10k long input | 90/90 |
| Tool calling | 60/60 |
| Multi-turn history | 30/30 |
| Total · 95% CI 98.78%–100% | 300/300 |
All 300 returned a valid, complete response first time. With 300 successes, the 95% lower bound is 98.78%, so the data only supports a failure rate below 1.22%. R 100 covers these 300 requests, not the whole path.
| Provider | First-attempt success | 95% interval |
|---|---|---|
| ByteDance | 300/300 | 98.78%–100% |
| Baidu | 300/300 | 98.78%–100% |
| Alibaba Cloud | 300/300 | 98.78%–100% |
| DeepSeek official | 300/300 | 98.78%–100% |
| Fireworks | 300/300 | 98.78%–100% |
FAQ
What does Q 41.42 cover?
BFCL, HLE and AA-LCR over 4,240 scored instances: 1,756 correct, 41.42%, 95% CI 39.13%–43.59%; the gap against the official channel is not distinguishable.
Why did both stop cases fail?
A request with stop returned an empty body in all three windows, while the same input without stop worked. Both streaming and non-streaming cases show it.
Does reasoning_effort=none turn thinking off?
No. It still returned reasoning content in all three windows; other requests used native thinking.type=disabled.
Why is a 10k input reported as 10,026 tokens?
The 10k case reported 10,026 against a native 10,000. The 32k and 100k cases reported the same 10,026, and a no-tools control already mismatched.
Why did 8k hit 3/9 while 32k hit 8/9?
Both used the same design and documented affinity settings. The difference is in server-reported reads: 8k hit in windows 1 and 2, 32k in all three.
What does R 100 cover?
300 dedicated requests, all valid on the first attempt; the 95% lower bound is 98.78%. Q and P failures are not counted in R.
Keep reading: the DeepSeek V4.1 Flash series
- ByteDance reported thinking tokens after thinking was disabled, as Fireworks did: DeepSeek V4.1 Flash on ByteDance: tool calls match, metering doesn't
- Baidu recorded Q 38.00 and P 37.77: DeepSeek V4.1 Flash on Baidu: no available channel hits Q and P
- Alibaba Cloud recorded a 1.34 s first token and 99.37 cache: DeepSeek V4.1 Flash on Alibaba Cloud: 1.34s first token, 99.37 cache
- The official channel is the Q baseline, with F 94.00: DeepSeek V4.1 Flash on the official channel: 94.00 F, 15 invalid probes
