ModelVerdict

Qwen

Qwen3.8 Flash

How Qwen3.8 Flash by Qwen handles real business documents: accuracy, true cost, speed and where it goes wrong.

List price: $0.15 per 1M input tokens, $0.47 per 1M output tokens · qwen/qwen3.8-flash

Results

Task · documentsRankCorrect fieldsError-free documentsPer 1,000 documentsTo fix / 1,000Speed (median)Bar
Contract clauses · English (US)9/1095.1%69%$0.6230714.1 secmeets the bar
Contract clauses · Czech8/1099.9%99%$0.631313.6 secmeets the bar
Contract clauses · German6/1098.0%87%$0.7812719.7 secmeets the bar
Email triage · English (US)6/1097.5%79%$0.302138.9 secmeets the bar
Email triage · Czech3/1098.5%87%$0.4513313.0 secmeets the bar
Email triage · German4/1097.9%81%$0.3918712.1 secmeets the bar
Invoice extraction · English (US)10/1082.4%57%$1.1542717.4 secbelow the bar
Invoice extraction · Czech10/1091.8%77%$1.0223317.0 secbelow the bar
Invoice extraction · German10/1093.5%87%$1.1212719.0 secbelow the bar
Invoice extraction · Spanish10/1087.9%74%$1.4026022.3 secbelow the bar
Invoice extraction · French10/1090.6%82%$1.2318019.8 secbelow the bar
Personal data detection · English (US)10/1099.2%99%$0.28137.5 secmeets the bar
Personal data detection · Czech10/1097.0%89%$0.411139.6 secmeets the bar
Personal data detection · German9/1099.6%99%$0.36712.2 secmeets the bar

Where it goes wrong

The fields it gets wrong most often and the kinds of errors, from all documents of the benchmark.

Contract clauses · English (US)

Weakest fields

  • Provider69.3%
  • Client69.3%
  • Effective date99.3%
  • Late-payment penalty (% per day)99.3%

Kinds of errors (number of fields)

  • wrong value92
  • missing field3

Contract clauses · Czech

Weakest fields

  • Contract type98.7%

Kinds of errors (number of fields)

  • wrong value2

Contract clauses · German

Weakest fields

  • Governing law91.3%
  • Provider ID94.7%
  • Client ID94.7%
  • Provider99.3%

Kinds of errors (number of fields)

  • invented value14
  • invalid JSON13
  • missing field12

Email triage · English (US)

Weakest fields

  • Sentiment80.7%
  • Priority97.3%
  • Deadline99.3%

Kinds of errors (number of fields)

  • wrong value33
  • invented value1

Email triage · Czech

Weakest fields

  • Sentiment88.7%
  • Priority99.3%
  • Customer name99.3%
  • Order number99.3%

Kinds of errors (number of fields)

  • wrong value20

Email triage · German

Weakest fields

  • Sentiment84.7%
  • Priority96.0%

Kinds of errors (number of fields)

  • wrong value29

Invoice extraction · English (US)

Weakest fields

  • Net by tax rate70.0%
  • Tax by rate76.0%
  • Payment reference79.3%
  • Bank account81.3%

Kinds of errors (number of fields)

  • invalid JSON154
  • missing field83
  • wrong rate23
  • wrong value18
  • invented value13

Invoice extraction · Czech

Weakest fields

  • Net by tax rate80.0%
  • Tax by rate80.7%
  • Bank account90.7%
  • Supplier93.3%

Kinds of errors (number of fields)

  • invalid JSON65
  • missing field41
  • wrong value36
  • wrong rate16
  • invented value1
  • net instead of total1

Invoice extraction · German

Weakest fields

  • Payment reference88.7%
  • Invoice number92.7%
  • Supplier94.0%
  • Supplier ID (EIN, IČO…)94.0%

Kinds of errors (number of fields)

  • invalid JSON117
  • invented value8
  • wrong value2

Invoice extraction · Spanish

Weakest fields

  • Payment reference85.0%
  • Supplier86.0%
  • VAT ID86.0%
  • Net by tax rate86.0%

Kinds of errors (number of fields)

  • invalid JSON130
  • missing field14
  • invented value10
  • wrong value2
  • lost diacritics1

Invoice extraction · French

Weakest fields

  • Payment reference86.0%
  • Due date89.0%
  • Invoice number90.0%
  • Bank account90.0%

Kinds of errors (number of fields)

  • invalid JSON65
  • missing field49
  • invented value5
  • wrong value3

Personal data detection · English (US)

Weakest fields

  • Addresses98.7%
  • Names99.3%
  • Email addresses99.3%
  • Phone numbers99.3%

Kinds of errors (number of fields)

  • invented value5
  • wrong value2
  • missing field1

Personal data detection · Czech

Weakest fields

  • Names90.7%
  • Addresses96.7%
  • Phone numbers97.3%
  • Email addresses98.0%

Kinds of errors (number of fields)

  • wrong value13
  • invalid JSON7
  • invented value6
  • missing field6

Personal data detection · German

Weakest fields

  • Names99.3%
  • Email addresses99.3%
  • Phone numbers99.3%
  • Addresses99.3%

Kinds of errors (number of fields)

  • missing field4

Language tax

How many more tokens this model needs for the same text than in English.

Language tax of all models →

Compliance

Only provider on OpenRouter: Alibaba (no EU or ZDR endpoint). DPA: general Alibaba Cloud International DPA.

How we verify compliance →

Use it now

The exact request of the benchmark (same prompt, JSON schema, temperature 0), ready to call Qwen3.8 Flash via OpenRouter.

Contract clauses · English (US)

Python · httpx · OpenRouter
import json
import os

import httpx

PROMPT = "Extract the key terms of the contract below and return only JSON with these keys:\n\nprovider_name         the party that delivers (provider, seller, landlord, licensor, contractor),\n                      exactly as written, including the legal form\nprovider_id           that party's company ID (EIN), if stated\nclient_name           the party that pays (client, buyer, tenant, licensee, customer)\nclient_id             that party's company ID (EIN), if stated\ncontract_type         one of: services, supply, lease, license, maintenance\nprice                 the agreed price excluding VAT / sales tax (number)\ncurrency              ISO 4217 code (USD, EUR, CZK, …)\nprice_period          one of: one_time, monthly, yearly\neffective_date        the date the contract takes effect (YYYY-MM-DD)\nend_date              the date a fixed term ends (YYYY-MM-DD); null if the term is indefinite\nnotice_period_months  the termination notice period in months (number); null if there is none\npenalty_pct_per_day   the late-payment penalty in percent of the overdue amount per day\n                      (e.g. 0.05 for 0.05 %); null if there is none\ngoverning_law         ISO 3166 country code of the governing law; US states as US-XX (e.g. US-NY)\n\nRules:\n- Resolve relative dates (\"on the date of signature\", \"the first day of the following month\")\n  against the signing date.\n- The signing date is not the effective date unless the contract says so.\n- Deposits, advance payments and fixed penalties for other breaches are not the price or the\n  late-payment penalty.\n- If a value is not in the contract, return null. Do not guess.\n- Amounts as numbers with a decimal point, no thousands separators and no currency symbol.\n"
SCHEMA = json.loads("{\"type\":\"object\",\"properties\":{\"provider_name\":{\"type\":[\"string\",\"null\"]},\"provider_id\":{\"type\":[\"string\",\"null\"]},\"client_name\":{\"type\":[\"string\",\"null\"]},\"client_id\":{\"type\":[\"string\",\"null\"]},\"contract_type\":{\"type\":[\"string\",\"null\"],\"enum\":[\"services\",\"supply\",\"lease\",\"license\",\"maintenance\",null]},\"price\":{\"type\":[\"number\",\"null\"]},\"currency\":{\"type\":[\"string\",\"null\"]},\"price_period\":{\"type\":[\"string\",\"null\"],\"enum\":[\"one_time\",\"monthly\",\"yearly\",null]},\"effective_date\":{\"type\":[\"string\",\"null\"],\"description\":\"YYYY-MM-DD\"},\"end_date\":{\"type\":[\"string\",\"null\"],\"description\":\"YYYY-MM-DD\"},\"notice_period_months\":{\"type\":[\"integer\",\"null\"]},\"penalty_pct_per_day\":{\"type\":[\"number\",\"null\"]},\"governing_law\":{\"type\":[\"string\",\"null\"]}},\"required\":[\"provider_name\",\"provider_id\",\"client_name\",\"client_id\",\"contract_type\",\"price\",\"currency\",\"price_period\",\"effective_date\",\"end_date\",\"notice_period_months\",\"penalty_pct_per_day\",\"governing_law\"],\"additionalProperties\":false}")


def run(document_text: str) -> dict:
    """Text of the document in, the extracted fields as a dict out."""
    response = httpx.post(
        "https://openrouter.ai/api/v1/chat/completions",
        headers={"Authorization": f"Bearer {os.environ['OPENROUTER_API_KEY']}"},
        json={
            "model": "qwen/qwen3.8-flash",
            "temperature": 0,
            "messages": [{"role": "user", "content": f"{PROMPT}\n\n---\n{document_text}"}],
            # For a scan or photo, send [{"type": "text", "text": PROMPT},
            # {"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}}] instead.
            "response_format": {
                "type": "json_schema",
                "json_schema": {"name": "contracts", "strict": False, "schema": SCHEMA},
            },
        },
        timeout=120,
    )
    response.raise_for_status()
    return json.loads(response.json()["choices"][0]["message"]["content"])

Email triage · English (US)

Python · httpx · OpenRouter
import json
import os

import httpx

PROMPT = "You triage the support inbox. Read the email below and return only JSON with these keys:\n\ncategory        one of: billing (invoices, payments, charges), delivery (where is the order, shipping,\n                address change), complaint (damaged, wrong or defective goods, warranty claims),\n                technical (login, app or system errors, integrations), cancellation (cancel an order,\n                contract or subscription), sales (quotes, prices, interest before buying)\npriority        one of: urgent, high, normal, low (rules below)\nsentiment       one of: positive, neutral, negative (the customer's tone)\ncustomer_name   full name of the customer who wrote the request\norder_number    the customer's order number, if mentioned\ninvoice_number  invoice number, if mentioned\namount          the amount of money the request is about (number)\ncurrency        ISO 4217 code of that amount (USD, EUR, CZK, …)\ndeadline        the date by which the customer wants a resolution or answer (YYYY-MM-DD)\n\nPriority rules:\n- urgent: the customer cannot work right now, or the deadline is at most 1 day after the email date\n- high: the deadline is at most 7 days after the email date, or the customer threatens to cancel\n  or escalate (lawyer, ending the business)\n- low: the customer says explicitly that it is not urgent\n- normal: everything else\n\nRules:\n- A thread: judge the newest message; order numbers, invoice numbers and amounts may come from the\n  quoted earlier messages. A forwarded email: the customer is the original sender, not the\n  colleague who forwarded it.\n- Resolve relative deadlines (\"by Friday\", \"by tomorrow\", \"by the end of the month\") against the\n  date of the message that states them.\n- Support ticket numbers, tax IDs and phone numbers are not order or invoice numbers.\n- If a value is not in the email, return null. Do not guess.\n- Amounts as numbers with a decimal point, no thousands separators and no currency symbol.\n"
SCHEMA = json.loads("{\"type\":\"object\",\"properties\":{\"category\":{\"type\":[\"string\",\"null\"],\"enum\":[\"billing\",\"delivery\",\"complaint\",\"technical\",\"cancellation\",\"sales\",null]},\"priority\":{\"type\":[\"string\",\"null\"],\"enum\":[\"urgent\",\"high\",\"normal\",\"low\",null]},\"sentiment\":{\"type\":[\"string\",\"null\"],\"enum\":[\"positive\",\"neutral\",\"negative\",null]},\"customer_name\":{\"type\":[\"string\",\"null\"]},\"order_number\":{\"type\":[\"string\",\"null\"]},\"invoice_number\":{\"type\":[\"string\",\"null\"]},\"amount\":{\"type\":[\"number\",\"null\"]},\"currency\":{\"type\":[\"string\",\"null\"]},\"deadline\":{\"type\":[\"string\",\"null\"],\"description\":\"YYYY-MM-DD\"}},\"required\":[\"category\",\"priority\",\"sentiment\",\"customer_name\",\"order_number\",\"invoice_number\",\"amount\",\"currency\",\"deadline\"],\"additionalProperties\":false}")


def run(document_text: str) -> dict:
    """Text of the document in, the extracted fields as a dict out."""
    response = httpx.post(
        "https://openrouter.ai/api/v1/chat/completions",
        headers={"Authorization": f"Bearer {os.environ['OPENROUTER_API_KEY']}"},
        json={
            "model": "qwen/qwen3.8-flash",
            "temperature": 0,
            "messages": [{"role": "user", "content": f"{PROMPT}\n\n---\n{document_text}"}],
            # For a scan or photo, send [{"type": "text", "text": PROMPT},
            # {"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}}] instead.
            "response_format": {
                "type": "json_schema",
                "json_schema": {"name": "emails", "strict": False, "schema": SCHEMA},
            },
        },
        timeout=120,
    )
    response.raise_for_status()
    return json.loads(response.json()["choices"][0]["message"]["content"])

Invoice extraction · English (US)

Python · httpx · OpenRouter
import json
import os

import httpx

PROMPT = "Extract the following data from the attached document and return only JSON with these keys:\n\nsupplier_name      supplier's name (exactly as printed, including the legal form)\nsupplier_id        supplier's tax ID (EIN), if printed\ninvoice_number     document number\npayment_reference  payment reference to quote with the payment\nissue_date         invoice date (YYYY-MM-DD)\ndue_date           due date (YYYY-MM-DD)\nnet_by_rate        amount per sales tax rate, e.g. {\"8.25\": 1845.00}; non-taxable amounts under \"0\"\nvat_by_rate        sales tax per rate, e.g. {\"8.25\": 152.21}\ntotal              total due (number, negative for credit notes)\ncurrency           ISO 4217 currency code (USD, GBP, EUR, …)\nbank_account       supplier's bank details as \"routing-number account-number\", or IBAN\n\nRules:\n- Return the supplier's data only, never the customer's.\n- If a value is not on the document, return null. Do not guess.\n- If the document has no tax breakdown, return net_by_rate and vat_by_rate as null.\n- Amounts as numbers with a decimal point, no thousands separators and no currency symbol.\n"
SCHEMA = json.loads("{\"type\":\"object\",\"properties\":{\"supplier_name\":{\"type\":[\"string\",\"null\"]},\"supplier_id\":{\"type\":[\"string\",\"null\"]},\"invoice_number\":{\"type\":[\"string\",\"null\"]},\"payment_reference\":{\"type\":[\"string\",\"null\"]},\"issue_date\":{\"type\":[\"string\",\"null\"],\"description\":\"YYYY-MM-DD\"},\"due_date\":{\"type\":[\"string\",\"null\"],\"description\":\"YYYY-MM-DD\"},\"net_by_rate\":{\"type\":[\"object\",\"null\"],\"additionalProperties\":{\"type\":\"number\"}},\"vat_by_rate\":{\"type\":[\"object\",\"null\"],\"additionalProperties\":{\"type\":\"number\"}},\"total\":{\"type\":[\"number\",\"null\"]},\"currency\":{\"type\":[\"string\",\"null\"]},\"bank_account\":{\"type\":[\"string\",\"null\"]}},\"required\":[\"supplier_name\",\"supplier_id\",\"invoice_number\",\"payment_reference\",\"issue_date\",\"due_date\",\"net_by_rate\",\"vat_by_rate\",\"total\",\"currency\",\"bank_account\"],\"additionalProperties\":false}")


def run(document_text: str) -> dict:
    """Text of the document in, the extracted fields as a dict out."""
    response = httpx.post(
        "https://openrouter.ai/api/v1/chat/completions",
        headers={"Authorization": f"Bearer {os.environ['OPENROUTER_API_KEY']}"},
        json={
            "model": "qwen/qwen3.8-flash",
            "temperature": 0,
            "messages": [{"role": "user", "content": f"{PROMPT}\n\n---\n{document_text}"}],
            # For a scan or photo, send [{"type": "text", "text": PROMPT},
            # {"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}}] instead.
            "response_format": {
                "type": "json_schema",
                "json_schema": {"name": "invoices", "strict": False, "schema": SCHEMA},
            },
        },
        timeout=120,
    )
    response.raise_for_status()
    return json.loads(response.json()["choices"][0]["message"]["content"])

Personal data detection · English (US)

Python · httpx · OpenRouter
import json
import os

import httpx

PROMPT = "Find all personal data of natural persons in the text below (for anonymisation) and return only\nJSON with these keys, each a list of strings (an empty list when there is none):\n\nperson_names       full names of people, each person once, as written in the text\nemail_addresses    email addresses of people\nphone_numbers      phone numbers of people, as written\npostal_addresses   home or delivery addresses of people, as written (street, city, ZIP)\nbirth_dates        dates of birth (YYYY-MM-DD)\nnational_ids       national identification numbers (e.g. SSN, birth numbers)\nbank_accounts      bank account numbers of people, as written\n\nRules:\n- Only data of natural persons. Company names, company registration or tax numbers (EIN, IČO,\n  VAT), generic company addresses such as info@ or support@, order and ticket numbers are not\n  personal data: leave them out.\n- Dates that are not dates of birth (meetings, deliveries, start dates) are not personal data.\n- Do not invent or complete values; copy them from the text.\n"
SCHEMA = json.loads("{\"type\":\"object\",\"properties\":{\"person_names\":{\"type\":[\"array\",\"null\"],\"items\":{\"type\":\"string\"}},\"email_addresses\":{\"type\":[\"array\",\"null\"],\"items\":{\"type\":\"string\"}},\"phone_numbers\":{\"type\":[\"array\",\"null\"],\"items\":{\"type\":\"string\"}},\"postal_addresses\":{\"type\":[\"array\",\"null\"],\"items\":{\"type\":\"string\"}},\"birth_dates\":{\"type\":[\"array\",\"null\"],\"items\":{\"type\":\"string\"}},\"national_ids\":{\"type\":[\"array\",\"null\"],\"items\":{\"type\":\"string\"}},\"bank_accounts\":{\"type\":[\"array\",\"null\"],\"items\":{\"type\":\"string\"}}},\"required\":[\"person_names\",\"email_addresses\",\"phone_numbers\",\"postal_addresses\",\"birth_dates\",\"national_ids\",\"bank_accounts\"],\"additionalProperties\":false}")


def run(document_text: str) -> dict:
    """Text of the document in, the extracted fields as a dict out."""
    response = httpx.post(
        "https://openrouter.ai/api/v1/chat/completions",
        headers={"Authorization": f"Bearer {os.environ['OPENROUTER_API_KEY']}"},
        json={
            "model": "qwen/qwen3.8-flash",
            "temperature": 0,
            "messages": [{"role": "user", "content": f"{PROMPT}\n\n---\n{document_text}"}],
            # For a scan or photo, send [{"type": "text", "text": PROMPT},
            # {"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}}] instead.
            "response_format": {
                "type": "json_schema",
                "json_schema": {"name": "pii", "strict": False, "schema": SCHEMA},
            },
        },
        timeout=120,
    )
    response.raise_for_status()
    return json.loads(response.json()["choices"][0]["message"]["content"])

Badge

Embed the current result on your site; the badge updates with every run.

<a href="https://modelverdict.ai/en/models/qwen3-8-flash/"><img src="https://modelverdict.ai/badge/qwen3-8-flash/contracts-en.svg" alt="Qwen3.8 Flash on ModelVerdict"></a>
<a href="https://modelverdict.ai/en/models/qwen3-8-flash/"><img src="https://modelverdict.ai/badge/qwen3-8-flash/contracts-cs.svg" alt="Qwen3.8 Flash on ModelVerdict"></a>
<a href="https://modelverdict.ai/en/models/qwen3-8-flash/"><img src="https://modelverdict.ai/badge/qwen3-8-flash/contracts-de.svg" alt="Qwen3.8 Flash on ModelVerdict"></a>
<a href="https://modelverdict.ai/en/models/qwen3-8-flash/"><img src="https://modelverdict.ai/badge/qwen3-8-flash/emails-en.svg" alt="Qwen3.8 Flash on ModelVerdict"></a>
<a href="https://modelverdict.ai/en/models/qwen3-8-flash/"><img src="https://modelverdict.ai/badge/qwen3-8-flash/emails-cs.svg" alt="Qwen3.8 Flash on ModelVerdict"></a>
<a href="https://modelverdict.ai/en/models/qwen3-8-flash/"><img src="https://modelverdict.ai/badge/qwen3-8-flash/emails-de.svg" alt="Qwen3.8 Flash on ModelVerdict"></a>
<a href="https://modelverdict.ai/en/models/qwen3-8-flash/"><img src="https://modelverdict.ai/badge/qwen3-8-flash/invoices-en.svg" alt="Qwen3.8 Flash on ModelVerdict"></a>
<a href="https://modelverdict.ai/en/models/qwen3-8-flash/"><img src="https://modelverdict.ai/badge/qwen3-8-flash/invoices-cs.svg" alt="Qwen3.8 Flash on ModelVerdict"></a>
<a href="https://modelverdict.ai/en/models/qwen3-8-flash/"><img src="https://modelverdict.ai/badge/qwen3-8-flash/invoices-de.svg" alt="Qwen3.8 Flash on ModelVerdict"></a>
<a href="https://modelverdict.ai/en/models/qwen3-8-flash/"><img src="https://modelverdict.ai/badge/qwen3-8-flash/invoices-es.svg" alt="Qwen3.8 Flash on ModelVerdict"></a>
<a href="https://modelverdict.ai/en/models/qwen3-8-flash/"><img src="https://modelverdict.ai/badge/qwen3-8-flash/invoices-fr.svg" alt="Qwen3.8 Flash on ModelVerdict"></a>
<a href="https://modelverdict.ai/en/models/qwen3-8-flash/"><img src="https://modelverdict.ai/badge/qwen3-8-flash/pii-en.svg" alt="Qwen3.8 Flash on ModelVerdict"></a>
<a href="https://modelverdict.ai/en/models/qwen3-8-flash/"><img src="https://modelverdict.ai/badge/qwen3-8-flash/pii-cs.svg" alt="Qwen3.8 Flash on ModelVerdict"></a>
<a href="https://modelverdict.ai/en/models/qwen3-8-flash/"><img src="https://modelverdict.ai/badge/qwen3-8-flash/pii-de.svg" alt="Qwen3.8 Flash on ModelVerdict"></a>

Compare with another model