ModelVerdict

Mistral · open weights

Mistral Small 4

How Mistral Small 4 by Mistral handles real business documents: accuracy, true cost, speed and where it goes wrong.

List price: $0.15 per 1M input tokens, $0.60 per 1M output tokens · mistralai/mistral-small-2603

Results

Task · documentsRankCorrect fieldsError-free documentsPer 1,000 documentsTo fix / 1,000Speed (median)Bar
Contract clauses · English (US)10/1091.5%43%$0.175671.1 secbelow the bar
Contract clauses · Czech10/1099.0%92%$0.23801.2 secmeets the bar
Contract clauses · German10/1094.0%45%$0.215531.2 secbelow the bar
Email triage · English (US)10/1089.7%37%$0.096270.8 secbelow the bar
Email triage · Czech10/1087.7%32%$0.126850.8 secbelow the bar
Email triage · German10/1087.1%29%$0.117060.8 secbelow the bar
Invoice extraction · English (US)9/1095.6%58%$0.414201.8 secmeets the bar
Invoice extraction · Czech8/1093.6%56%$0.444402.0 secbelow the bar
Invoice extraction · German9/1094.6%50%$0.445002.0 secbelow the bar
Invoice extraction · Spanish9/1091.9%31%$0.446912.2 secbelow the bar
Invoice extraction · French8/1094.2%50%$0.455002.1 secbelow the bar
Personal data detection · English (US)9/1099.4%96%$0.08400.8 secmeets the bar
Personal data detection · Czech8/1098.8%92%$0.09810.9 secmeets the bar
Personal data detection · German10/1099.1%94%$0.09650.9 secmeets the bar

Where it goes wrong

The fields it gets wrong most often and the kinds of errors, from all documents of the benchmark.

Contract clauses · English (US)

Weakest fields

  • Provider43.3%
  • Client46.7%

Kinds of errors (number of fields)

  • wrong value165

Contract clauses · Czech

Weakest fields

  • Provider95.3%
  • Client95.3%
  • Effective date96.7%

Kinds of errors (number of fields)

  • wrong value16
  • swapped dates3

Contract clauses · German

Weakest fields

  • Provider ID62.4%
  • Client ID63.1%
  • Provider98.6%
  • Client98.6%

Kinds of errors (number of fields)

  • invented value99
  • wrong value11

Email triage · English (US)

Weakest fields

  • Priority61.3%
  • Sentiment71.3%
  • Deadline76.0%
  • Amount99.3%

Kinds of errors (number of fields)

  • wrong value103
  • invented value34
  • missing field2

Email triage · Czech

Weakest fields

  • Priority52.4%
  • Deadline69.8%
  • Sentiment72.5%
  • Amount98.0%

Kinds of errors (number of fields)

  • wrong value115
  • invented value44
  • missing field6

Email triage · German

Weakest fields

  • Priority55.2%
  • Sentiment65.0%
  • Deadline70.6%
  • Amount96.5%

Kinds of errors (number of fields)

  • wrong value114
  • invented value46
  • missing field6

Invoice extraction · English (US)

Weakest fields

  • Net by tax rate72.0%
  • Payment reference87.3%
  • Invoice date97.3%
  • Bank account98.0%

Kinds of errors (number of fields)

  • wrong rate39
  • wrong value20
  • invented value13
  • swapped dates1

Invoice extraction · Czech

Weakest fields

  • Tax point76.0%
  • Net by tax rate90.0%
  • Bank account90.0%
  • Supplier91.3%

Kinds of errors (number of fields)

  • wrong value59
  • invented value57
  • lost diacritics4
  • swapped dates4

Invoice extraction · German

Weakest fields

  • Supplier ID (EIN, IČO…)80.0%
  • Payment reference87.1%
  • Tax point87.9%
  • Bank account91.4%

Kinds of errors (number of fields)

  • invented value64
  • wrong value22
  • missing field10
  • swapped dates2

Invoice extraction · Spanish

Weakest fields

  • VAT ID53.6%
  • Tax point69.1%
  • Payment reference88.1%
  • Bank account95.2%

Kinds of errors (number of fields)

  • invented value72
  • wrong value12
  • swapped dates3
  • missing field1

Invoice extraction · French

Weakest fields

  • Bank account75.0%
  • Payment reference85.5%
  • Tax point92.1%
  • Supplier ID (EIN, IČO…)93.4%

Kinds of errors (number of fields)

  • wrong value33
  • invented value23
  • swapped dates1

Personal data detection · English (US)

Weakest fields

  • Names97.3%
  • Addresses98.7%

Kinds of errors (number of fields)

  • invented value3
  • missing field3

Personal data detection · Czech

Weakest fields

  • Names94.0%
  • Addresses98.7%
  • Dates of birth98.7%

Kinds of errors (number of fields)

  • invented value8
  • missing field5

Personal data detection · German

Weakest fields

  • Names95.0%
  • Addresses99.3%
  • Bank accounts99.3%

Kinds of errors (number of fields)

  • invented value7
  • missing field1
  • wrong value1

Compliance

EU and ZDR: Mistral. Mistral hosts data in the EU by default.

How we verify compliance →

Use it now

The exact request of the benchmark (same prompt, JSON schema, temperature 0), ready to call Mistral Small 4 via OpenRouter.

Contract clauses · English (US)

Python · httpx · OpenRouter
import json
import os

import httpx

PROMPT = "Extract the key terms of the contract below and return only JSON with these keys:\n\nprovider_name         the party that delivers (provider, seller, landlord, licensor, contractor),\n                      exactly as written, including the legal form\nprovider_id           that party's company ID (EIN), if stated\nclient_name           the party that pays (client, buyer, tenant, licensee, customer)\nclient_id             that party's company ID (EIN), if stated\ncontract_type         one of: services, supply, lease, license, maintenance\nprice                 the agreed price excluding VAT / sales tax (number)\ncurrency              ISO 4217 code (USD, EUR, CZK, …)\nprice_period          one of: one_time, monthly, yearly\neffective_date        the date the contract takes effect (YYYY-MM-DD)\nend_date              the date a fixed term ends (YYYY-MM-DD); null if the term is indefinite\nnotice_period_months  the termination notice period in months (number); null if there is none\npenalty_pct_per_day   the late-payment penalty in percent of the overdue amount per day\n                      (e.g. 0.05 for 0.05 %); null if there is none\ngoverning_law         ISO 3166 country code of the governing law; US states as US-XX (e.g. US-NY)\n\nRules:\n- Resolve relative dates (\"on the date of signature\", \"the first day of the following month\")\n  against the signing date.\n- The signing date is not the effective date unless the contract says so.\n- Deposits, advance payments and fixed penalties for other breaches are not the price or the\n  late-payment penalty.\n- If a value is not in the contract, return null. Do not guess.\n- Amounts as numbers with a decimal point, no thousands separators and no currency symbol.\n"
SCHEMA = json.loads("{\"type\":\"object\",\"properties\":{\"provider_name\":{\"type\":[\"string\",\"null\"]},\"provider_id\":{\"type\":[\"string\",\"null\"]},\"client_name\":{\"type\":[\"string\",\"null\"]},\"client_id\":{\"type\":[\"string\",\"null\"]},\"contract_type\":{\"type\":[\"string\",\"null\"],\"enum\":[\"services\",\"supply\",\"lease\",\"license\",\"maintenance\",null]},\"price\":{\"type\":[\"number\",\"null\"]},\"currency\":{\"type\":[\"string\",\"null\"]},\"price_period\":{\"type\":[\"string\",\"null\"],\"enum\":[\"one_time\",\"monthly\",\"yearly\",null]},\"effective_date\":{\"type\":[\"string\",\"null\"],\"description\":\"YYYY-MM-DD\"},\"end_date\":{\"type\":[\"string\",\"null\"],\"description\":\"YYYY-MM-DD\"},\"notice_period_months\":{\"type\":[\"integer\",\"null\"]},\"penalty_pct_per_day\":{\"type\":[\"number\",\"null\"]},\"governing_law\":{\"type\":[\"string\",\"null\"]}},\"required\":[\"provider_name\",\"provider_id\",\"client_name\",\"client_id\",\"contract_type\",\"price\",\"currency\",\"price_period\",\"effective_date\",\"end_date\",\"notice_period_months\",\"penalty_pct_per_day\",\"governing_law\"],\"additionalProperties\":false}")


def run(document_text: str) -> dict:
    """Text of the document in, the extracted fields as a dict out."""
    response = httpx.post(
        "https://openrouter.ai/api/v1/chat/completions",
        headers={"Authorization": f"Bearer {os.environ['OPENROUTER_API_KEY']}"},
        json={
            "model": "mistralai/mistral-small-2603",
            "temperature": 0,
            "messages": [{"role": "user", "content": f"{PROMPT}\n\n---\n{document_text}"}],
            # For a scan or photo, send [{"type": "text", "text": PROMPT},
            # {"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}}] instead.
            "response_format": {
                "type": "json_schema",
                "json_schema": {"name": "contracts", "strict": False, "schema": SCHEMA},
            },
        },
        timeout=120,
    )
    response.raise_for_status()
    return json.loads(response.json()["choices"][0]["message"]["content"])

Email triage · English (US)

Python · httpx · OpenRouter
import json
import os

import httpx

PROMPT = "You triage the support inbox. Read the email below and return only JSON with these keys:\n\ncategory        one of: billing (invoices, payments, charges), delivery (where is the order, shipping,\n                address change), complaint (damaged, wrong or defective goods, warranty claims),\n                technical (login, app or system errors, integrations), cancellation (cancel an order,\n                contract or subscription), sales (quotes, prices, interest before buying)\npriority        one of: urgent, high, normal, low (rules below)\nsentiment       one of: positive, neutral, negative (the customer's tone)\ncustomer_name   full name of the customer who wrote the request\norder_number    the customer's order number, if mentioned\ninvoice_number  invoice number, if mentioned\namount          the amount of money the request is about (number)\ncurrency        ISO 4217 code of that amount (USD, EUR, CZK, …)\ndeadline        the date by which the customer wants a resolution or answer (YYYY-MM-DD)\n\nPriority rules:\n- urgent: the customer cannot work right now, or the deadline is at most 1 day after the email date\n- high: the deadline is at most 7 days after the email date, or the customer threatens to cancel\n  or escalate (lawyer, ending the business)\n- low: the customer says explicitly that it is not urgent\n- normal: everything else\n\nRules:\n- A thread: judge the newest message; order numbers, invoice numbers and amounts may come from the\n  quoted earlier messages. A forwarded email: the customer is the original sender, not the\n  colleague who forwarded it.\n- Resolve relative deadlines (\"by Friday\", \"by tomorrow\", \"by the end of the month\") against the\n  date of the message that states them.\n- Support ticket numbers, tax IDs and phone numbers are not order or invoice numbers.\n- If a value is not in the email, return null. Do not guess.\n- Amounts as numbers with a decimal point, no thousands separators and no currency symbol.\n"
SCHEMA = json.loads("{\"type\":\"object\",\"properties\":{\"category\":{\"type\":[\"string\",\"null\"],\"enum\":[\"billing\",\"delivery\",\"complaint\",\"technical\",\"cancellation\",\"sales\",null]},\"priority\":{\"type\":[\"string\",\"null\"],\"enum\":[\"urgent\",\"high\",\"normal\",\"low\",null]},\"sentiment\":{\"type\":[\"string\",\"null\"],\"enum\":[\"positive\",\"neutral\",\"negative\",null]},\"customer_name\":{\"type\":[\"string\",\"null\"]},\"order_number\":{\"type\":[\"string\",\"null\"]},\"invoice_number\":{\"type\":[\"string\",\"null\"]},\"amount\":{\"type\":[\"number\",\"null\"]},\"currency\":{\"type\":[\"string\",\"null\"]},\"deadline\":{\"type\":[\"string\",\"null\"],\"description\":\"YYYY-MM-DD\"}},\"required\":[\"category\",\"priority\",\"sentiment\",\"customer_name\",\"order_number\",\"invoice_number\",\"amount\",\"currency\",\"deadline\"],\"additionalProperties\":false}")


def run(document_text: str) -> dict:
    """Text of the document in, the extracted fields as a dict out."""
    response = httpx.post(
        "https://openrouter.ai/api/v1/chat/completions",
        headers={"Authorization": f"Bearer {os.environ['OPENROUTER_API_KEY']}"},
        json={
            "model": "mistralai/mistral-small-2603",
            "temperature": 0,
            "messages": [{"role": "user", "content": f"{PROMPT}\n\n---\n{document_text}"}],
            # For a scan or photo, send [{"type": "text", "text": PROMPT},
            # {"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}}] instead.
            "response_format": {
                "type": "json_schema",
                "json_schema": {"name": "emails", "strict": False, "schema": SCHEMA},
            },
        },
        timeout=120,
    )
    response.raise_for_status()
    return json.loads(response.json()["choices"][0]["message"]["content"])

Invoice extraction · English (US)

Python · httpx · OpenRouter
import json
import os

import httpx

PROMPT = "Extract the following data from the attached document and return only JSON with these keys:\n\nsupplier_name      supplier's name (exactly as printed, including the legal form)\nsupplier_id        supplier's tax ID (EIN), if printed\ninvoice_number     document number\npayment_reference  payment reference to quote with the payment\nissue_date         invoice date (YYYY-MM-DD)\ndue_date           due date (YYYY-MM-DD)\nnet_by_rate        amount per sales tax rate, e.g. {\"8.25\": 1845.00}; non-taxable amounts under \"0\"\nvat_by_rate        sales tax per rate, e.g. {\"8.25\": 152.21}\ntotal              total due (number, negative for credit notes)\ncurrency           ISO 4217 currency code (USD, GBP, EUR, …)\nbank_account       supplier's bank details as \"routing-number account-number\", or IBAN\n\nRules:\n- Return the supplier's data only, never the customer's.\n- If a value is not on the document, return null. Do not guess.\n- If the document has no tax breakdown, return net_by_rate and vat_by_rate as null.\n- Amounts as numbers with a decimal point, no thousands separators and no currency symbol.\n"
SCHEMA = json.loads("{\"type\":\"object\",\"properties\":{\"supplier_name\":{\"type\":[\"string\",\"null\"]},\"supplier_id\":{\"type\":[\"string\",\"null\"]},\"invoice_number\":{\"type\":[\"string\",\"null\"]},\"payment_reference\":{\"type\":[\"string\",\"null\"]},\"issue_date\":{\"type\":[\"string\",\"null\"],\"description\":\"YYYY-MM-DD\"},\"due_date\":{\"type\":[\"string\",\"null\"],\"description\":\"YYYY-MM-DD\"},\"net_by_rate\":{\"type\":[\"object\",\"null\"],\"additionalProperties\":{\"type\":\"number\"}},\"vat_by_rate\":{\"type\":[\"object\",\"null\"],\"additionalProperties\":{\"type\":\"number\"}},\"total\":{\"type\":[\"number\",\"null\"]},\"currency\":{\"type\":[\"string\",\"null\"]},\"bank_account\":{\"type\":[\"string\",\"null\"]}},\"required\":[\"supplier_name\",\"supplier_id\",\"invoice_number\",\"payment_reference\",\"issue_date\",\"due_date\",\"net_by_rate\",\"vat_by_rate\",\"total\",\"currency\",\"bank_account\"],\"additionalProperties\":false}")


def run(document_text: str) -> dict:
    """Text of the document in, the extracted fields as a dict out."""
    response = httpx.post(
        "https://openrouter.ai/api/v1/chat/completions",
        headers={"Authorization": f"Bearer {os.environ['OPENROUTER_API_KEY']}"},
        json={
            "model": "mistralai/mistral-small-2603",
            "temperature": 0,
            "messages": [{"role": "user", "content": f"{PROMPT}\n\n---\n{document_text}"}],
            # For a scan or photo, send [{"type": "text", "text": PROMPT},
            # {"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}}] instead.
            "response_format": {
                "type": "json_schema",
                "json_schema": {"name": "invoices", "strict": False, "schema": SCHEMA},
            },
        },
        timeout=120,
    )
    response.raise_for_status()
    return json.loads(response.json()["choices"][0]["message"]["content"])

Personal data detection · English (US)

Python · httpx · OpenRouter
import json
import os

import httpx

PROMPT = "Find all personal data of natural persons in the text below (for anonymisation) and return only\nJSON with these keys, each a list of strings (an empty list when there is none):\n\nperson_names       full names of people, each person once, as written in the text\nemail_addresses    email addresses of people\nphone_numbers      phone numbers of people, as written\npostal_addresses   home or delivery addresses of people, as written (street, city, ZIP)\nbirth_dates        dates of birth (YYYY-MM-DD)\nnational_ids       national identification numbers (e.g. SSN, birth numbers)\nbank_accounts      bank account numbers of people, as written\n\nRules:\n- Only data of natural persons. Company names, company registration or tax numbers (EIN, IČO,\n  VAT), generic company addresses such as info@ or support@, order and ticket numbers are not\n  personal data: leave them out.\n- Dates that are not dates of birth (meetings, deliveries, start dates) are not personal data.\n- Do not invent or complete values; copy them from the text.\n"
SCHEMA = json.loads("{\"type\":\"object\",\"properties\":{\"person_names\":{\"type\":[\"array\",\"null\"],\"items\":{\"type\":\"string\"}},\"email_addresses\":{\"type\":[\"array\",\"null\"],\"items\":{\"type\":\"string\"}},\"phone_numbers\":{\"type\":[\"array\",\"null\"],\"items\":{\"type\":\"string\"}},\"postal_addresses\":{\"type\":[\"array\",\"null\"],\"items\":{\"type\":\"string\"}},\"birth_dates\":{\"type\":[\"array\",\"null\"],\"items\":{\"type\":\"string\"}},\"national_ids\":{\"type\":[\"array\",\"null\"],\"items\":{\"type\":\"string\"}},\"bank_accounts\":{\"type\":[\"array\",\"null\"],\"items\":{\"type\":\"string\"}}},\"required\":[\"person_names\",\"email_addresses\",\"phone_numbers\",\"postal_addresses\",\"birth_dates\",\"national_ids\",\"bank_accounts\"],\"additionalProperties\":false}")


def run(document_text: str) -> dict:
    """Text of the document in, the extracted fields as a dict out."""
    response = httpx.post(
        "https://openrouter.ai/api/v1/chat/completions",
        headers={"Authorization": f"Bearer {os.environ['OPENROUTER_API_KEY']}"},
        json={
            "model": "mistralai/mistral-small-2603",
            "temperature": 0,
            "messages": [{"role": "user", "content": f"{PROMPT}\n\n---\n{document_text}"}],
            # For a scan or photo, send [{"type": "text", "text": PROMPT},
            # {"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}}] instead.
            "response_format": {
                "type": "json_schema",
                "json_schema": {"name": "pii", "strict": False, "schema": SCHEMA},
            },
        },
        timeout=120,
    )
    response.raise_for_status()
    return json.loads(response.json()["choices"][0]["message"]["content"])

Badge

Embed the current result on your site; the badge updates with every run.

<a href="https://modelverdict.ai/en/models/mistral-small-4/"><img src="https://modelverdict.ai/badge/mistral-small-4/contracts-en.svg" alt="Mistral Small 4 on ModelVerdict"></a>
<a href="https://modelverdict.ai/en/models/mistral-small-4/"><img src="https://modelverdict.ai/badge/mistral-small-4/contracts-cs.svg" alt="Mistral Small 4 on ModelVerdict"></a>
<a href="https://modelverdict.ai/en/models/mistral-small-4/"><img src="https://modelverdict.ai/badge/mistral-small-4/contracts-de.svg" alt="Mistral Small 4 on ModelVerdict"></a>
<a href="https://modelverdict.ai/en/models/mistral-small-4/"><img src="https://modelverdict.ai/badge/mistral-small-4/emails-en.svg" alt="Mistral Small 4 on ModelVerdict"></a>
<a href="https://modelverdict.ai/en/models/mistral-small-4/"><img src="https://modelverdict.ai/badge/mistral-small-4/emails-cs.svg" alt="Mistral Small 4 on ModelVerdict"></a>
<a href="https://modelverdict.ai/en/models/mistral-small-4/"><img src="https://modelverdict.ai/badge/mistral-small-4/emails-de.svg" alt="Mistral Small 4 on ModelVerdict"></a>
<a href="https://modelverdict.ai/en/models/mistral-small-4/"><img src="https://modelverdict.ai/badge/mistral-small-4/invoices-en.svg" alt="Mistral Small 4 on ModelVerdict"></a>
<a href="https://modelverdict.ai/en/models/mistral-small-4/"><img src="https://modelverdict.ai/badge/mistral-small-4/invoices-cs.svg" alt="Mistral Small 4 on ModelVerdict"></a>
<a href="https://modelverdict.ai/en/models/mistral-small-4/"><img src="https://modelverdict.ai/badge/mistral-small-4/invoices-de.svg" alt="Mistral Small 4 on ModelVerdict"></a>
<a href="https://modelverdict.ai/en/models/mistral-small-4/"><img src="https://modelverdict.ai/badge/mistral-small-4/invoices-es.svg" alt="Mistral Small 4 on ModelVerdict"></a>
<a href="https://modelverdict.ai/en/models/mistral-small-4/"><img src="https://modelverdict.ai/badge/mistral-small-4/invoices-fr.svg" alt="Mistral Small 4 on ModelVerdict"></a>
<a href="https://modelverdict.ai/en/models/mistral-small-4/"><img src="https://modelverdict.ai/badge/mistral-small-4/pii-en.svg" alt="Mistral Small 4 on ModelVerdict"></a>
<a href="https://modelverdict.ai/en/models/mistral-small-4/"><img src="https://modelverdict.ai/badge/mistral-small-4/pii-cs.svg" alt="Mistral Small 4 on ModelVerdict"></a>
<a href="https://modelverdict.ai/en/models/mistral-small-4/"><img src="https://modelverdict.ai/badge/mistral-small-4/pii-de.svg" alt="Mistral Small 4 on ModelVerdict"></a>

Compare with another model