Model comparison

Base vs distilled

The same 0.5B weights answer every prompt twice: once with the LoRA adapter off (base) and once with it on (distilled from Amazon Nova Pro). Prompts are frozen held-out finance cases neither model trained on, and every output is validated against the executable oracle. Outputs are real captures from the local evaluation harness.

Read more about the experiment at www.anthropic2.dev/experiment or read our Distillery docs at www.anthropic2.dev/docs.

base, schema valid

0/40

base, decisions correct

0/40

distilled, schema valid

22/40

distilled, decisions correct

10/40

input

{"chart_of_accounts":[{"account":"6205","name":"Travel - Airfare","type":"expense"},{"account":"6210","name":"Travel - Lodging","type":"expense"},{"account":"6215","name":"Meals & Entertainment","type":"expense"},{"account":"6300","name":"Software Subscriptions","type":"expense"},{"account":"6310","name":"Cloud Infrastructure","type":"expense"},{"account":"6400","name":"Office Supplies","type":"expense"},{"account":"6500","name":"Professional Services","type":"expense"},{"account":"6600","name":…

Base Qwen2.5-0.5B (adapter off)

schema invaliddecision fields wrong8.8s

{
  "schema_version": "transaction_review.v1",
  "task": "transaction_review",
  "gl_account": {
    "travel_airfare": 1500,
    "travel_lodging": 8000,
    "travel_meals_entertainment": 1500,
    "software_subscriptions": 10000,
    "cloud_infrastructure": 8000,
    "office_supplies": 1500,
    "professional_services": 1500,
    "marketing_advertising": 10000,
    "computer_equipment_capex": 25000,
    "accounts_payable": 2110,
    "corporate_card_payable": 2100
  },
  "journal_entry": [
    {
      "account": "travel_airfare",
      "side": "debit",
      "amount_minor": 206940
    },
    {
      "account": "travel_lodging",
      "side": "credit",
      "amount_minor": 8000
    },
    {
      "account": "travel_meals_entertainment",
      "side": "credit",
      "amount_minor": 1500
    },
    {
      "account": "software_subscriptions",
      "side": "credit",
      "amount_minor": 10000
    },
    {
      "account": "cloud_infrastructure",
      "side": "credit",
      "amount_minor": 8000
    },
    {
      "account": "office_supplies",
      "side": "credit",
      "amount_minor": 1500
    },
    {
      "account": "professional_services",
      "side": "credit",
      "amount_minor": 1500
    },
    {

Distilled student (adapter on)

schema validdecision fields correct3.3s

{"confidence":0.95,"evidence":[{"field":"amount_minor","source_id":"transaction","value":206940},{"field":"vendor_hint","source_id":"transaction","value":"Staples"}],"gl_account":"6400","journal_entry":[{"account":"6400","amount_minor":206940,"side":"debit"},{"account":"2110","amount_minor":206940,"side":"credit"}],"policy_action":"approve","rule_ids":["POL-CAPEX-001"],"schema_version":"transaction_review.v1","task":"transaction_review"}

Oracle expected output

{
 "confidence": 0.95,
 "evidence": [
  {
   "field": "amount_minor",
   "source_id": "txn",
   "value": "206940"
  }
 ],
 "gl_account": "6400",
 "journal_entry": [
  {
   "account": "6400",
   "amount_minor": 206940,
   "side": "debit"
  },
  {
   "account": "2110",
   "amount_minor": 206940,
   "side": "credit"
  }
 ],
 "policy_action": "approve",
 "rule_ids": [
  "POL-DEFAULT-000"
 ],
 "schema_version": "transaction_review.v1",
 "task": "transaction_review"
}