{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "# Demo 3 — LLM bidders in a second-price auction\n\nIn a sealed-bid second-price (Vickrey) auction, the highest bidder wins but pays the **second-highest** bid. With private, independent values and quasilinear utility, bidding one's true value is a weakly dominant strategy: underbidding can lose a profitable win, while overbidding can create an unprofitable win.\n\nThis Week 4 notebook tests whether LLM bidders actually follow that prescription. Core calls are pinned to `protected.Claude Sonnet 4.5` through `https://chat.tamu.ai/api`, using `temperature=1` and `max_tokens=16384`. Credentials come from `TAMU_API_KEY` and `CF_COOKIE` (`CF_Authorization=eyJ...`) or hidden prompts and are never stored.\n\nThe experiment is pedagogical: it probes an ordinary scalar-value auction before the course moves to richer content mechanisms.\n"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "## Why ask an empirical question if truth-telling is dominant?\n\nThe theorem describes an ideal optimizer with the stated utility. An LLM is a stochastic policy produced by a prompt. It may misunderstand the payment rule, overgeneralize from first-price auctions, round its bid, or emit an unparsable answer.\n\nWe will distinguish three observations:\n\n- **truthful:** `bid == value`\n- **overbid:** `bid > value`\n- **underbid:** `bid < value`\n\nA deviation may be payoff-relevant or payoff-irrelevant in the realized round. That distinction matters when interpreting whether an LLM's mistake breaks the mechanism's intended guarantee.\n"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": "import getpass\nimport os\nimport re\nfrom openai import OpenAI\n\nTAMU_BASE_URL = \"https://chat.tamu.ai/api\"\nSONNET_MODEL = \"protected.Claude Sonnet 4.5\"\nN_BIDDERS = 3\nN_ROUNDS = 20\nSEED = 42\n\nBIDDER_PROFILES = [\n    \"Theory-aware: explicitly use the second-price dominant-strategy argument.\",\n    \"Concise: focus only on the auction rule and your private value.\",\n    \"Cautious: check whether shading or inflating the bid could improve utility.\",\n]\n\ndef normalize_cf_cookie(raw):\n    raw = raw.strip().strip('\"').strip(\"'\")\n    match = re.search(r\"CF_Authorization=([^;\\s]+)\", raw)\n    return f\"CF_Authorization={match.group(1)}\" if match else (\n        f\"CF_Authorization={raw}\" if raw.startswith(\"eyJ\") else raw\n    )\n\ndef make_client():\n    key = os.environ.get(\"TAMU_API_KEY\") or getpass.getpass(\"TAMU_API_KEY: \")\n    raw = os.environ.get(\"CF_COOKIE\") or getpass.getpass(\"CF_COOKIE: \")\n    return OpenAI(api_key=key, base_url=TAMU_BASE_URL,\n                  default_headers={\"Cookie\": normalize_cf_cookie(raw)})\n\nclient = make_client()\nprint(f\"Endpoint: {TAMU_BASE_URL}\")\nprint(f\"Pinned model: {SONNET_MODEL}\")\nprint(f\"Configured {N_BIDDERS} bidder prompts for {N_ROUNDS} rounds (seed={SEED}).\")\nfor index, profile in enumerate(BIDDER_PROFILES, start=1):\n    print(f\"Bidder {index}: {profile}\")\n"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": "# Self-contained one-round auction with full visible bidder transcripts.\nimport getpass\nimport os\nimport re\nfrom openai import OpenAI\n\nTAMU_BASE_URL = \"https://chat.tamu.ai/api\"\nSONNET_MODEL = \"protected.Claude Sonnet 4.5\"\nBIDDER_PROFILES = [\n    \"Theory-aware: explicitly use the second-price dominant-strategy argument.\",\n    \"Concise: focus only on the auction rule and your private value.\",\n    \"Cautious: check whether shading or inflating the bid could improve utility.\",\n]\n\ndef _cookie(raw):\n    raw = raw.strip().strip('\"').strip(\"'\")\n    match = re.search(r\"CF_Authorization=([^;\\s]+)\", raw)\n    return f\"CF_Authorization={match.group(1)}\" if match else (\n        f\"CF_Authorization={raw}\" if raw.startswith(\"eyJ\") else raw\n    )\n\nif \"client\" not in globals():\n    key = os.environ.get(\"TAMU_API_KEY\") or getpass.getpass(\"TAMU_API_KEY: \")\n    raw = os.environ.get(\"CF_COOKIE\") or getpass.getpass(\"CF_COOKIE: \")\n    client = OpenAI(api_key=key, base_url=TAMU_BASE_URL,\n                    default_headers={\"Cookie\": _cookie(raw)})\n\ndef request_bid(value, profile):\n    prompt = f\"\"\"\nYou are in a sealed-bid second-price auction for one item.\nYour private value is ${value}. The highest bid wins and pays the second-highest bid.\nYour utility is value minus price if you win, and zero if you lose.\nBid an integer from 0 to 100. {profile}\nReturn exactly two lines:\nRATIONALE: one short visible sentence\nBID: integer\nDo not provide private chain-of-thought.\n\"\"\".strip()\n    response = client.chat.completions.create(\n        model=SONNET_MODEL,\n        messages=[{\"role\": \"user\", \"content\": prompt}],\n        temperature=1,\n        max_tokens=16_384,\n    )\n    visible = re.sub(r\"<think>.*?</think>\", \"\", response.choices[0].message.content,\n                     flags=re.DOTALL).strip()\n    matches = re.findall(r\"(?im)^\\s*BID\\s*:\\s*\\$?(-?\\d+)\\s*$\", visible)\n    if not matches:\n        raise ValueError(f\"Could not parse BID from {visible!r}\")\n    bid = max(0, min(100, int(matches[-1])))\n    return bid, visible, response.usage\n\nvalues = [73, 41, 88]  # fixed values make the live walkthrough easy to compare with notes\nround_rows = []\nfor bidder_id, (value, profile) in enumerate(zip(values, BIDDER_PROFILES), start=1):\n    bid, transcript, usage = request_bid(value, profile)\n    round_rows.append({\"bidder\": bidder_id, \"value\": value, \"bid\": bid})\n    print(f\"\\n[Bidder {bidder_id} visible transcript]\\n{transcript}\")\n\nranked = sorted(round_rows, key=lambda row: (-row[\"bid\"], row[\"bidder\"]))\nwinner = ranked[0]\nprice = ranked[1][\"bid\"]\nsurplus = winner[\"value\"] - price\nprint(\"\\n=== Round result ===\")\nprint(f\"Bids: {[(row['bidder'], row['bid']) for row in round_rows]}\")\nprint(f\"Winner: Bidder {winner['bidder']} (value={winner['value']}, bid={winner['bid']})\")\nprint(f\"Second-highest price: {price}; winner surplus: {surplus}\")\nfor row in round_rows:\n    label = \"truthful\" if row[\"bid\"] == row[\"value\"] else (\n        \"overbid\" if row[\"bid\"] > row[\"value\"] else \"underbid\"\n    )\n    print(f\"Bidder {row['bidder']}: {label} (bid-value={row['bid'] - row['value']:+d})\")\n\n# Illustrative expected structure: three transcripts, bids, winner, second price, surplus.\n# Actual bids are stochastic and must not be replaced by the lecture's sample numbers.\n"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": "# Self-contained 20-round experiment. Rerunning starts fresh from seed 42.\nimport getpass\nimport os\nimport random\nimport re\nimport time\nimport pandas as pd\nfrom openai import OpenAI, APIConnectionError, APIError, RateLimitError\n\nTAMU_BASE_URL = \"https://chat.tamu.ai/api\"\nSONNET_MODEL = \"protected.Claude Sonnet 4.5\"\nN_ROUNDS = 20\nSEED = 42\nBIDDER_PROFILES = [\n    \"Theory-aware: explicitly use the second-price dominant-strategy argument.\",\n    \"Concise: focus only on the auction rule and your private value.\",\n    \"Cautious: check whether shading or inflating the bid could improve utility.\",\n]\n\ndef _cookie(raw):\n    raw = raw.strip().strip('\"').strip(\"'\")\n    match = re.search(r\"CF_Authorization=([^;\\s]+)\", raw)\n    return f\"CF_Authorization={match.group(1)}\" if match else (\n        f\"CF_Authorization={raw}\" if raw.startswith(\"eyJ\") else raw\n    )\n\nif \"client\" not in globals():\n    key = os.environ.get(\"TAMU_API_KEY\") or getpass.getpass(\"TAMU_API_KEY: \")\n    raw = os.environ.get(\"CF_COOKIE\") or getpass.getpass(\"CF_COOKIE: \")\n    client = OpenAI(api_key=key, base_url=TAMU_BASE_URL,\n                    default_headers={\"Cookie\": _cookie(raw)})\n\ndef request_bid(value, profile):\n    prompt = f\"\"\"You are in a sealed-bid second-price auction. Your private value is ${value}.\nThe highest bid wins and pays the second-highest bid. Utility is value minus price if you win,\nand zero otherwise. Bid an integer from 0 to 100. {profile}\nReturn exactly: RATIONALE: one short visible sentence, then BID: integer.\nDo not provide private chain-of-thought.\"\"\"\n    for attempt in range(4):\n        try:\n            response = client.chat.completions.create(\n                model=SONNET_MODEL,\n                messages=[{\"role\": \"user\", \"content\": prompt}],\n                temperature=1,\n                max_tokens=16_384,\n            )\n            visible = re.sub(r\"<think>.*?</think>\", \"\", response.choices[0].message.content,\n                             flags=re.DOTALL).strip()\n            matches = re.findall(r\"(?im)^\\s*BID\\s*:\\s*\\$?(-?\\d+)\\s*$\", visible)\n            if not matches:\n                raise ValueError(f\"Could not parse BID from {visible!r}\")\n            bid = max(0, min(100, int(matches[-1])))\n            usage = response.usage\n            return bid, visible, (\n                getattr(usage, \"prompt_tokens\", 0) or 0,\n                getattr(usage, \"completion_tokens\", 0) or 0,\n            )\n        except (RateLimitError, APIConnectionError, APIError, ValueError) as exc:\n            if attempt == 3:\n                raise\n            delay = 2 ** attempt\n            print(f\"Transient {type(exc).__name__}; retrying in {delay}s...\")\n            time.sleep(delay)\n\nrng = random.Random(SEED)\nauction_records = []\nprompt_tokens = completion_tokens = 0\nfirst_transcript = None\n\nfor round_number in range(1, N_ROUNDS + 1):\n    round_rows = []\n    values = [rng.randint(0, 100) for _ in BIDDER_PROFILES]\n    for bidder_id, (value, profile) in enumerate(zip(values, BIDDER_PROFILES), start=1):\n        bid, transcript, usage = request_bid(value, profile)\n        prompt_tokens += usage[0]\n        completion_tokens += usage[1]\n        row = {\n            \"round\": round_number, \"bidder\": bidder_id,\n            \"value\": value, \"bid\": bid, \"deviation\": bid - value,\n        }\n        round_rows.append(row)\n        if first_transcript is None:\n            first_transcript = {**row, \"visible_response\": transcript}\n\n    ranked = sorted(round_rows, key=lambda row: (-row[\"bid\"], row[\"bidder\"]))\n    winner_id, price = ranked[0][\"bidder\"], ranked[1][\"bid\"]\n    for row in round_rows:\n        row[\"won\"] = row[\"bidder\"] == winner_id\n        row[\"price\"] = price if row[\"won\"] else 0\n        row[\"utility\"] = row[\"value\"] - price if row[\"won\"] else 0\n        row[\"classification\"] = (\n            \"truthful\" if row[\"deviation\"] == 0 else\n            \"overbid\" if row[\"deviation\"] > 0 else \"underbid\"\n        )\n        auction_records.append(row)\n    print(f\"Round {round_number:02d}/{N_ROUNDS}: values={values}, \"\n          f\"bids={[row['bid'] for row in round_rows]}, winner={winner_id}, price={price}\")\n\nauction_df = pd.DataFrame(auction_records)\ncounts = auction_df[\"classification\"].value_counts().reindex(\n    [\"truthful\", \"overbid\", \"underbid\"], fill_value=0\n)\nper_bidder = auction_df.groupby(\"bidder\").agg(\n    truthful_rate=(\"deviation\", lambda x: (x == 0).mean()),\n    mean_deviation=(\"deviation\", \"mean\"),\n    mean_abs_deviation=(\"deviation\", lambda x: x.abs().mean()),\n)\nestimated_cost = prompt_tokens / 1_000_000 * 3 + completion_tokens / 1_000_000 * 15\n\nprint(\"\\n=== First visible transcript ===\")\nprint(first_transcript)\nprint(\"\\n=== Aggregate deviations ===\")\nprint(counts)\nprint(f\"Mean absolute deviation: {auction_df['deviation'].abs().mean():.2f}\")\ndisplay(per_bidder)\nprint(f\"Tokens: prompt={prompt_tokens}, completion={completion_tokens}\")\nprint(f\"Approximate Sonnet cost: ${estimated_cost:.4f} of $5.00/day\")\n\n# Expected output shape: 20 round summaries, 60 bidder observations, aggregate\n# truthful/overbid/underbid counts, per-bidder deviations, and token accounting.\n"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": "# Plot live data when present; otherwise use labeled illustrative data solely to\n# demonstrate the visualization code without consuming API credit.\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\n\nif \"auction_df\" not in globals():\n    print(\"No live auction data found: plotting ILLUSTRATIVE values.\")\n    rng = np.random.default_rng(42)\n    values = rng.integers(0, 101, size=60)\n    deviations = rng.choice([-4, 0, 0, 0, 0, 3], size=60)\n    auction_df = pd.DataFrame({\n        \"round\": np.repeat(np.arange(1, 21), 3),\n        \"bidder\": np.tile([1, 2, 3], 20),\n        \"value\": values,\n        \"bid\": np.clip(values + deviations, 0, 100),\n    })\n    auction_df[\"deviation\"] = auction_df[\"bid\"] - auction_df[\"value\"]\n\nfig, ax = plt.subplots(figsize=(7, 6))\nfor bidder_id, group in auction_df.groupby(\"bidder\"):\n    ax.scatter(group[\"value\"], group[\"bid\"], alpha=0.72, label=f\"Bidder {bidder_id}\")\nax.plot([0, 100], [0, 100], \"k--\", linewidth=1.5, label=\"Truthful: bid = value\")\n\nif auction_df[\"value\"].nunique() > 1:\n    slope, intercept = np.polyfit(auction_df[\"value\"], auction_df[\"bid\"], 1)\n    x = np.array([0, 100])\n    ax.plot(x, slope * x + intercept, color=\"tab:red\", linewidth=1.5,\n            label=f\"OLS: bid = {slope:.2f}·value + {intercept:.2f}\")\n\nax.set(xlabel=\"Private value\", ylabel=\"Submitted bid\", xlim=(-2, 102), ylim=(-2, 102),\n       title=\"LLM bids versus private values\")\nax.legend()\nax.grid(alpha=0.2)\nplt.show()\n\ndeviation = auction_df[\"bid\"] - auction_df[\"value\"]\nprint(f\"Observations: {len(auction_df)}\")\nprint(f\"Truthful: {(deviation == 0).sum()} ({(deviation == 0).mean():.1%})\")\nprint(f\"Overbids: {(deviation > 0).sum()} ({(deviation > 0).mean():.1%})\")\nprint(f\"Underbids: {(deviation < 0).sum()} ({(deviation < 0).mean():.1%})\")\nprint(f\"Mean deviation: {deviation.mean():.2f}\")\nprint(f\"Mean absolute deviation: {deviation.abs().mean():.2f}\")\n"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "## Discussion: theory, observed behavior, and content auctions\n\nAsk first whether deviations are systematic and whether they change allocation or payment. A harmless-looking overbid in one realized round can still be strategically dangerous because another bid profile could make it win above value. Likewise, truthful samples do not prove the model understood dominance; they only show that its submitted action matched the prescription.\n\nThis scalar-value auction is a **bridge**, not a reproduction of Dütting et al. Their WWW 2024 paper *Mechanism Design for Large Language Models* studies jointly generated content: participants' preferences over stochastic outputs are encoded by LLMs, and single-dimensional bids influence aggregation **token by token**. Under a monotonicity condition, they construct a second-price payment rule even without starting from explicit scalar valuation functions. Our notebook isolates the familiar truth-telling benchmark before those richer output-distribution questions.\n\n**Reference:** Paul Dütting, Vahab Mirrokni, Renato Paes Leme, Haifeng Xu, and Song Zuo (2024), *Mechanism Design for Large Language Models*, Proceedings of The Web Conference (WWW), pp. 144–155; arXiv:2310.10826.\n"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "## Extensions for student investigation\n\n1. **Prompt ablation:** remove the phrase \"second-price\" or the dominant-strategy reminder and compare bid deviations with confidence intervals.\n2. **Mechanism comparison:** run the same bidders under a first-price rule, where bid shading can be rational.\n3. **Payoff relevance:** label each deviation by whether changing the bid to the true value would alter the winner, price, or bidder utility in that round.\n4. **Population robustness:** compare models, prompt styles, and model versions while holding value draws fixed.\n5. **Strategic interaction:** reveal limited public history and test whether bidders adapt across rounds.\n6. **Content mechanism:** represent preferences over candidate text continuations and prototype a simplified token-level aggregation rule inspired by Dütting et al.; state clearly which assumptions of their mechanism are omitted.\n\nFor every extension, save the exact prompt, model identifier, value seed, raw visible responses, parsing failures, sample count, and token use. Never save the personal Cloudflare cookie or course API key in submitted data.\n"
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}