Convened anonymously

As a seed-stage AI product spending ~$8k/month on frontier-model API calls, should we keep building on closed frontier APIs (Claude/GPT/Gemini) or migrate core inference to self-hosted open-weight models — weighing cost at scale, capability gaps, control/latency/privacy, and the second-order strategic effects, given mid-2026 pricing and open-model capability?

Chair’s Decision

minimax-pro

Chair's Decision

Stay on closed frontier APIs as the primary inference path, but stop treating that as the strategic decision — the strategic decision is to build the option to leave cheaply, starting this week.

Why this survives the dissent: The Devil's Advocate argued the cost math has flipped and self-hosting a 70B-class model on a single A100 saves 10–30× per token. The Quantifier and Senior Expert both rebutted this on the same axis: at $8k/month (≈160M–1B tokens), a production-grade self-hosted frontier model requires 4–8× H100/H200 and ~$12–16k/month all-in including MLOps — more than the current bill, not less. The DA's cheap self-host math only works on a single A100 running a quantized 70B, which the Senior Expert correctly identifies as a capability downgrade on the tasks (agentic reliability, long-horizon reasoning) that frontier APIs still lead on by 10–14 points. The DA's strongest case is actually the hosted open-weight API middle path — which this decision adopts directly, just not as a wholesale migration.

Dissent I am not papering over: The Devil's Advocate is right that "stay on closed APIs" as a posture is strategically dangerous if it means continuing to write provider-specific code without a router or model-agnostic evals. The verdict stands because the decision bundles the hosted open-weight pilot and the router/eval build into the stay-on-APIs recommendation — addressing the optionality erosion the DA warned about, without paying the 2.5–3× TCO multiplier of premature owned-GPU self-hosting. The Strategist's framing — "are you building the option to migrate, or burning it?" — is what survives.

Residual risk: If open-weight models continue closing the gap on agentic reliability (currently a 10–14 point gap on WebArena/OSWorld), the "frontier API for hard 20%, open-weight for routine 80%" split could invert within 12 months, and this decision will look late. Watch the Strategist's third leading indicator: if API spend as % of revenue grows faster than revenue itself for two consecutive quarters, accelerate the hosted open-weight share toward 50–60%.

Concrete action (next 30 days):

  1. Stand up LiteLLM or OpenRouter as the inference gateway in front of all current calls.
  2. Enable prompt caching on Anthropic + semantic caching — target 30–50% spend reduction immediately, taking the $8k bill to $4–5.5k.
  3. Run a parallel 2-week eval-gated pilot routing 15–20% of traffic to hosted open-weight (DeepSeek V4 Flash / GLM 5.2 via Fireworks or Together), measured against your existing eval suite.
  4. By end of Q2 2026, set a re-evaluation gate at $25k/month API spend OR if hosted open-weight eval deltas stay within 2–3 points of frontier — whichever comes first.

Council recommendation

Verdict
Recommendation

The council recommends staying with closed frontier APIs for now but emphasizes the strategic importance of building the option to migrate later by investing in model-neutral infrastructure, including a routing layer, eval suite, and workload-class-targeted hosted open-weight models.

Where the council agrees

The council agrees that self-hosting a frontier-class model does not make economic sense at $8k/month, given the Total Cost of Ownership (TCO) calculations. They also concur on the importance of building a model-agnostic infrastructure to maintain flexibility and prevent future lock-in.

Where the council splits

The primary split is between the Devil's Advocate and the rest of the council regarding the timing and extent of migrating some traffic to open-weight models. The Devil's Advocate argues for an immediate, binding migration of 15-20% of traffic to hosted open-weight models to force infrastructure development, while others suggest a more cautious approach through piloting and developing the infrastructure alongside current API use.

Strongest seat
Senior Expert
Suggested action

Institute a model-neutral infrastructure by establishing a routing layer and conducting a workload-class-targeted pilot for hosted open-weight models focusing on routine tasks. Implement a harness-neutrality rule to avoid provider lock-in. Monitor for economic milestones and refine strategy accordingly.

How minds moved

Seats shifted in their emphasis on specific action items. The Strategist sharpened focus on the harness neutrality issue while the Devil's Advocate insisted on immediate partial migration. The Senior Expert adjusted its emphasis to integrate more validation checks before migration.

The deliberation

6 labs · 6 seats

Council members

Senior Expert

Seat 3 · Standard · Kimi

Let me think about what a Round 2 Senior Expert response should do here.

My Round 1 position (Seat 3, Senior Expert, kimi) was: false binary — three tiers, not two. Own the router and evals, rent the weights. Strangler-fig migration per workload class. At $8K/month self-hosting is 3–10× below the lowest defensible threshold. Watch four boundary conditions. Named patterns: model cascade/routing layer (RouteLLM, LiteLLM/OpenRouter gateway), strangler fig, Wardley mapping. Named failure modes: list-price fallacy, capability freeze.

Now in Round 2, I need to engage with the other seats' actual arguments:

  1. Framer (Seat 1): Posed Decision Zero — is self-hosting economically rational at all? Surfaced the hosted open-weight API middle path. Unknowns: token volume shape, capability tier needed, data-residency obligations. My Round 1 aligns strongly with the Framer. I can credit the Framer for naming Decision Zero correctly and note that the entire panel converged on answering it "no."

  2. Quantifier (Seat 2, gemini-pro): Interesting — there are TWO quantifier-ish texts in the prompt. The first panel block (Seat 1's big combined output) contains a "Seat 2 — Quantifier" section that computed ~$7.80/M blended, ~1.03B tokens/month, self-host floor $11.8–15.7K/month, break-even at $25–35K/month. But then there's a separate "Seat 2 — Quantifier (gemini-pro)" that starts with GPT-4o at $2.50/$10 (that's stale mid-2025 pricing, not mid-2026!) and cuts off mid-sentence ("currently processing **" — truncated). The actual gemini-pro response is truncated and uses GPT-4o pricing — a stale model for a mid-2026 scenario. As Senior Expert I can note that the Quantifier's math is right in structure but anchored on a deprecated price point (GPT-4o was retired from the API in 2025/2026 — actually I should be careful with time-pinned claims; GPT-4.5 deprecation was April 2025 per the Strategist's citation; GPT-4o deprecation... I shouldn't assert specifics without search. But I CAN note that GPT-4o pricing is a stale reference point for a mid-2026 question — the panel's own sources use Sonnet 4.6/5.5-class pricing). The break-even conclusions converge anyway ($25K+ for hybrid), so this is a refinement, not a fight.

  3. Strategist (Seat 4, glm): Capability debt compounding, harness lock-in (LangChain citation), three leading indicators (provider-specific code ratio >30%, fallback tested monthly, API spend as % of revenue). This complements my position. I should concede/incorporate: the Strategist sharpened something I said ("own the evals, rent the weights") into measurable leading indicators — the 30% provider-specific coupling ratio is a concrete early tell I can adopt. Also "harness lock-in > model lock-in" is a genuinely sharper formulation than my deprecation-churn point. I should concede that the Strategist reframed my "deprecation churn" mitigation correctly: version pinning + evals is necessary but not sufficient; the deeper coupling is at the harness/agent-SDK layer.

  4. Devil's Advocate (Seat 5, deepseek): Claims cost math "already flipped": 500M tokens/month on a single A100 80GB at $1,440/month, $0.004–0.006 per million tokens... wait, that math is internally inconsistent: $2,200–2,800/month at 500M tokens/month = $4.40–5.60 per million tokens? No: $2,800/500M tokens = $5.6 per 1M tokens. The DA wrote "$0.004–0.006 per million tokens" — that's off by 1000×. Then later says "$0.17–0.62 per million tokens at full utilization" — $2,800/500 = $5.6/1M, so even $0.17–0.62 is wrong by ~10–30×. This is a glaring arithmetic error I should catch as the Senior Expert — a 1000× unit error in the DA's central claim. Also the DA's throughput assumption: can a single A100 80GB serving a 4-bit 70B model actually do 500M tokens/month? 500M tokens/month = ~193 tokens/second sustained average, 24/7. A single A100 running quantized Llama 70B with vLLM can do maybe 1,000–3,000 tok/s aggregate with heavy batching at high concurrency... actually for 4-bit 70B on A100, aggregate throughput with continuous batching can reach ~1,500–3,000 tok/s under ideal batch conditions, but seed-stage traffic is bursty, not batched 24/7. So 193 tok/s average might be feasible but requires ~100% sustained utilization at night too — exactly the list-price fallacy I named (assuming 24/7 utilization on bursty traffic). And P95 latency at high batch = poor. Also single box = no redundancy, no failover — a single point of failure for a production product. That's a classic junior-TCO error: no HA pair.

Also the DA claims "Llama 3.1 405B matches GPT-4o and Claude Sonnet within 1–3%" — but then proposes self-hosting a 70B on a single A100. You can't run 405B on a single A100 80GB, even quantized (405B at 4-bit = ~230GB+ weights). So the DA's capability claim (405B-class parity) and cost claim (70B on one A100) are about different models — a bait-and-switch within their own argument. The model that fits the budget (70B) is NOT the model that matches frontier (405B). That's the sharpest technical catch available, and it's exactly a Senior Expert's job: pattern-name it as "benchmark bait-and-switch" — quoting parity numbers from the flagship open model, then pricing the deployment of the mid-tier one.

Also the DA says the breakeven happens at "3–6 weeks" — inconsistent with their own numbers and with every TCO analysis in the record.

Also, note there are two DA texts — one truncated in the Framer's block ("was correct in" cuts off) and one full one later. I'll engage with the full one.

Also the DA's point about Claude Sonnet 5 introductory pricing jumping 50% on Sept 1, 2026 — that's a time-pinned claim I can't verify and shouldn't lean on; I can note it's the deprecation/repricing risk which I already acknowledged, mitigated by version pinning + router.

  1. Chair (Seat 6, minimax-pro): Decision: stay on closed APIs + build the option to leave; 30-day actions (gateway, caching, 15–20% pilot to hosted open-weight, re-eval gate at $25K/month or eval deltas within 2–3 points). The Chair adopted my framework almost wholesale (router, hosted middle path, strangler-style pilot). In Round 2 I should: endorse the decision structure, but sharpen two things where the Chair's actions could go wrong in practice — this is where Senior Expert authority adds value:

    a) The eval-gated pilot will fail if the evals don't measure tail behavior. Named failure mode from Round 1: benchmark optimism hides tail regressions. Concrete: the pilot must include canary replay (shadow mode) — run the open model in shadow against production traffic and diff outputs before any user-visible routing. Named pattern: shadow deployment / dark launch / canary analysis. Also: route by task class via a classifier, not randomly 15–20% of traffic — random routing mixes difficulty strata and contaminates the eval.

    b) The "$25K/month re-evaluation gate" needs a second trigger: spend composition, not just spend level. If the $25K is 90% one workload class, self-host/hybrid math changes. Also add the Strategist's API-spend-as-%-of-revenue trigger, which the Chair adopted as residual risk but not as a gate.

    c) Prompt caching caveat: caching discounts only help if cache hit rate is high; the Framer/Strategist mentioned 40% hit-rate threshold. Concrete guidance: measure cache-hit rate before projecting the 30–50% reduction; products with highly dynamic per-user context (RAG-heavy with unique retrieved chunks) see far lower hit rates. The early tell that caching won't save you: stable system prompt is <20% of token mass.

Also, the Strategist's GPT-4.5 deprecation example and my Gemini 2.0 Flash shutdown — version pinning mitigates but note: pinned versions eventually get sunset too; pinning buys months, not years. The real mitigation is the router + evals. That's a refinement of my own Round 1 claim ("version pinning plus evals") — I should refine: pinning is a stay of execution, not a defense; the eval suite is the defense. Actually I said "the mitigation is version pinning plus evals, not GPU ownership" — I can sharpen this in light of the Strategist's harness lock-in point: pinning doesn't protect you from harness-level coupling at all.

What changed my mind / refined:

  • Concede to Strategist: harness lock-in is the sharper risk frame; my deprecation-churn point was the visible tip. Adopt the 30% provider-specific coupling ratio as an early tell. My Round 1 treated control risk as mainly deprecation churn; the deeper control risk is behavioral assumptions baked into product UX via provider-specific features (structured outputs, caching, agent SDKs). This refines my boundary condition #1/#4.
  • Concede partially to DA: the one thing the DA got right that the consensus undersells — fine-tuning on proprietary data. My Round 1 boundary #2 mentioned distill/fine-tune for one dominant workload. The DA pushed that a seed company's moat is data adaptation, and managed fine-tuning APIs are limited/expensive. I should hold that this only activates under the >60% single-workload condition, but concede it's the strongest part of the DA's case and it deserves a place in the Chair's 30-day plan: start logging and structuring your own interaction data now, regardless of stack decision, because that dataset is the option collateral for any future distillation. That's cheap and stage-appropriate. Actually my Round 1 already said fine-tune on your own logged data. The refinement: make data-logging a day-one action, not a migration-time action.
  • Hold against DA: the arithmetic is wrong by orders of magnitude (catch the $0.004–0.006/1M error), the 405B-vs-70B bait-and-switch, single-box = no HA, and the "weekend of setup" claim ignores the capability freeze (every 6–8 week open SOTA churn is a re-validation project — my Round 1 point, which the DA never addressed).
  • Refine vs. Quantifier: the truncated gemini-pro response anchored on GPT-4o pricing ($2.50/$10) — stale for mid-2026; the correct anchor is Sonnet 4.6/5-class pricing, which changes the blended rate (~$4.50 → ~$6–8/M) and thus the implied token volume downward (~1B → ~600M–1B). The conclusion (break-even far above $8K) is robust to the error, but the record should be corrected. Actually wait — the first combined block has a fuller quantifier section with $7.80 blended. The standalone gemini-pro seat has $4.50 blended on GPT-4o. I should correct the standalone one. Hmm, but careful: is GPT-4o even available mid-2026? The Strategist cited GPT-4.5 deprecation April 2025. GPT-4o deprecation — I recall OpenAI announced GPT-4o retirement from API... I shouldn't assert without search. I can just say "anchored on a legacy price point" and note the panel's verified mid-2026 pricing is Sonnet-4.6/5.5-class. Do I need to search? My role says verify time-pinned claims. The correction itself is about which anchor is current — the Framer's block already contains verified July-2026 pricing sources. I could do one search to verify GPT-4o API status in 2026 to make the "stale anchor" claim checkable. That would be a good grounding move. Let me do a quick search on GPT-4o API deprecation 2026 status, and maybe one on A100 70B throughput to check my throughput rebuttal. Actually the throughput rebuttal: is 500M tokens/month feasible on a single A100 with 4-bit 70B? Let me sanity check: 500M/month ≈ 5.8M/day ≈ 193 tok/s average. vLLM on A100 80GB with AWQ 70B: prefill is compute-bound, decode is memory-bandwidth-bound; with continuous batching at high concurrency, aggregate decode throughput for 70B Q4 on A100 is roughly 800–2,500 tok/s depending on batch size and context length. So 193 tok/s average is technically within reach if you batch aggressively 24/7 — but seed traffic is bursty with idle nights, so average utilization ~40–50% at best, and latency degrades at high batch. So the honest rebuttal: it's physically possible only at utilization levels seed-stage traffic doesn't have, and with zero redundancy. The deeper point stands: even if the box could serve the volume, the DA's own per-token math is off by 1,000× ($2,800/500M = $5.60/1M, not $0.0056/1M). Wait, recompute: $2,800 ÷ 500M tokens = $0.0056 per token? No. $2,800 / 500,000,000 tokens = $5.6e-6 per token = $5.60 per 1M tokens. The DA claimed "$0.004–0.006 per million tokens" — off by 1,000× (they computed per-token and labeled it per-million). And their second figure "$0.17–0.62 per million tokens at full utilization" is also inconsistent with $5.60/1M at stated volume. So even granting the DA's own hardware and volume assumptions, self-hosted lands at $4.40–5.60/1M — the same order as the mid-tier API blended rate ($6–8/1M), not 10–30× cheaper. That completely collapses the "already flipped" claim on the DA's own numbers. This is the kill shot and it's pure arithmetic — perfect for a Senior Expert catch (well, more Quantifier-flavored, but pattern-named: this is the list-price fallacy I predicted, appearing live in the council record — unit confusion between per-token and per-million).

That's a beautiful moment: my Round 1 named the failure mode (list-price fallacy: divide GPU hourly by theoretical throughput, conclude "10× cheaper"), and the DA's response exhibits it verbatim, with a 1000× unit error. I should point that out crisply: "The early tell I named was a spreadsheet with no labor line and 24/7 utilization; the DA's spreadsheet additionally divides by a thousand."

Let me also double check the DA's "$1,440/month A100" — that's ~$2/hr which matches Lambda/RunPod-ish A100 80GB pricing. Fine. And 4-bit 70B on 80GB: 70B × 0.5 bytes ≈ 35GB weights + KV cache — fits, yes. So hardware claim is plausible; the problem is throughput/utilization, HA, capability tier, and arithmetic.

Also the DA's claim "Llama 3.1 405B matches GPT-4o within 1–3%" — mid-2024 benchmarks vs mid-2026 frontier is anachronistic; GPT-4o and Claude 3.5 Sonnet are not the mid-2026 frontier (Opus 4.8, GPT-5.5 are). The DA is comparing 2024 open models to 2024 closed models while the question is about mid-2026. My Round 1's presenc.ai snapshot had the current gaps (~5–9 pts SWE-bench, 12–14 agentic). I can note the anachronism: citing 2024-era parity studies in a mid-2026 decision is a stale-prior error — the gap narrative has moved to agentic/long-horizon, where it persists.

Structure for my Round 2 response:

Senior Expert — Round 2

Opening: what I'm holding, what I'm conceding, what I'm correcting.

1. Held: the DA's cost case collapses on its own arithmetic (challenge Seat 5)

  • The 1,000× unit error: $2,800/mo ÷ 500M tokens = $5.60/1M, not $0.0056/1M. On the DA's own numbers, self-host lands at parity with mid-tier API blended rates — before labor, before HA, before the capability downgrade.
  • The benchmark bait-and-switch: parity claims cite 405B-class flagships; the deployment plan prices a 70B on one A100. You cannot serve 405B-class on a single 80GB card (~230GB+ quantized weights). The model that fits the budget is not the model that matches the benchmarks. Also 405B needs 8× GPUs = $12–25K/mo — back above the API bill.
  • Single box = single point of failure for a production product; no failover. A "weekend of setup" gets you a demo, not production serving (no autoscaling, no rollback, no eval gate on model upgrades — the capability freeze I named; the DA never answered it).
  • Anachronistic benchmarks: Llama 3.1 405B vs GPT-4o is a 2024 comparison. The mid-2026 question is Opus 4.8 / GPT-5.5-class, where the residual gap is concentrated in agentic reliability — the exact workload class a growing AI product is usually adding, not shedding.
  • Conclusion: this is my Round 1 failure mode appearing live — the list-price fallacy, complete with the 24/7-utilization assumption and a unit error.

But concede the DA's strongest kernel: data moat / fine-tuning. Fold into recommendation: log and structure interaction data from day one (consent/ToS-clean), because that dataset is the collateral for the boundary-#2 distillation play. The DA is wrong about GPUs now, right about data now.

2. Conceded

Bring your own decision

This was one council. Yours is next.

Reserve a seat and we’ll bring you in — with $5 in founder credit to convene your first councils on us.

No card required. One email at most per week, only when there’s news worth your attention.