researchowl/docs/shortsmith-phase2-spec.md

Phase 2 — researchowl → shortsmith integration

Handoff document for Claude Code. Prerequisite: shortsmith v1 deployed and healthy (shortsmith-svc.shortsmith.svc.cluster.local:8080).

• Repo touched: git.chemavx.xyz/chemavx/researchowl only. shortsmith is not modified.
• Deliverable: /generate short_en produces a rendered MP4 from a research session and delivers it to Telegram for human review.
• Explicitly out of scope: YouTube upload. That is phase 3. See §11.

---

1. What changes

/research <case>
        ↓
/generate blog en   →  Ghost article published, URL stored
        ↓
/generate short_en  →  Haiku writes a shot spec (JSON)
                    →  grounding check against source chunks
                    →  POST to shortsmith, poll, fetch MP4
                    →  Telegram: video + claims report
                    →  human reviews, uploads to YouTube manually

The Ghost step comes first and is a hard dependency: the Short's description links to the article, so the article URL must exist before the spec is generated. §6 covers what happens when it doesn't.

---

2. Why this is not like generating prose

Every other output type in generator.py produces text a human reads and judges. A shot spec is different in three ways, and each needs its own mitigation:

Property	Consequence	Mitigation
It's a typed contract, not prose	Malformed output is unusable, not merely poor	Validation retry loop, §4
It contains figures and quotes	These are exactly what an LLM fabricates	Grounding check, §5
It becomes a published video	An error is public and hard to retract	Human review gate, §8

The grounding check is the one that matters most. The channel's entire premise is that its numbers come from primary sources. A fabricated radar figure in a 40-second video is worse than no video.

---

3. Prompt construction — fetch the contract, don't hardcode it

shortsmith exposes GET /templates, which returns each template's prop schema. Fetch it at generation time and inject it into the prompt. Do not copy the schemas into researchowl.

This means adding a template to shortsmith makes it immediately available to the generator with no change here. Hardcoding the schemas would create a second source of truth that silently drifts — the same class of failure as the ffmpeg version delta that caused the v1 OOM.

Cache the response for the lifetime of the process; refetch on validation failure, in case the renderer was updated mid-run.

Narrative shapes

Free template choice produces mush. Give the model three shapes matching the three article types actually published, and have it pick one:

Shape	Arc	Fits
case_file	hook → date/place → witness credentials → escalation → evidence → official explanation and its problem → close	JAL 1628, Belgium, Ariel School
debunk	the claim → why it spread → the method → the finding → what it means → close	Roswell crater video, Yellow Sea star
document_drop	what was released → the standout item → context → what's still missing → close	PURSUE releases

examples/jal1628.json in the shortsmith repo is a worked case_file. Include it in the prompt as a full example — one concrete example is worth more than any amount of description.

Constraints to state explicitly

• Total duration 20–45 s. Not the 180 s contract ceiling; that is a hard limit, not a target.
• Per-template max_length limits exist and are enforced. Listing them in the prompt turns a rejection into a non-event.
• Colours are palette names (ink, amber, amber_dark, muted, dim, red), never hex.
• Every figure and quote must come from the supplied chunks. No outside knowledge, even if correct.

---

4. Validation retry loop

for attempt in range(3):
    spec = await _generate_spec(prompt, feedback)
    try:
        validated = await client.validate(spec)   # POST /render dry-run or local pydantic
        break
    except ValidationError as e:
        feedback = _format_errors(e)              # feed the exact paths back
else:
    return _fallback(spec)                        # §6

shortsmith's discriminated union produces precise error locations (shots.0.radar_sweep.props.sweeeps, or shots.0 with the valid template names listed). Feed those paths back verbatim. They are more useful to the model than any paraphrase.

Cap at 3 attempts. Log attempts-to-valid as a metric — if it trends above 1.5, the prompt needs work, not the retry limit.

---

5. Grounding check — the important part

After the spec validates and before rendering, verify every factual string in it appears in the session's source material.

def check_grounding(spec, chunks) -> list[Ungrounded]:
    """Extract figures and quoted strings from spec props, confirm each
       appears in at least one source chunk."""

What to extract from the spec:

• Every quoted string (quote_a, quote_b, quote, anything in “ ”)
• Every number with a unit or magnitude (35,000 FT, 1,500, ~1,600 – 2,000 FT, 50 minutes)
• Every date (17 NOV 1986, 5 MARCH 1987)
• Proper nouns in label/key positions (ELMENDORF ROCC, CAPT. KENJU TERAUCHI)

Matching: normalise both sides — case-fold, strip thousands separators, collapse whitespace, normalise quote glyphs and dashes. Then substring match against chunk text. No LLM in this path: it must be deterministic and free.

On failure: do not silently drop the shot and do not retry blindly. Return the spec plus the list of ungrounded strings, and surface them in Telegram (§8). A human decides whether it is a real fabrication or a formatting artefact.

Expect false positives at first — "twice the size of an aircraft carrier" appears in the source but a rephrasing would not match. That is the correct bias: a false positive costs a glance, a false negative costs the channel's credibility.

This is the automated version of the fact-check table that was written by hand for the first Short. That table is in short-01-jal1628-script.md if you want the shape of the output.

---

6. ShortsmithClient

Mirror the GhostPublisher shape in generator.py. Layer rule holds: generator/ does not import from bot/; progress is reported through a generic callable.

class ShortsmithClient:
    def __init__(self, base_url: str, timeout: float = 600.0)
    async def templates(self) -> dict
    async def render(self, spec: dict) -> str            # -> job_id
    async def poll(self, job_id, on_progress=None) -> JobResult
    async def fetch_video(self, job_id) -> bytes

Polling: 2 s interval, 10 min ceiling. A 42 s Short renders in ~32 s; the 180 s ceiling takes ~138 s. Anything past 10 min is a stuck job, not a slow one.

Fallbacks always (repo convention). If shortsmith is unreachable, or the job errors, or grounding fails hard — return the spec JSON to Telegram as a file. The expensive part is the generation, not the render. Never discard it.

Config (src/config.py, Pydantic Settings, env-direct — no secret):

SHORTSMITH_URL = http://shortsmith-svc.shortsmith.svc.cluster.local:8080
SHORTSMITH_TIMEOUT = 600
SHORTSMITH_ENABLED = true

SHORTSMITH_ENABLED=false must make /generate short_en reply that the feature is off, not crash. This is the kill switch if the renderer misbehaves while nobody is watching.

---

7. Database

No migrations (CREATE TABLE IF NOT EXISTS convention holds).

• outputs takes output_type='short_en', content = the spec JSON as text.
• New: the Ghost article URL must be retrievable. Check whether GhostPublisher already persists it; if not, store it on the outputs row for the blog post, or add a published_url column to outputs (nullable, ALTER TABLE guarded by a column check). The spec generator needs it for the description.
• Store the rendered MP4 on disk, not in SQLite. /data/shorts/{session_id}.mp4. Blobs in SQLite will make the WAL pathological.

---

8. Telegram flow

/generate short_en — reuse ProgressReporter, editing a single message:

🎬 Writing shot spec…            (Haiku, ~5 s)
🔍 Checking claims against sources…
🎞  Rendering… 40%               (progress from shortsmith poll)
📤 Uploading…

Then send the MP4 as a video message (not a document, so it plays inline), with a caption carrying the title and the article URL.

Immediately after, send the claims report as a separate message. This is the review gate and it must be impossible to miss:

✅ 11 claims matched to sources
⚠️ 2 not found:
   • "roughly 1,600 feet across"
   • "NORAD confirmed"

Sources: 14 chunks from 9 URLs
Cost: $0.004

Zero ungrounded claims still sends the report, saying so. A silent success trains the reader to stop looking.

Also add /short_spec to return the last spec JSON as a file, for hand-editing and re-rendering without regenerating.

---

9. Cost

One Haiku call over the top-scored chunks. ~$0.003–0.008, plus retries. Rendering is free (own hardware). A Short costs roughly what a /generate blog costs, which for practical purposes is nothing — the constraint on volume is review time, not money.

---

10. Tests

Area	Assert
Spec generation	Mocked Haiku response validates; malformed response triggers retry with error paths fed back; 3 failures fall through to fallback
Grounding	Known-good spec over known chunks yields zero ungrounded; a spec with an injected fabricated figure flags exactly that string; normalisation handles thousands separators, curly quotes, en-dashes
Client	Poll loop handles queued→running→done, error status, timeout, connection refused
Fallback	Every failure path returns the spec JSON rather than nothing
Layer separation	grep that generator/ does not import from bot/

Golden eval, worth building once: run the generator against the stored JAL 1628 session and compare the output structurally to examples/jal1628.json — shape count, templates chosen, total duration, zero ungrounded claims. Not string equality; the model will phrase differently. It answers "could this pipeline have produced the video we already know is good?"

---

11. Out of scope — phase 3

YouTube upload via Data API v3. Deliberately excluded: it needs OAuth with a stored refresh token in researchowl-secrets (managed imperatively), a new failure surface, and it removes the human from the loop at exactly the point where the human is most valuable.

Ship phase 2, publish five or six Shorts by hand, then decide whether the review step is actually a bottleneck. It probably is not.

short_es for Zona de Exclusión is nearly free once this works — shortsmith draws whatever strings it is given and does not care about language. Only the prompt and the narrative shapes need translating. Do it after short_en has produced something worth publishing, not before.

---

12. Implementation order

One change at a time, verified before the next.

1. ShortsmithClient + config + tests, against the live service. No generation yet — prove the plumbing by POSTing examples/jal1628.json and getting the MP4 back.
2. Grounding checker + tests, standalone. Test it against the known-good JAL 1628 spec and against a deliberately corrupted copy.
3. Spec generation: prompt, GET /templates injection, retry loop.
4. Wire output_type='short_en' into generator.py; article URL retrieval.
5. Telegram /generate short_en and /short_spec.
6. Golden eval against the JAL 1628 session.

Step 2 before step 3 is deliberate. Build the check before the thing it checks, so the first generated spec is graded by a checker that was written without knowledge of it.