# 🦉 ResearchOwl

**Exhaustive research engine with Telegram interface.**

Recursively discovers, scrapes, and processes sources from across the web,
then generates podcast scripts, blog posts, reports, or social threads using Ollama.

## Architecture

```
Telegram (/research <topic>)
    ↓
ExhaustiveScraper
    ├── DuckDuckGo (8 queries × 5 results)
    ├── Wikipedia + recursive internal links
    ├── Reddit (top posts + top comments)
    ├── YouTube (transcripts)
    ├── PDFs (public documents)
    └── Web scraping (trafilatura)
         ↓ recursive expansion (depth 1-3)
ContentProcessor (Ollama qwen2.5:7b + bge-m3 embeddings)
    ├── Chunking (800 token chunks, 100 overlap)
    ├── Quality scoring (0-10 per chunk)
    ├── Embeddings (cosine similarity RAG)
    └── Deduplication
         ↓
OutputGenerator (Ollama)
    ├── 🎙️ Podcast script (20-30 min)
    ├── 📝 Blog post (1500-2500 words)
    ├── 📊 Research report (structured)
    └── 🐦 Social thread (15-25 tweets)
```

## Telegram Commands

| Command | Description |
|---------|-------------|
| `/research <topic>` | Start exhaustive research |
| `/status` | Check progress |
| `/finish` | Stop early, proceed to generation |
| `/generate podcast\|blog\|report\|thread` | Generate output |
| `/generate short_en` | Vertical Short: shot spec → grounding check → MP4 |
| `/short_spec` | Last shot spec as a JSON file, to hand-edit and re-render |
| `/sources` | List all sources found |
| `/cancel` | Cancel current research |

## Shorts (`/generate short_en`)

Claude writes a **shot spec** — typed JSON, not prose — which
[shortsmith](https://git.chemavx.xyz/chemavx/shortsmith) renders into a 1080×1920
MP4. The bot sends the video and, in a separate message, a **claims report**.

```
/research JAL 1628 Alaska 1986 …
/generate blog en          → Ghost draft, article URL stored on the output row
/generate short_en         → spec → grounding → render → video + claims report
                             (YouTube upload is deliberately manual)
```

Three things make this different from generating text, and each has its own
mitigation:

- **It is a contract, not prose.** The template schemas are fetched live from
  `GET /templates` and never copied here, so a template added to shortsmith is
  available immediately. A spec is validated locally against those schemas
  before anything renders, and the exact error paths
  (`shots.0.radar_sweep.props.sweeeps`) go back to the model verbatim — up to 3
  attempts.
- **It contains figures and quotes.** `grounding.py` extracts every quote,
  figure, date and proper noun and checks it against the exact chunks the model
  was given. No LLM in that path: normalisation plus substring, deterministic
  and free. Whatever is not in the chunks is checked against the worked example
  that travels in the prompt, so a figure lifted from it is reported as a
  **prompt leak**, not as an invention — different diagnosis, different fix.
  Neither ever blocks the render: both are surfaced next to the video and a
  human decides.
- **It becomes a published video.** Nothing is uploaded anywhere. The MP4 lands
  in Telegram for review, and in `/data/shorts/{session_id}.mp4`.

Fallbacks hold throughout: if shortsmith is unreachable, the job errors, or the
spec never validates, the spec JSON comes back as a file. The expensive part is
the generation, not the render.

Full spec of the phase: `docs/shortsmith-phase2-spec.md`.

## Local Development

```bash
# 1. Clone and setup
git clone https://git.chemavx.xyz/chemavx/researchowl
cd researchowl

# 2. Create virtualenv
python3 -m venv venv && source venv/bin/activate
pip install -r requirements.txt

# 3. Configure
cp .env.example .env
# Edit .env with your values

# 4. Run
python main.py
```

## Deploy to k3s

```bash
# 1. Create namespace and secrets
kubectl create namespace researchowl
kubectl create secret generic researchowl-secrets \
  --from-literal=telegram-bot-token=YOUR_TOKEN \
  --from-literal=telegram-allowed-users=YOUR_USER_ID \
  -n researchowl

# 2. Copy manifests to your k8s-manifests repo
cp k8s/*.yaml /path/to/k8s-manifests/researchowl/

# 3. Apply ArgoCD app
kubectl apply -f k8s/argocd-app.yaml

# 4. Push to Gitea → Gitea Actions builds → ArgoCD deploys
git add . && git commit -m "feat: add researchowl" && git push
```

## Tuning

| Variable | Default | Description |
|----------|---------|-------------|
| `MAX_SOURCES` | 150 | Hard cap on sources |
| `MAX_DEPTH` | 3 | Link recursion depth |
| `QUALITY_THRESHOLD` | 0.4 | Min chunk quality (0-1) |
| `REQUEST_DELAY` | 1.0s | Delay between requests |

**Want more thoroughness?**
- Increase `MAX_SOURCES` to 300+
- Increase `MAX_DEPTH` to 4-5
- Lower `QUALITY_THRESHOLD` to 0.3

**Want faster results?**
- Lower `MAX_SOURCES` to 50
- Set `MAX_DEPTH` to 1-2
- Higher `QUALITY_THRESHOLD` to 0.6

## Bot avatar

The profile picture of `@chemavx_researchowl_bot` is not an opaque binary
checked into the repo: `assets/make_avatar.py` draws it with PIL at 4× and
scales it down, so the emblem can be retouched without hunting for an original.
Telegram crops avatars to a **circle**, so everything that matters lives inside
the inscribed circle; verified legible at 48 px.

```bash
python3 assets/make_avatar.py     # writes assets/avatar.png
```

It is applied **over the API with the token from the secret, no BotFather**.
Watch out for `setMyProfilePhoto`: its `photo` parameter is not the file, it is
an `InputProfilePhoto` object pointing at the attachment. Posting the file on
its own gets you a baffling `photo isn't specified`.

```bash
TOK=$(kubectl get secret researchowl-secrets-infisical -n researchowl \
        -o jsonpath='{.data.telegram-bot-token}' | base64 -d)
curl -s -F 'photo={"type":"static","photo":"attach://av"}' \
     -F "av=@assets/avatar.png" \
     "https://api.telegram.org/bot$TOK/setMyProfilePhoto"
unset TOK
```

## Notes

- Uses **qwen2.5:7b** (scoring) and **bge-m3** (embeddings) on your existing Ollama — zero API cost
- Optionally add `ANTHROPIC_API_KEY` for Claude fallback on generation
- SQLite database stored in `/data/researchowl.db`
- All outputs saved to DB and available via `/outputs`