ChemaVXandClaude Opus 5
20c8d03aa7
Build & Deploy ResearchOwl / build-and-push (push) Successful in 9s
Añade /generate short_en y /short_spec. El pipeline genera un shot spec con Haiku, verifica cada cifra, fecha y cita contra los chunks de la sesión, lo renderiza en shortsmith y entrega el MP4 por Telegram junto a un informe de claims. - ShortsmithClient con sondeo y fallback al spec JSON si el render falla - Contrato de plantillas obtenido de GET /templates, no codificado - Comprobación de fundamento determinista, sin LLM - outputs.published_url para enlazar el artículo de Ghost - Normalización de comillas rectas a tipográficas (ver KNOWN-ISSUES.md) Lo que no aparece en los chunks se contrasta contra el ejemplo del prompt: si casa ahí es fuga, no invención, y se informa como tal. El purgado de sesiones se lleva también su MP4. La subida a YouTube queda fuera a propósito: fase 3. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
176 lines
6.2 KiB
Markdown
176 lines
6.2 KiB
Markdown
# 🦉 ResearchOwl
|
||
|
||
**Exhaustive research engine with Telegram interface.**
|
||
|
||
Recursively discovers, scrapes, and processes sources from across the web,
|
||
then generates podcast scripts, blog posts, reports, or social threads using Ollama.
|
||
|
||
## Architecture
|
||
|
||
```
|
||
Telegram (/research <topic>)
|
||
↓
|
||
ExhaustiveScraper
|
||
├── DuckDuckGo (8 queries × 5 results)
|
||
├── Wikipedia + recursive internal links
|
||
├── Reddit (top posts + top comments)
|
||
├── YouTube (transcripts)
|
||
├── PDFs (public documents)
|
||
└── Web scraping (trafilatura)
|
||
↓ recursive expansion (depth 1-3)
|
||
ContentProcessor (Ollama qwen2.5:7b + bge-m3 embeddings)
|
||
├── Chunking (800 token chunks, 100 overlap)
|
||
├── Quality scoring (0-10 per chunk)
|
||
├── Embeddings (cosine similarity RAG)
|
||
└── Deduplication
|
||
↓
|
||
OutputGenerator (Ollama)
|
||
├── 🎙️ Podcast script (20-30 min)
|
||
├── 📝 Blog post (1500-2500 words)
|
||
├── 📊 Research report (structured)
|
||
└── 🐦 Social thread (15-25 tweets)
|
||
```
|
||
|
||
## Telegram Commands
|
||
|
||
| Command | Description |
|
||
|---------|-------------|
|
||
| `/research <topic>` | Start exhaustive research |
|
||
| `/status` | Check progress |
|
||
| `/finish` | Stop early, proceed to generation |
|
||
| `/generate podcast\|blog\|report\|thread` | Generate output |
|
||
| `/generate short_en` | Vertical Short: shot spec → grounding check → MP4 |
|
||
| `/short_spec` | Last shot spec as a JSON file, to hand-edit and re-render |
|
||
| `/sources` | List all sources found |
|
||
| `/cancel` | Cancel current research |
|
||
|
||
## Shorts (`/generate short_en`)
|
||
|
||
Claude writes a **shot spec** — typed JSON, not prose — which
|
||
[shortsmith](https://git.chemavx.xyz/chemavx/shortsmith) renders into a 1080×1920
|
||
MP4. The bot sends the video and, in a separate message, a **claims report**.
|
||
|
||
```
|
||
/research JAL 1628 Alaska 1986 …
|
||
/generate blog en → Ghost draft, article URL stored on the output row
|
||
/generate short_en → spec → grounding → render → video + claims report
|
||
(YouTube upload is deliberately manual)
|
||
```
|
||
|
||
Three things make this different from generating text, and each has its own
|
||
mitigation:
|
||
|
||
- **It is a contract, not prose.** The template schemas are fetched live from
|
||
`GET /templates` and never copied here, so a template added to shortsmith is
|
||
available immediately. A spec is validated locally against those schemas
|
||
before anything renders, and the exact error paths
|
||
(`shots.0.radar_sweep.props.sweeeps`) go back to the model verbatim — up to 3
|
||
attempts.
|
||
- **It contains figures and quotes.** `grounding.py` extracts every quote,
|
||
figure, date and proper noun and checks it against the exact chunks the model
|
||
was given. No LLM in that path: normalisation plus substring, deterministic
|
||
and free. Whatever is not in the chunks is checked against the worked example
|
||
that travels in the prompt, so a figure lifted from it is reported as a
|
||
**prompt leak**, not as an invention — different diagnosis, different fix.
|
||
Neither ever blocks the render: both are surfaced next to the video and a
|
||
human decides.
|
||
- **It becomes a published video.** Nothing is uploaded anywhere. The MP4 lands
|
||
in Telegram for review, and in `/data/shorts/{session_id}.mp4`.
|
||
|
||
Fallbacks hold throughout: if shortsmith is unreachable, the job errors, or the
|
||
spec never validates, the spec JSON comes back as a file. The expensive part is
|
||
the generation, not the render.
|
||
|
||
Full spec of the phase: `docs/shortsmith-phase2-spec.md`.
|
||
|
||
## Local Development
|
||
|
||
```bash
|
||
# 1. Clone and setup
|
||
git clone https://git.chemavx.xyz/chemavx/researchowl
|
||
cd researchowl
|
||
|
||
# 2. Create virtualenv
|
||
python3 -m venv venv && source venv/bin/activate
|
||
pip install -r requirements.txt
|
||
|
||
# 3. Configure
|
||
cp .env.example .env
|
||
# Edit .env with your values
|
||
|
||
# 4. Run
|
||
python main.py
|
||
```
|
||
|
||
## Deploy to k3s
|
||
|
||
```bash
|
||
# 1. Create namespace and secrets
|
||
kubectl create namespace researchowl
|
||
kubectl create secret generic researchowl-secrets \
|
||
--from-literal=telegram-bot-token=YOUR_TOKEN \
|
||
--from-literal=telegram-allowed-users=YOUR_USER_ID \
|
||
-n researchowl
|
||
|
||
# 2. Copy manifests to your k8s-manifests repo
|
||
cp k8s/*.yaml /path/to/k8s-manifests/researchowl/
|
||
|
||
# 3. Apply ArgoCD app
|
||
kubectl apply -f k8s/argocd-app.yaml
|
||
|
||
# 4. Push to Gitea → Gitea Actions builds → ArgoCD deploys
|
||
git add . && git commit -m "feat: add researchowl" && git push
|
||
```
|
||
|
||
## Tuning
|
||
|
||
| Variable | Default | Description |
|
||
|----------|---------|-------------|
|
||
| `MAX_SOURCES` | 150 | Hard cap on sources |
|
||
| `MAX_DEPTH` | 3 | Link recursion depth |
|
||
| `QUALITY_THRESHOLD` | 0.4 | Min chunk quality (0-1) |
|
||
| `REQUEST_DELAY` | 1.0s | Delay between requests |
|
||
|
||
**Want more thoroughness?**
|
||
- Increase `MAX_SOURCES` to 300+
|
||
- Increase `MAX_DEPTH` to 4-5
|
||
- Lower `QUALITY_THRESHOLD` to 0.3
|
||
|
||
**Want faster results?**
|
||
- Lower `MAX_SOURCES` to 50
|
||
- Set `MAX_DEPTH` to 1-2
|
||
- Higher `QUALITY_THRESHOLD` to 0.6
|
||
|
||
## Bot avatar
|
||
|
||
The profile picture of `@chemavx_researchowl_bot` is not an opaque binary
|
||
checked into the repo: `assets/make_avatar.py` draws it with PIL at 4× and
|
||
scales it down, so the emblem can be retouched without hunting for an original.
|
||
Telegram crops avatars to a **circle**, so everything that matters lives inside
|
||
the inscribed circle; verified legible at 48 px.
|
||
|
||
```bash
|
||
python3 assets/make_avatar.py # writes assets/avatar.png
|
||
```
|
||
|
||
It is applied **over the API with the token from the secret, no BotFather**.
|
||
Watch out for `setMyProfilePhoto`: its `photo` parameter is not the file, it is
|
||
an `InputProfilePhoto` object pointing at the attachment. Posting the file on
|
||
its own gets you a baffling `photo isn't specified`.
|
||
|
||
```bash
|
||
TOK=$(kubectl get secret researchowl-secrets-infisical -n researchowl \
|
||
-o jsonpath='{.data.telegram-bot-token}' | base64 -d)
|
||
curl -s -F 'photo={"type":"static","photo":"attach://av"}' \
|
||
-F "av=@assets/avatar.png" \
|
||
"https://api.telegram.org/bot$TOK/setMyProfilePhoto"
|
||
unset TOK
|
||
```
|
||
|
||
## Notes
|
||
|
||
- Uses **qwen2.5:7b** (scoring) and **bge-m3** (embeddings) on your existing Ollama — zero API cost
|
||
- Optionally add `ANTHROPIC_API_KEY` for Claude fallback on generation
|
||
- SQLite database stored in `/data/researchowl.db`
|
||
- All outputs saved to DB and available via `/outputs`
|