v0

1 files changed, 166 insertions, 0 deletions

diff --git a/CLAUDE.md b/CLAUDE.md
new file mode 100644
index 0000000..b803720
--- /dev/null
+++ b/CLAUDE.md
@@ -0,0 +1,166 @@
+# CLAUDE.md - Technical Notes for LLM Council
+
+This file contains technical details, architectural decisions, and important implementation notes for future development sessions.
+
+## Project Overview
+
+LLM Council is a 3-stage deliberation system where multiple LLMs collaboratively answer user questions. The key innovation is anonymized peer review in Stage 2, preventing models from playing favorites.
+
+## Architecture
+
+### Backend Structure (`backend/`)
+
+**`config.py`**
+- Contains `COUNCIL_MODELS` (list of OpenRouter model identifiers)
+- Contains `CHAIRMAN_MODEL` (model that synthesizes final answer)
+- Uses environment variable `OPENROUTER_API_KEY` from `.env`
+- Backend runs on **port 8001** (NOT 8000 - user had another app on 8000)
+
+**`openrouter.py`**
+- `query_model()`: Single async model query
+- `query_models_parallel()`: Parallel queries using `asyncio.gather()`
+- Returns dict with 'content' and optional 'reasoning_details'
+- Graceful degradation: returns None on failure, continues with successful responses
+
+**`council.py`** - The Core Logic
+- `stage1_collect_responses()`: Parallel queries to all council models
+- `stage2_collect_rankings()`:
+  - Anonymizes responses as "Response A, B, C, etc."
+  - Creates `label_to_model` mapping for de-anonymization
+  - Prompts models to evaluate and rank (with strict format requirements)
+  - Returns tuple: (rankings_list, label_to_model_dict)
+  - Each ranking includes both raw text and `parsed_ranking` list
+- `stage3_synthesize_final()`: Chairman synthesizes from all responses + rankings
+- `parse_ranking_from_text()`: Extracts "FINAL RANKING:" section, handles both numbered lists and plain format
+- `calculate_aggregate_rankings()`: Computes average rank position across all peer evaluations
+
+**`storage.py`**
+- JSON-based conversation storage in `data/conversations/`
+- Each conversation: `{id, created_at, messages[]}`
+- Assistant messages contain: `{role, stage1, stage2, stage3}`
+- Note: metadata (label_to_model, aggregate_rankings) is NOT persisted to storage, only returned via API
+
+**`main.py`**
+- FastAPI app with CORS enabled for localhost:5173 and localhost:3000
+- POST `/api/conversations/{id}/message` returns metadata in addition to stages
+- Metadata includes: label_to_model mapping and aggregate_rankings
+
+### Frontend Structure (`frontend/src/`)
+
+**`App.jsx`**
+- Main orchestration: manages conversations list and current conversation
+- Handles message sending and metadata storage
+- Important: metadata is stored in the UI state for display but not persisted to backend JSON
+
+**`components/ChatInterface.jsx`**
+- Multiline textarea (3 rows, resizable)
+- Enter to send, Shift+Enter for new line
+- User messages wrapped in markdown-content class for padding
+
+**`components/Stage1.jsx`**
+- Tab view of individual model responses
+- ReactMarkdown rendering with markdown-content wrapper
+
+**`components/Stage2.jsx`**
+- **Critical Feature**: Tab view showing RAW evaluation text from each model
+- De-anonymization happens CLIENT-SIDE for display (models receive anonymous labels)
+- Shows "Extracted Ranking" below each evaluation so users can validate parsing
+- Aggregate rankings shown with average position and vote count
+- Explanatory text clarifies that boldface model names are for readability only
+
+**`components/Stage3.jsx`**
+- Final synthesized answer from chairman
+- Green-tinted background (#f0fff0) to highlight conclusion
+
+**Styling (`*.css`)**
+- Light mode theme (not dark mode)
+- Primary color: #4a90e2 (blue)
+- Global markdown styling in `index.css` with `.markdown-content` class
+- 12px padding on all markdown content to prevent cluttered appearance
+
+## Key Design Decisions
+
+### Stage 2 Prompt Format
+The Stage 2 prompt is very specific to ensure parseable output:
+```
+1. Evaluate each response individually first
+2. Provide "FINAL RANKING:" header
+3. Numbered list format: "1. Response C", "2. Response A", etc.
+4. No additional text after ranking section
+```
+
+This strict format allows reliable parsing while still getting thoughtful evaluations.
+
+### De-anonymization Strategy
+- Models receive: "Response A", "Response B", etc.
+- Backend creates mapping: `{"Response A": "openai/gpt-5.1", ...}`
+- Frontend displays model names in **bold** for readability
+- Users see explanation that original evaluation used anonymous labels
+- This prevents bias while maintaining transparency
+
+### Error Handling Philosophy
+- Continue with successful responses if some models fail (graceful degradation)
+- Never fail the entire request due to single model failure
+- Log errors but don't expose to user unless all models fail
+
+### UI/UX Transparency
+- All raw outputs are inspectable via tabs
+- Parsed rankings shown below raw text for validation
+- Users can verify system's interpretation of model outputs
+- This builds trust and allows debugging of edge cases
+
+## Important Implementation Details
+
+### Relative Imports
+All backend modules use relative imports (e.g., `from .config import ...`) not absolute imports. This is critical for Python's module system to work correctly when running as `python -m backend.main`.
+
+### Port Configuration
+- Backend: 8001 (changed from 8000 to avoid conflict)
+- Frontend: 5173 (Vite default)
+- Update both `backend/main.py` and `frontend/src/api.js` if changing
+
+### Markdown Rendering
+All ReactMarkdown components must be wrapped in `<div className="markdown-content">` for proper spacing. This class is defined globally in `index.css`.
+
+### Model Configuration
+Models are hardcoded in `backend/config.py`. Chairman can be same or different from council members. The current default is Gemini as chairman per user preference.
+
+## Common Gotchas
+
+1. **Module Import Errors**: Always run backend as `python -m backend.main` from project root, not from backend directory
+2. **CORS Issues**: Frontend must match allowed origins in `main.py` CORS middleware
+3. **Ranking Parse Failures**: If models don't follow format, fallback regex extracts any "Response X" patterns in order
+4. **Missing Metadata**: Metadata is ephemeral (not persisted), only available in API responses
+
+## Future Enhancement Ideas
+
+- Configurable council/chairman via UI instead of config file
+- Streaming responses instead of batch loading
+- Export conversations to markdown/PDF
+- Model performance analytics over time
+- Custom ranking criteria (not just accuracy/insight)
+- Support for reasoning models (o1, etc.) with special handling
+
+## Testing Notes
+
+Use `test_openrouter.py` to verify API connectivity and test different model identifiers before adding to council. The script tests both streaming and non-streaming modes.
+
+## Data Flow Summary
+
+```
+User Query
+    ↓
+Stage 1: Parallel queries → [individual responses]
+    ↓
+Stage 2: Anonymize → Parallel ranking queries → [evaluations + parsed rankings]
+    ↓
+Aggregate Rankings Calculation → [sorted by avg position]
+    ↓
+Stage 3: Chairman synthesis with full context
+    ↓
+Return: {stage1, stage2, stage3, metadata}
+    ↓
+Frontend: Display with tabs + validation UI
+```
+
+The entire flow is async/parallel where possible to minimize latency.