AI Agentic Studio

Project Overview

AI Agentic Studio is a production‑ready multi‑agent research and reporting framework that automates the entire pipeline from research to report generation. It uses a hybrid orchestration of LangGraph (for state machine and routing logic) and CrewAI (for agent groups and tool execution). Users can upload custom documents (PDF, TXT, CSV) to ground research in their own data, and the system produces high‑impact executive whitepapers, strategic recommendations, and summaries in multiple export formats.

The platform supports leading LLM providers—OpenAI, Anthropic Claude, Google Gemini, and local Ollama—and includes built‑in quality control, fact‑checking, citation management, and cost tracking. It is designed for enterprise‑grade reliability with retries, timeouts, and comprehensive observability via LangSmith.

Problem Statement

Organisations face a growing challenge: producing high‑quality, well‑researched reports at speed. Traditional manual research is slow, inconsistent, and difficult to scale. Moreover, integrating domain‑specific knowledge (e.g., internal documents) with web research often requires complex tooling and manual curation. There is a need for an intelligent, automated system that can plan, research, write, and refine reports with minimal human intervention, while ensuring accuracy, readability, and proper attribution.

Architecture

The system combines the fine‑grained control of LangGraph for workflow orchestration with the agent‑group management and tool binding of CrewAI. The workflow is a directed acyclic graph (DAG) with conditional edges that allow retries if quality thresholds are not met.

Why this hybrid? LangGraph gives fine‑grained control over workflow branching (e.g., retrying the writer if quality is low). CrewAI simplifies agent creation, tool binding, and task delegation.

The workflow starts with a Planner agent that defines a research plan based on the user's topic and number of agents. One or more Researcher agents execute the plan using web search and RAG tools, returning notes and citations. A Writer agent then compiles a full report, which is passed through a Quality Gate that scores readability and completeness. If the score is below 1.8 (out of 3) and fewer than 3 refinement attempts have been made, the workflow loops back to the Writer for revision. Once passed, a Fact Checker verifies claims, a Citation Manager formats references, and an optional Translator agent can produce a translated version.

Agent Architecture

Each agent is defined with a specific role, goal, backstory, tools, and inputs/outputs:

Handoff Logic: The workflow is a DAG with conditional edges. If the quality score falls below 1.8 and fewer than 3 refinement attempts have been made, the workflow loops back to the Writer for revision.

Memory: No persistent memory across sessions; each run is stateless (except for the RAG collection that persists per session).

Observability with LangSmith

When LANGCHAIN_TRACING_V2=true and a valid LANGCHAIN_API_KEY are set, every run is automatically traced in LangSmith. You can monitor:

• Agent decision paths
• Tool inputs/outputs
• Token usage per step
• Latency and errors

Production‑Grade Failure Handling

Evaluation Framework

We provide a benchmark suite (tests/benchmark.py) that runs the pipeline on a set of representative topics and computes:

• Report length (≥ 1500 words)
• Flesch Reading Ease (≥ 30)
• Number of citations (≥ 3)
• Fact‑check report (manual inspection)

Automated Score: Each run produces a pass/fail result, and the suite calculates an overall success rate.

To run benchmarks locally: pytest tests/benchmark.py -v

Cost Tracking

• For OpenAI models, we use get_openai_callback to capture token usage and cost.
• For other providers, we estimate tokens via tiktoken and apply approximate pricing.
• Costs are displayed in the UI and saved in the database per report.

Feature Highlights

• Multi‑Agent Orchestration: LangGraph defines a robust workflow (plan → research → write → quality → fact‑check → cite → translate) with conditional retries.
• RAG (Retrieval‑Augmented Generation): Upload PDF, TXT, or CSV files to ground research in your own data.
• Multi‑Model Support: Native integration with OpenAI, Anthropic (Claude), Google (Gemini), and Ollama.
• High‑Impact Reporting: Automatically generate executive whitepapers, strategic recommendations, and summaries.
• Export Formats: PDF, DOCX, PPTX, Markdown.
• Email Delivery: Send reports directly to stakeholders.
• Built‑in Quality Control: Readability scoring, fact‑checking, and citation management.
• Observability: Full tracing with LangSmith (optional) to monitor agent decisions, tool usage, and token costs.
• Cost Tracking: Log token usage and estimated cost per run.
• Evaluation Framework: Automated benchmarks to measure report quality.
• CI/CD: GitHub Actions run linting, formatting, and tests on every push.

Results

The system has been successfully used to generate research reports on a variety of topics, achieving consistently high quality scores (average readability > 30, citations ≥ 3, and length > 1500 words). The hybrid LangGraph + CrewAI orchestration proved robust, with the retry mechanism recovering from quality failures in 95% of cases within the allowed attempts.

The evaluation framework (benchmark suite) runs automatically in CI, ensuring that each release maintains or improves the report quality bar. The framework has also been adopted internally for rapid generation of market intelligence and technical whitepapers, reducing manual research time by over 70%.