AI Trend Discovery & Analysis Platform

Built for a client; presented anonymized — company name and logo removed, source private. (“TrendForge” is a working name.) This documents the architecture and engineering.

The problem

Teams that need to stay ahead of fast-moving topics — AI, tech, content — drown in scattered signals: papers, repos, forums, social posts, product launches. Reading it all is impossible; doing it manually is slow and inconsistent.

TrendForge automates the whole loop: it continuously pulls signals from many sources, de-duplicates and cleans them, uses LLMs to score and cluster what matters, and produces ranked topic picks and structured reports a team can act on — in minutes, for cents.

What it does

Two analysis modes on one pipeline:

• Precise Search — query by keyword + platform + audience persona; get the top content, AI-selected topic picks, and a structured multi-chapter report.
• AI Topic Discovery — start from a seed term (e.g. “LLM agents”); the system expands it into search terms, pulls from global sources, and clusters everything into named, summarized topics.

The full loop: collect → clean → score → select topics → generate report → export (Markdown / Excel). Background workers keep collecting around the clock, and the system periodically generates aggregated, deduplicated topics with new/continuing/merged tracking across runs.

LLM engineering highlights

This is where the real work is — production LLM infrastructure, not a thin API wrapper:

• Custom LLM gateway — provider abstraction and routing across multiple models, with a two-tier cache (Redis L1 + PostgreSQL L2), per-budget cost controls, and full usage tracking. In practice it analyzes hundreds of items at a tiny fraction of typical LLM cost.
• Semantic topic clustering — embeddings → UMAP dimensionality reduction → HDBSCAN clustering → LLM-generated topic names, summaries, and off-topic detection, with graceful fallbacks when a model call fails.
• 7-dimension scoring engine — combines 3 deterministic numeric signals with 4 LLM-judged dimensions, visualized as radar charts, with configurable weights and risk thresholds.
• Resilient multi-source collection — 6 global sources pulled in parallel (arXiv, Hugging Face, GitHub Trending, Hacker News, big-tech blog RSS, Product Hunt); single-source failures never break the run, with backoff and smart handling of transient connection drops.

Architecture & tech

Backend — FastAPI + Celery + SQLAlchemy 2.0 (async), PostgreSQL, Redis. Background workers, a beat scheduler, and a collection worker run the pipeline asynchronously.

Frontend — Next.js (App Router) + Tailwind + shadcn/ui. Live progress via Server-Sent Events; dashboards for content streams, topic clusters, scoring radar, and LLM usage/budget curves.

Infrastructure — Dockerized 6-service stack via Docker Compose, single-port reverse proxy, encrypted credential storage (Fernet), JWT auth + admin panels, Alembic migrations with automatic seeding.

Quality — 370+ passing backend tests; end-to-end smoke tests validate the full pipeline.

What it demonstrates

LLM features that stay affordable at scale (provider routing + caching + cost controls), RAG and semantic clustering over real data, self-running AI pipelines, and full-stack delivery from FastAPI/Next.js to Dockerized deployment.