Pipeline flowchart
Step details
1. Parse & Validate URL
1. Parse & Validate URL
Purpose: Extract and normalize the input (URL, ID, or batch file) into a structured
ParsedURL object.Accepted formats:- Full YouTube watch URL (
https://youtube.com/watch?v=...) - Short URL (
https://youtu.be/...) - Embed URL or Shorts URL
- Bare 11-character video ID
- Bare playlist ID (
PL,UU,LL,FL,RD,UL,WL,OLAK5uy_) - Playlist URL (
?list=...) - Path to a
.txtbatch file (one URL per line)
ParsedURL with url_type ("video" or "playlist"), video_id, and/or playlist_id.Module:
youtube/parser.py → parse_youtube_url()2. Fetch Video Metadata
2. Fetch Video Metadata
Module:
youtube/metadata.py → get_video_metadata()Retrieves: Title, duration (seconds), and chapter list.Early-exit checks:- Cache hit — if video ID is in SQLite cache and
--forceis not set → emitVIDEO_SKIPPED - Output path check — if notes already exist in output directory and
--forceis not set → skip
3. Fetch Transcript
3. Fetch Transcript
Module:
youtube/transcript.py → fetch_transcript()Language preference: Tries languages in order (default: ["en"]), falling back if unavailable. Uses cookie file if configured.Retries: Up to 3 attempts with backoff on network errors.Returns: VideoTranscript with list of TranscriptSegment (text, start time, duration).4. Chunk & Generate
4. Chunk & Generate
Module:
pipeline/generation.py → StudyMaterialGeneratorToken counting uses LiteLLM’s token_counter. If transcript exceeds 4,000 tokens, it’s split into overlapping 200-token chunks.Chapter-level generation
Chapter-level generation
Activated when video has chapters and duration exceeds 3,600 seconds (1 hour).
- Chapters processed concurrently (up to 3 at a time)
- Each chapter independently chunked and reduced to Markdown
Flat generation
Flat generation
Used for short videos or videos without chapter data.
- Single-pass for small transcripts
- Multi-chunk with combine step for large ones
5. Write Outputs
5. Write Outputs
Module:
pipeline/_artifacts.py| Condition | Output |
|---|---|
| Standard video | <OUTPUT_DIR>/<sanitized title>.md |
| Chapter-aware video | <OUTPUT_DIR>/<title>/01_<chapter>.md, 02_<chapter>.md, … |
--quiz | <title>_quiz.md alongside notes |
--export-transcript txt | <title>_transcript.txt |
--export-transcript json | <title>_transcript.json |
6. Persist to Cache
6. Persist to Cache
Module:
storage/repository.py → DatabaseRepository.aupsert_video_cache()Fields written:- Video metadata (id, title, duration)
- Raw transcript text and language
- Token usage (prompt, completion, total)
- Cost estimate (USD)
- Timing (transcript fetch, generation seconds)
- Model name
stats / history commands.7. Events & Result
7. Events & Result
PipelineEvent objects emitted via on_event callback throughout. CLI subscribes via PipelineDashboard for live UI updates.PipelineResult returns: success_count, failure_count, total_count, errors, and metrics.See Pipeline Events for full event reference.Concurrency Model
notewise usesasyncio throughout.
| Level | Default | Config |
|---|---|---|
| Video-level | 5 concurrent videos | MAX_CONCURRENT_VIDEOS |
| Chapter-level | 3 concurrent chapters | Code default only |
| YouTube requests | 10/minute | YOUTUBE_REQUESTS_PER_MINUTE |
MAX_CONCURRENT_VIDEOS. Within a single long video with chapters, chapters are processed concurrently up to DEFAULT_MAX_CONCURRENT_CHAPTERS.
A PipelineSharedState object is passed into CorePipeline during batch runs so all video instances share the same semaphores.