YouTube & Search Pipeline

Solstice implements a robust, multi-tier search and extraction system for YouTube content. The system handles Spotify link interception, YouTube search with multiple fallback strategies, stream URL extraction with Redis caching, and a self-refreshing proxy pool.

Package: internal/youtube/

File	Purpose
search.go	YouTube search with 3-tier fallback strategy + Spotify interception
ytdlp.go	Stream URL extraction via yt-dlp + video info + cookie validation
redis.go	Upstash Redis cache client (HTTP REST API)
proxypool.go	Auto-refreshing HTTP proxy pool for search scraping

---

Search Pipeline

When a user sends /play <query>, the search follows a 3-tier fallback strategy:

User Query
    │
    ├── Spotify link? ──▶ Resolve via oEmbed API ──▶ "Title Artist audio"
    │
    ▼
┌─── Step 1: Native HTML Scrape ─────────────────────────┐
│ Direct GET to youtube.com/results?search_query=...     │
│ Parse ytInitialData JSON from HTML                     │
│ Extract videoRenderer objects                          │
│ + Fastest (no subprocess, no proxy)                    │
│ - Fails if YouTube rate-limits or returns captcha      │
└────────────────────────────────────────────────────────┘
    │ fails?
    ▼
┌─── Step 2: Proxy HTML Scrape ──────────────────────────┐
│ Same HTML scraping, but routed through a random proxy  │
│ Proxy is selected from GlobalProxyPool                 │
│ + Bypasses IP-based rate limiting                      │
│ - Proxy may be slow, dead, or blocked                  │
└────────────────────────────────────────────────────────┘
    │ fails?
    ▼
┌─── Step 3: yt-dlp Native Search ──────────────────────┐
│ Shells out to: python3 -m yt_dlp "ytsearchN:query"    │
│ Uses --dump-json --flat-playlist                      │
│ + Most reliable (yt-dlp handles all anti-bot logic)   │
│ - Slowest (subprocess overhead + yt-dlp startup)      │
└───────────────────────────────────────────────────────┘

Spotify Link Interception

Before any YouTube search, the query is checked for Spotify track links:

func InterceptSpotify(query string) string {
    if !strings.Contains(query, "open.spotify.com/track") {
        return query  // Not a Spotify link, pass through
    }
    // Hit Spotify's free oEmbed API (no API key needed)
    // GET https://open.spotify.com/oembed?url=<spotify_url>
    // Extract: { "title": "Song Name", "author_name": "Artist" }
    // Return: "Song Name Artist audio"
}

This converts Spotify links to YouTube-searchable queries without requiring Spotify API credentials.

HTML Scraping (SearchHTML)

The native scraper:

1. Sends a GET request to youtube.com/results?search_query=<query>
2. Mimics a Chrome browser via User-Agent header
3. Extracts ytInitialData JSON from the page using regex
4. Navigates the JSON tree to find videoRenderer objects
5. Extracts: videoId, title, lengthText

// JSON path to video results:
contentsPath := "contents.twoColumnSearchResultsRenderer.primaryContents" +
    ".sectionListRenderer.contents.0.itemSectionRenderer.contents"

Search Result Structure

type SearchResult struct {
    ID        string  // YouTube video ID (e.g., "dQw4w9WgXcQ")
    Title     string  // Video title
    Time      string  // Duration (e.g., "3:45")
    Link      string  // Full YouTube URL
    Thumbnail string  // Thumbnail URL (hqdefault.jpg)
}

---

Stream URL Extraction

File: ytdlp.go

Once a video is selected, GetStreamURL() extracts the direct playable stream URL.

Format Selection

Mode	yt-dlp Format String	Priority
Audio	251/250/bestaudio[ext=m4a]/bestaudio	Opus → M4A → any
Video	22/18/best[ext=mp4]	720p MP4 → 360p MP4 → any

Extraction Flow

GetStreamURL(videoID, isVideo)
    │
    ├── 1. Check Redis Cache
    │   └── Key: "cache:audio:<videoID>" or "cache:video:<videoID>"
    │   └── Hit? Return cached URL immediately
    │
    ├── 2. Run yt-dlp with cookies
    │   └── python3 -m yt_dlp --remote-components ejs:github
    │       --js-runtimes node -f <format> -g --cookies cookies.txt <url>
    │   └── Success? Cache and return
    │
    ├── 3. Retry without cookies (anonymous)
    │   └── Same command but without --cookies flag
    │   └── Success? Cache and return
    │
    └── 4. Return error

Cookie Fallback Logic

// If extraction with cookies fails, strip cookie args and retry anonymously
if hasCookies {
    log.Printf("[YT-DLP] Cookie extraction failed, retrying anonymously...")
    // Remove --cookies/--cookies-from-browser args
    // Retry with clean args
}

This ensures the bot remains functional even when cookies expire.

yt-dlp Flags

Flag	Purpose
--remote-components ejs:github	Load enhanced JavaScript extractor from GitHub
--js-runtimes node	Use Node.js for JavaScript execution
-f <format>	Format selection (audio or video)
-g	Output only the stream URL (no download)
--no-playlist	Don't expand playlists
--cookies cookies.txt	Use saved YouTube cookies
--dump-json	Output JSON metadata (for search/info)
--flat-playlist	Don't resolve individual videos in playlists
--simulate	Don't download anything (for cookie checking)

---

Redis Caching

File: redis.go

Stream URLs are cached in Upstash Redis (serverless Redis accessed via HTTP REST API) to avoid repeated yt-dlp extractions.

Smart TTL

YouTube stream URLs contain an expire parameter. The cache TTL is calculated dynamically:

expire := ExtractExpire(streamURL)  // Parse ?expire=<unix_timestamp> from URL
ttl := expire - int(time.Now().Unix()) - 15  // 15-second safety buffer

This means cached URLs are evicted just before they actually expire, maximizing cache hit rate without serving stale URLs.

Cache Key Format

cache:audio:<videoID>   →  direct stream URL (audio)
cache:video:<videoID>   →  direct stream URL (video)

API Protocol

The Redis client uses Upstash's REST API (not the Redis protocol):

// SET
POST /
Authorization: Bearer <token>
Body: ["SET", "cache:audio:dQw4w9WgXcQ", "<stream_url>", "EX", 21585]

// GET
GET /get/cache:audio:dQw4w9WgXcQ
Authorization: Bearer <token>
Response: { "result": "<stream_url>" }

If UPSTASH_REDIS_REST_URL or UPSTASH_REDIS_REST_TOKEN are not set, caching is silently disabled.

---

Proxy Pool

File: proxypool.go

The ProxyPool provides free HTTP proxies for YouTube search scraping when direct access is rate-limited.

How It Works

type ProxyPool struct {
    proxies    []string      // IP:PORT list
    lastUpdate time.Time
    mu         sync.RWMutex
}

var GlobalProxyPool = &ProxyPool{}

Refresh Logic

• Refreshes every 15 minutes (or when proxy count drops below 5)
• Fetches from 3 public proxy sources:
  • api.proxyscrape.com - ProxyScrape API
  • clarketm/proxy-list - GitHub raw list
  • monosans/proxy-list - GitHub raw list
• Deduplicates, shuffles, and caps at 100 proxies
• Thread-safe via sync.RWMutex

Usage in Search

// In SearchVideo():
GlobalProxyPool.Refresh()
proxyURL := GlobalProxyPool.GetRandom()  // Random selection

if proxyURL != "" {
    results, err = SearchHTML(query, limit, proxyURL)
}

---

Related Videos (Autoplay)

File: internal/telegram/bot.go → findNextAutoplaySong() & search.go → GetRelatedVideos()

When autoplay is enabled and the queue empties, Solstice fetches related videos using a Breadth-First Search (BFS) algorithm to guarantee a unique, unplayed song is always found.

The Autoplay BFS Algorithm

To prevent the bot from playing songs that have already been played recently, it maintains a rolling cache of the last 50 songs. If the initial YouTube recommendations have all been played, the bot dynamically traverses down the recommendation tree up to 4 levels deep.

Current Song Finishes
 └── Fetch Related (6 Recommendations)
      ├── Song 1 (Played) ──> Added to search queue
      ├── Song 2 (Played) ──> Added to search queue
      └── Song 3 (Played) ──> Added to search queue

[Queue processing]
 └── Pop Song 1 ──> Fetch Related (6 Recommendations)
      ├── Song 1.1 (Played)
      └── Song 1.2 (Played)

 └── Pop Song 2 ──> Fetch Related (6 Recommendations)
      ├── Song 2.1 (Unplayed!) ──> SELECT & PLAY
      └── Song 2.2 (Unplayed)

1. Initial Fetch: Gets 6 recommendations for the just-finished song.
2. Duplicate Check: Checks each recommendation against the 50-song Redis history.
3. Queueing (BFS): If all 6 songs are already played, they are added to a search queue.
4. Deep Search: It pops the first song from the queue, fetches its 6 recommendations, and checks them. It repeats this process until a unique song is found.
5. Safety Limit: The search stops after 4 API calls (checking ~24 songs) to ensure fast response times. If absolutely no unique song is found, it falls back to the top recommendation.

func GetRelatedVideos(videoID string, limit int) ([]SearchResult, error) {
    // Uses YouTube's "Radio" (Mix) playlist: RD<videoID>
    radioURL := fmt.Sprintf(
        "https://www.youtube.com/watch?v=%s&list=RD%s",
        videoID, videoID,
    )
    // Skip the first track (the song itself)
    playlistItems := fmt.Sprintf("2-%d", limit+1)

    // Shell out to yt-dlp with --flat-playlist
    args := []string{"-m", "yt_dlp", radioURL,
        "--dump-json", "--flat-playlist",
        "--playlist-items", playlistItems}
}

This leverages YouTube's algorithmic "Radio" mix to find contextually similar songs, paired with our BFS logic to guarantee fresh playback.

---

Video Info Extraction

GetVideoInfo() retrieves metadata for a specific video without downloading:

type VideoInfo struct {
    ID        string  // YouTube video ID
    Title     string  // Full title
    Duration  string  // Formatted "M:SS"
    Thumbnail string  // Thumbnail URL
    Link      string  // Full YouTube URL
}

Used for displaying "Now Playing" cards with correct metadata.

---

Cookie Management

YouTube cookies are critical for avoiding rate limiting and bot detection.

Cookie Sources (Priority Order)

1. MongoDB - database.GetYoutubeCookies() - highest priority, persists across restarts
2. Environment - YOUTUBE_COOKIES env var - saved to MongoDB on first load
3. Disk - cookies.txt file - legacy fallback

Cookie Filtering

When cookies are set (via env var or /setcookies), they're filtered to keep only YouTube/Google domains to minimize storage:

for _, line := range lines {
    if strings.HasPrefix(trimmed, "#") ||
       strings.Contains(trimmed, "youtube.com") ||
       strings.Contains(trimmed, "google.com") {
        filteredLines = append(filteredLines, line)
    }
}

Cookie Validation

/checkcookies runs a lightweight validation:

func CheckCookies() error {
    // Try to extract metadata for Rick Astley - Never Gonna Give You Up
    // --simulate flag = no download, just test auth
    // If stderr contains "Sign in to confirm you're not a bot":
    //   → cookies are invalid/expired
}