TwitchDropsMiner/AGENTS.md

AGENTS.md

AGENTS.md Specific Instructions

This file provides guidance to AI Agents when working with code in this repository.

Development Guidelines

1. Testing:

   • Always add unit tests for backend changes.
   • Frontend changes should have tests if possible.

2. Code Style & Architecture:

   • DRY (Don't Repeat Yourself): Codebase must follow DRY principle.
   • OOP (Object-Oriented Programming): Required for all backend code.

3. Refactoring:

   • You are authorized to refactor code to align with DRY/OOP principles.
   • Permission Required: You MUST ask for user permission before significant refactoring.

4. Localization (i18n):

   • Update translation files if there are changes to UI text or console messages.

5. Documentation:

   • Always update README.md and all agent instruction files when making changes.
   • The contents of all agent instruction files should be identical except for the Specific Instructions section. Any agent-specific instructions must be added to that section.

Project Overview

Twitch Drops Miner is a Python application that automatically mines timed Twitch drops without downloading stream data. It uses Twitch's GraphQL API and websocket connections to simulate watching streams while tracking drop progress.

Key Characteristics:

• Python 3.12+ required
• Web-based GUI using FastAPI and Socket.IO
• Async/await architecture with asyncio
• Session persistence via cookies
• No stream video/audio download (bandwidth-efficient)
• Docker-ready for easy deployment

Architecture

The application now uses a clean src/ package structure with clear separation of concerns.

Project Structure

src/
├── models/          # Domain models (Game, Channel, Campaign, Drop, Benefit)
├── config/          # Configuration (constants, paths, operations, settings, client_info)
├── utils/           # Pure utilities (string, JSON, async helpers, rate_limiter, backoff)
├── i18n/            # Translation system (Translator class, TypedDict schemas)
├── auth/            # Authentication (auth_state for OAuth and token management)
├── api/             # External API (HTTP client, GraphQL client)
├── websocket/       # Real-time updates (websocket connection, pool)
├── web/             # Web GUI (app, gui_manager, api/)
│   └── managers/    # Individual UI managers (status, console, channels, campaigns, inventory, login, settings, cache, broadcaster)
├── services/        # Business logic services (channel, inventory, watch, maintenance, message_handlers)
├── core/            # Core client (Twitch client)
├── exceptions.py    # Custom exceptions
├── version.py       # Version string
└── __main__.py      # Entry point

lang/                # Translation JSON files (19 languages)
├── English.json     # Default/fallback translations
├── Español.json
├── Français.json
├── Deutsch.json
└── ...              # 15 more languages

Core Components

main.py - Simple launcher:

• Runs the src package as a module using runpy.run_module("src")
• All application logic is now in src/__main__.py

src/main.py - Entry point:

• Parses command-line arguments
• Initializes Settings, Twitch client, and WebGUIManager
• Starts the FastAPI web server (uvicorn on port 8080)
• Runs the main asyncio event loop
• Handles signals (SIGINT, SIGTERM on Linux) and exit codes

src/core/client.py - Central client (Twitch class):

• State machine: IDLE, INVENTORY_FETCH, GAMES_UPDATE, CHANNELS_CLEANUP, CHANNELS_FETCH, CHANNEL_SWITCH, EXIT
• Composes _AuthState, HTTPClient, and GQLClient
• Delegates to service layer for business logic
• Drop progress monitoring via periodic "watch" payloads
• Manages WebsocketPool and maintenance tasks

src/services/ - Business logic layer (fully implemented):

• ChannelService: Channel management and selection logic
• InventoryService: Campaign and drop inventory operations
• WatchService: Drop mining watch payload logic
• MaintenanceService: Periodic maintenance tasks
• MessageHandlerService: Websocket message routing and handling

src/models/channel.py - Channel and Stream:

• Channel class: Twitch channel with online/offline status
• Stream class: Active stream with game, viewers, drop status
• Stream URL fetching and validation
• ACL-based vs directory channels

src/models/campaign.py - Drop campaigns:

• DropsCampaign: Campaign with game, timeframe, allowed channels
• Time-based eligibility and progress tracking

src/models/drop.py - Drop types:

• TimedDrop: Drops with minute requirements and progress
• BaseDrop: Base class with claim logic
• Precondition chains for sequential drops

src/web/gui_manager.py - Web GUI:

• WebGUIManager: Main GUI coordinator
• Composes individual managers for different UI concerns (status, console, channels, campaigns, inventory, login, settings, cache)
• Uses WebSocketBroadcaster for real-time Socket.IO updates
• Pure asyncio, no tkinter dependency

src/web/app.py - FastAPI application:

• REST API endpoints: /api/status, /api/channels, /api/campaigns, /api/settings, /api/login, /api/oauth/confirm, /api/reload, /api/close, /api/version
• Socket.IO server for real-time bi-directional communication
• Serves static web frontend from web/ directory
• Integrates with WebGUIManager via set_managers()

src/websocket/pool.py - WebSocket management:

• Sharded connections (up to 50 topics per socket, max 199 channels)
• Topics: User.Drops, User.Notifications, Channel.StreamState, Channel.StreamUpdate
• Automatic reconnection with exponential backoff
• Message routing to registered callbacks

src/config/settings.py - Application settings:

• Games to watch list (auto-populated from available campaigns if empty)
• Connection quality multiplier
• Language selection
• Proxy support (including verification)
• Logging and dump flags from command-line arguments
• Persistence to JSON file (settings.json) in DATA_DIR
• Inventory filters (Status, Benefit Type, Game Search)

State Machine Flow

1. IDLE - Waiting for campaigns or user action
2. INVENTORY_FETCH - Fetch campaigns from GraphQL, claim completed drops
3. GAMES_UPDATE - Determine wanted games based on priority/exclude lists
4. CHANNELS_CLEANUP - Remove channels not streaming wanted games
5. CHANNELS_FETCH - Discover channels via ACL lists or game directories
6. CHANNEL_SWITCH - Select best channel to watch based on priority/ACL
7. Loop between CHANNEL_SWITCH and periodic INVENTORY_FETCH (hourly)

Authentication

• Uses OAuth device code flow (user enters code at twitch.tv/activate)
• Managed by src/auth/auth_state.py (_AuthState class)
• Access tokens stored in cookies.jar in DATA_DIR
• Device ID from Twitch's unique_id cookie
• Session ID generated per run
• Client info defined in src/config/client_info.py (presents as Android app with Client-Id and User-Agent spoofing)

Drop Mining Mechanism

The application sends periodic "watch" payloads to a spade URL every ~20 seconds:

• Payload contains minute-watched events with channel/broadcast IDs
• Twitch reports progress via websocket (User.Drops topic)
• If websocket updates stop, fallback to GQL CurrentDrop query
• Extrapolation via "bump minutes" when no updates received

GraphQL Operations

Defined in src/config/operations.py as GQL_OPERATIONS:

• Inventory - Fetch in-progress campaigns and claimed benefits
• Campaigns - List available active/upcoming campaigns
• CampaignDetails - Detailed drop info for a campaign
• GameDirectory - Find live streams for a game with drops enabled
• GetStreamInfo - Check if channel is online and get stream details
• CurrentDrop - Query currently mined drop progress
• ClaimDrop - Claim a completed drop
• AvailableDrops - Check which campaigns a channel qualifies for (badge validation)
• NotificationsDelete - Delete Twitch notifications

Channel Selection Priority

1. Selected channel (if user clicked one)
2. ACL-based channels over directory channels
3. Game priority order (from settings)
4. Viewer count (descending)
5. Maximum 199 channels tracked simultaneously

Maintenance Task

Runs in background to trigger:

• Channel cleanup when drops start/end (based on time_triggers)
• Inventory reload every ~60 minutes

Translation System

Architecture:

• All translations stored as JSON files in lang/ directory (19 languages supported)
• English (lang/English.json) is the single source of truth and fallback language
• Strongly typed with TypedDict schema defined in src/i18n/translator.py
• Translator class (src/i18n/translator.py) handles language loading and fallback
• Singleton instance _ available via from src.i18n import _

Supported Languages:

• English, Dansk (Danish), Deutsch (German), Español (Spanish), Français (French)
• Indonesian, Italiano (Italian), Nederlandse (Dutch), Polski (Polish), Português (Portuguese)
• Română (Romanian), Türkçe (Turkish), Čeština (Czech)
• Русский (Russian), Українська (Ukrainian), العربية (Arabic)
• 日本語 (Japanese), 简体中文 (Simplified Chinese), 繁體中文 (Traditional Chinese)

Translation Structure:

Translation = {
    "language_name": str,      # Display name of language
    "english_name": str,       # English name of language
    "status": StatusMessages,  # Console status messages
    "login": LoginMessages,    # Login-related messages
    "error": ErrorMessages,    # Error messages
    "gui": GUIMessages        # All web GUI text (tabs, settings, help, etc.)
}

Usage:

from src.i18n import _

# Access translations
status_text = _.t["gui"]["status"]["idle"]  # Returns "Idle"
login_text = _.t["login"]["status"]["logged_in"]  # Returns "Logged in"

Language Persistence:

• Language selection persisted in settings.json (DATA_DIR)
• Dynamic language switching supported in web GUI
• Changes take effect immediately without restart

Key Files

• src/config/constants.py - Core enums (State, WebsocketTopic), logging config, type aliases
• src/config/operations.py - GraphQL operation definitions (GQL_OPERATIONS)
• src/config/paths.py - Path management and Docker environment detection
• src/config/client_info.py - Twitch client info (Client-Id, User-Agent)
• src/config/settings.py - Application settings with JSON persistence
• src/exceptions.py - Custom exceptions (MinerException, ExitRequest, RequestException, RequestInvalid, WebsocketClosed, LoginException, CaptchaRequired, GQLException)
• src/utils/ - Helper utilities (string_utils, json_utils, async_helpers, rate_limiter, backoff)
• src/i18n/ - Internationalization package with TypedDict schema and Translator class
  • translator.py - Translator class with typed translation schema (Translation TypedDict)
  • init.py - Exports translation types and _ (Translator instance)
• lang/ - Translation JSON files for 19 languages (English.json is the single source of truth)
• src/version.py - Version string
• src/web/app.py - FastAPI application with REST API and Socket.IO
• src/web/managers/cache.py - ImageCache for campaign artwork caching
• web/ - Frontend assets (index.html, static/app.js, static/styles.css)

Development Commands

IMPORTANT: Always activate the virtual environment first!

The project uses a virtual environment located at env/. All Python commands must be run within this environment:

# Activate the virtual environment (required before any Python commands)
source env/bin/activate

Running the Application

# Run from source (remember to activate venv first!)
source env/bin/activate && python main.py

# With verbose logging (stackable: -vv, -vvv)
source env/bin/activate && python main.py -v

# Create data dump for debugging
source env/bin/activate && python main.py --dump

# Access the web interface at http://localhost:8080

Development Setup

The application requires:

• Python 3.12+
• Virtual environment at env/ (must be activated before running commands)
• Dependencies from pyproject.toml (includes FastAPI, uvicorn, Socket.IO)

Docker deployment:

# Build and run with docker-compose
docker-compose up -d

# Access at http://localhost:8080

Testing

Automated Tests

The project includes a test suite in the tests/ directory:

# Activate virtual environment and run tests
source env/bin/activate && python -m pytest tests/

Test Files:

• tests/test_proxy_settings.py - Tests for proxy settings configuration
• tests/test_verify_proxy.py - Tests for proxy verification functionality

Manual Testing

1. Run with -vvv for maximum verbosity (levels: -v, -vv, -vvv, -vvvv)
2. Use --dump to generate debug data dumps
3. Check log files in ./logs/ directory
4. Use --debug-ws for websocket debug logging
5. Use --debug-gql for GraphQL debug logging
6. Monitor web GUI console output and browser developer tools

Web GUI Architecture

The application uses a web-based interface accessible via browser:

Web GUI Components

src/web/gui_manager.py - WebGUIManager class:

• Managers: StatusManager, ConsoleOutputManager, ChannelListManager, CampaignProgressManager, InventoryManager, LoginFormManager, SettingsManager, CacheManager
• Uses WebSocketBroadcaster to push real-time updates to connected clients via Socket.IO
• Pure async/await implementation

src/web/app.py - FastAPI application:

• REST API endpoints: /api/status, /api/channels, /api/campaigns, /api/settings, /api/login, /api/oauth/confirm, /api/reload, /api/close, /api/version
• Socket.IO server for real-time bi-directional communication
• Serves static web frontend from web/ directory
• Integrates with WebGUIManager via set_managers()

web/ - Frontend assets:

• index.html - Single-page application layout with tabs
• static/app.js - Socket.IO client, real-time UI updates, API calls, Inventory Filtering logic
• static/styles.css - Responsive design with dark mode support

Communication Protocol

Server → Client (Socket.IO events):

• initial_state - Full state on connect
• status_update - Status bar changes
• console_output - New log lines
• channel_add/update/remove - Channel list changes
• drop_progress - Drop mining progress
• campaign_add - New campaign added
• login_required - Prompt for credentials
• settings_updated - Settings changed

Client → Server:

• REST API for actions (login, settings, channel selection)
• Socket.IO for connection management

Docker Integration

src/config/paths.py:

• Detects Docker environment via DOCKER_ENV env var or /.dockerenv file
• Docker: Uses /app for code, /app/data for persistent storage
• Development: Uses <project_root>/data for persistent storage
• All user data (cookies, settings, cache, logs) stored in DATA_DIR
• Provides _resource_path() helper for locating bundled resources

Dockerfile:

• Based on python:3
• Installs dependencies from pyproject.toml
• Exposes port 8080
• Health check on /api/status

docker-compose.yml:

• Volume mounts ./data:/app/data for persistence
• Port mapping 8080:8080
• Auto-restart policy
• Timezone configuration

Key Design Decisions

• WebSocket for real-time - Socket.IO chosen for reliability (fallback to polling)
• Single-page app - Simpler than full framework (React/Vue), fast load times
• Direct Docker support - Environment detection, proper path handling
• OAuth device code flow - Works great for web-based deployment

Project Scope

Supported:

• ✅ Web GUI - browser-based interface with advanced filtering
• ✅ Docker deployment - containerized for any platform
• ✅ Remote access - access from any device on network
• ✅ Headless operation - no display server required

NOT supported:

• Multi-account support
• Channel points mining
• Mining for unlinked campaigns
• Desktop GUI