projects/Dockers/puppeteer-api/README.md

Playwright API Server

A FastAPI-based server that uses Playwright to scrape websites and extract various types of data.

Recent Migration (v3.0)

Migration from Puppeteer to Playwright

This project has been successfully migrated from Puppeteer (pyppeteer) to Playwright for better performance, reliability, and maintainability.

Key Benefits of Playwright

• Better performance: More efficient browser management and faster page loads
• Improved stability: Better handling of modern web applications
• Enhanced features: Better support for modern web standards
• Active development: More frequent updates and better community support
• Multi-browser support: Can easily switch between Chromium, Firefox, and WebKit

Migration Changes

• Dependencies: Updated from pyppeteer to playwright
• Browser management: Improved browser pool with better resource management
• API compatibility: All existing endpoints remain the same
• Docker image: Updated to use Playwright browsers instead of Chrome

Recent Fixes (v2.0)

Issues Fixed

• Resource temporarily unavailable (Errno 11) - Fixed by implementing proper browser cleanup and resource limits
• 100% CPU usage - Fixed by reducing browser pool size, adding operation limits, and improving cleanup
• Server hanging - Fixed by adding timeouts, emergency cleanup, and better error handling

Key Improvements

1. Resource Management

• Reduced browser pool size: From 5 to 3 browsers maximum
• Shorter browser TTL: From 1 hour to 30 minutes
• Concurrent operation limits: Maximum 5 concurrent operations
• Queue size limits: Prevent unbounded growth

2. Better Cleanup

• Automatic browser recycling: Browsers are recycled every 30 minutes
• Emergency cleanup: Force cleanup all browsers when needed
• Proper page cleanup: Ensure pages are closed after each operation
• Signal handlers: Graceful shutdown on SIGTERM/SIGINT

3. Monitoring & Debugging

• System status endpoint: /status - Monitor CPU, memory, file descriptors
• Browser pool monitoring: Track active browsers and their ages
• Emergency cleanup endpoint: /emergency-cleanup - Force cleanup when needed
• Timeout-based cleanup: /force-cleanup-old - Force close old browser/page instances
• Monitoring script: monitor.py - Real-time system monitoring

4. Timeout Management

• Configurable timeout: BROWSER_INSTANCE_TIMEOUT_MINUTES (default: 10 minutes)
• Automatic cleanup: Old browser and page instances are automatically force closed
• Page tracking: All page instances are tracked with creation timestamps
• Health check integration: Timeout cleanup runs every 2 minutes as part of health checks

5. Error Handling

• Timeout handling: 30s timeout for getting browsers from pool
• Emergency recovery: Auto-cleanup when timeouts occur
• Better exception handling: More detailed error logging

Environment Variables

# Required
API_KEY=your-api-key-here

# Optional (with defaults)
MAX_BROWSERS=3                    # Maximum browser instances (default: 3)
BROWSER_TTL=1800                  # Browser time-to-live in seconds (default: 1800 = 30min)
MAX_CONCURRENT_OPERATIONS=5       # Max concurrent operations (default: 5)
BROWSER_INSTANCE_TIMEOUT_MINUTES=10  # Force close old browser/page instances after N minutes (default: 10)
CACHE_EXPIRY_HOURS=36            # Cache expiry in hours (default: 36)
CLEANUP_CRON=0 3 * * *           # Cache cleanup schedule (default: daily at 3 AM)
RATE_LIMIT_MINUTE=60             # Requests per minute (default: 60)

API Endpoints

Core Endpoints

• GET / - Visit URL and get HTML content
• GET /seo - Extract SEO information
• GET /meta - Extract meta tags and Open Graph data
• GET /json - Fetch JSON content from a website
• GET /resulting-url - Get the final URL after navigation (handles redirects)

The /json endpoint intelligently extracts JSON data from websites by:

• Directly parsing JSON responses (when Content-Type is application/json)
• Extracting JSON from <script type="application/json"> tags
• Finding JSON in data-json attributes
• Searching for JSON-like structures in page content
• Parsing the entire page content as JSON if possible

Management Endpoints

• HEAD / - Health check
• GET /status - System status and browser pool information
• POST /emergency-cleanup - Force cleanup all browsers
• POST /force-cleanup-old - Force cleanup old browser/page instances (based on timeout)
• GET /cache/stats - Cache statistics
• GET /cache/clear - Clear all cache

Usage Examples

Basic Usage

# Visit a URL
curl -H "X-API-Key: your-api-key" "http://localhost:8000/?url=https://example.com"

# Extract SEO data
curl -H "X-API-Key: your-api-key" "http://localhost:8000/seo?url=https://example.com"

# Fetch JSON content
curl -H "X-API-Key: your-api-key" "http://localhost:8000/json?url=https://api.example.com/data"

# Get resulting URL (handles redirects)
curl -H "X-API-Key: your-api-key" "http://localhost:8000/resulting-url?url=https://example.com"

# Get system status
curl -H "X-API-Key: your-api-key" "http://localhost:8000/status"

# Force cleanup old browser/page instances
curl -X POST -H "X-API-Key: your-api-key" "http://localhost:8000/force-cleanup-old"

Monitoring

# Monitor system every 30 seconds
python monitor.py your-api-key 30

# Monitor with custom URL
python monitor.py your-api-key 30 http://your-server:8000

# Trigger emergency cleanup
python monitor.py your-api-key 30 http://localhost:8000 cleanup

Docker Usage

# Build the image
docker build -t playwright-api .

# Run with environment variables
docker run -d \
  --name playwright-api \
  -p 8000:8000 \
  -e API_KEY=your-api-key \
  -e MAX_BROWSERS=3 \
  -e BROWSER_TTL=1800 \
  -v /path/to/cache:/db \
  playwright-api

Testing the Migration

To verify that the Playwright migration works correctly:

# Run the migration test script
python test_playwright_migration.py

This will test:

• Playwright installation and basic functionality
• Browser utilities module
• Browser service functionality

Troubleshooting

High CPU Usage

1. Check system status: GET /status
2. If browser pool is at capacity, trigger cleanup: POST /emergency-cleanup
3. Force cleanup old instances: POST /force-cleanup-old
4. Consider reducing MAX_BROWSERS or MAX_CONCURRENT_OPERATIONS
5. Adjust BROWSER_INSTANCE_TIMEOUT_MINUTES to a lower value (e.g., 5 minutes)

Resource Errors (Errno 11)

1. The system now automatically handles this with better cleanup
2. Monitor with python monitor.py to track resource usage
3. If persistent, trigger emergency cleanup

Server Hanging

1. Check for stuck operations with monitoring script
2. Trigger emergency cleanup
3. Restart the container if needed

Playwright Installation Issues

1. Ensure Playwright browsers are installed: playwright install chromium
2. Check Docker build logs for browser installation
3. Verify system dependencies are installed

Performance Tips

1. Use caching: The API caches results for 36 hours by default
2. Monitor resources: Use the monitoring script to track usage
3. Adjust limits: Tune MAX_BROWSERS and MAX_CONCURRENT_OPERATIONS based on your server capacity
4. Regular cleanup: The system automatically recycles browsers every 30 minutes

Logs

The server provides detailed logging for:

• Browser creation and cleanup
• Resource usage warnings
• Error conditions
• Cache operations

Monitor logs to identify patterns and adjust configuration accordingly.