projects/Dockers/puppeteer-api/README.md

Puppeteer API Server

A FastAPI-based server that uses Puppeteer (via pyppeteer) to scrape websites and extract various types of data.

Recent Fixes (v2.0)

Issues Fixed

• Resource temporarily unavailable (Errno 11) - Fixed by implementing proper browser cleanup and resource limits
• 100% CPU usage - Fixed by reducing browser pool size, adding operation limits, and improving cleanup
• Server hanging - Fixed by adding timeouts, emergency cleanup, and better error handling

Key Improvements

1. Resource Management

• Reduced browser pool size: From 5 to 3 browsers maximum
• Shorter browser TTL: From 1 hour to 30 minutes
• Concurrent operation limits: Maximum 5 concurrent operations
• Queue size limits: Prevent unbounded growth

2. Better Cleanup

• Automatic browser recycling: Browsers are recycled every 30 minutes
• Emergency cleanup: Force cleanup all browsers when needed
• Proper page cleanup: Ensure pages are closed after each operation
• Signal handlers: Graceful shutdown on SIGTERM/SIGINT

3. Monitoring & Debugging

• System status endpoint: /status - Monitor CPU, memory, file descriptors
• Browser pool monitoring: Track active browsers and their ages
• Emergency cleanup endpoint: /emergency-cleanup - Force cleanup when needed
• Monitoring script: monitor.py - Real-time system monitoring

4. Error Handling

• Timeout handling: 30s timeout for getting browsers from pool
• Emergency recovery: Auto-cleanup when timeouts occur
• Better exception handling: More detailed error logging

Environment Variables

# Required
API_KEY=your-api-key-here

# Optional (with defaults)
MAX_BROWSERS=3                    # Maximum browser instances (default: 3)
BROWSER_TTL=1800                  # Browser time-to-live in seconds (default: 1800 = 30min)
MAX_CONCURRENT_OPERATIONS=5       # Max concurrent operations (default: 5)
CACHE_EXPIRY_HOURS=36            # Cache expiry in hours (default: 36)
CLEANUP_CRON=0 3 * * *           # Cache cleanup schedule (default: daily at 3 AM)
RATE_LIMIT_MINUTE=60             # Requests per minute (default: 60)

API Endpoints

Core Endpoints

• GET / - Visit URL and get HTML content
• GET /seo - Extract SEO information
• GET /meta - Extract meta tags and Open Graph data

Management Endpoints

• HEAD / - Health check
• GET /status - System status and browser pool information
• POST /emergency-cleanup - Force cleanup all browsers
• GET /cache/stats - Cache statistics
• GET /cache/clear - Clear all cache

Usage Examples

Basic Usage

# Visit a URL
curl -H "X-API-Key: your-api-key" "http://localhost:8000/?url=https://example.com"

# Extract SEO data
curl -H "X-API-Key: your-api-key" "http://localhost:8000/seo?url=https://example.com"

# Get system status
curl -H "X-API-Key: your-api-key" "http://localhost:8000/status"

Monitoring

# Monitor system every 30 seconds
python monitor.py your-api-key 30

# Monitor with custom URL
python monitor.py your-api-key 30 http://your-server:8000

# Trigger emergency cleanup
python monitor.py your-api-key 30 http://localhost:8000 cleanup

Docker Usage

# Build the image
docker build -t puppeteer-api .

# Run with environment variables
docker run -d \
  --name puppeteer-api \
  -p 8000:8000 \
  -e API_KEY=your-api-key \
  -e MAX_BROWSERS=3 \
  -e BROWSER_TTL=1800 \
  -v /path/to/cache:/db \
  puppeteer-api

Troubleshooting

High CPU Usage

1. Check system status: GET /status
2. If browser pool is at capacity, trigger cleanup: POST /emergency-cleanup
3. Consider reducing MAX_BROWSERS or MAX_CONCURRENT_OPERATIONS

Resource Errors (Errno 11)

1. The system now automatically handles this with better cleanup
2. Monitor with python monitor.py to track resource usage
3. If persistent, trigger emergency cleanup

Server Hanging

1. Check for stuck operations with monitoring script
2. Trigger emergency cleanup
3. Restart the container if needed

Performance Tips

1. Use caching: The API caches results for 36 hours by default
2. Monitor resources: Use the monitoring script to track usage
3. Adjust limits: Tune MAX_BROWSERS and MAX_CONCURRENT_OPERATIONS based on your server capacity
4. Regular cleanup: The system automatically recycles browsers every 30 minutes

Logs

The server provides detailed logging for:

• Browser creation and cleanup
• Resource usage warnings
• Error conditions
• Cache operations

Monitor logs to identify patterns and adjust configuration accordingly.