try another fix

Dockers/puppeteer-api/README.md

@@ -0,0 +1,154 @@
+# Puppeteer API Server
+
+A FastAPI-based server that uses Puppeteer (via pyppeteer) to scrape websites and extract various types of data.
+
+## Recent Fixes (v2.0)
+
+### Issues Fixed
+
+- **Resource temporarily unavailable (Errno 11)** - Fixed by implementing proper browser cleanup and resource limits
+- **100% CPU usage** - Fixed by reducing browser pool size, adding operation limits, and improving cleanup
+- **Server hanging** - Fixed by adding timeouts, emergency cleanup, and better error handling
+
+### Key Improvements
+
+#### 1. Resource Management
+
+- **Reduced browser pool size**: From 5 to 3 browsers maximum
+- **Shorter browser TTL**: From 1 hour to 30 minutes
+- **Concurrent operation limits**: Maximum 5 concurrent operations
+- **Queue size limits**: Prevent unbounded growth
+
+#### 2. Better Cleanup
+
+- **Automatic browser recycling**: Browsers are recycled every 30 minutes
+- **Emergency cleanup**: Force cleanup all browsers when needed
+- **Proper page cleanup**: Ensure pages are closed after each operation
+- **Signal handlers**: Graceful shutdown on SIGTERM/SIGINT
+
+#### 3. Monitoring & Debugging
+
+- **System status endpoint**: `/status` - Monitor CPU, memory, file descriptors
+- **Browser pool monitoring**: Track active browsers and their ages
+- **Emergency cleanup endpoint**: `/emergency-cleanup` - Force cleanup when needed
+- **Monitoring script**: `monitor.py` - Real-time system monitoring
+
+#### 4. Error Handling
+
+- **Timeout handling**: 30s timeout for getting browsers from pool
+- **Emergency recovery**: Auto-cleanup when timeouts occur
+- **Better exception handling**: More detailed error logging
+
+## Environment Variables
+
+```bash
+# Required
+API_KEY=your-api-key-here
+
+# Optional (with defaults)
+MAX_BROWSERS=3                    # Maximum browser instances (default: 3)
+BROWSER_TTL=1800                  # Browser time-to-live in seconds (default: 1800 = 30min)
+MAX_CONCURRENT_OPERATIONS=5       # Max concurrent operations (default: 5)
+CACHE_EXPIRY_HOURS=36            # Cache expiry in hours (default: 36)
+CLEANUP_CRON=0 3 * * *           # Cache cleanup schedule (default: daily at 3 AM)
+RATE_LIMIT_MINUTE=60             # Requests per minute (default: 60)
+```
+
+## API Endpoints
+
+### Core Endpoints
+
+- `GET /` - Visit URL and get HTML content
+- `GET /seo` - Extract SEO information
+- `GET /meta` - Extract meta tags and Open Graph data
+
+### Management Endpoints
+
+- `HEAD /` - Health check
+- `GET /status` - System status and browser pool information
+- `POST /emergency-cleanup` - Force cleanup all browsers
+- `GET /cache/stats` - Cache statistics
+- `GET /cache/clear` - Clear all cache
+
+## Usage Examples
+
+### Basic Usage
+
+```bash
+# Visit a URL
+curl -H "X-API-Key: your-api-key" "http://localhost:8000/?url=https://example.com"
+
+# Extract SEO data
+curl -H "X-API-Key: your-api-key" "http://localhost:8000/seo?url=https://example.com"
+
+# Get system status
+curl -H "X-API-Key: your-api-key" "http://localhost:8000/status"
+```
+
+### Monitoring
+
+```bash
+# Monitor system every 30 seconds
+python monitor.py your-api-key 30
+
+# Monitor with custom URL
+python monitor.py your-api-key 30 http://your-server:8000
+
+# Trigger emergency cleanup
+python monitor.py your-api-key 30 http://localhost:8000 cleanup
+```
+
+## Docker Usage
+
+```bash
+# Build the image
+docker build -t puppeteer-api .
+
+# Run with environment variables
+docker run -d \
+  --name puppeteer-api \
+  -p 8000:8000 \
+  -e API_KEY=your-api-key \
+  -e MAX_BROWSERS=3 \
+  -e BROWSER_TTL=1800 \
+  -v /path/to/cache:/db \
+  puppeteer-api
+```
+
+## Troubleshooting
+
+### High CPU Usage
+
+1. Check system status: `GET /status`
+2. If browser pool is at capacity, trigger cleanup: `POST /emergency-cleanup`
+3. Consider reducing `MAX_BROWSERS` or `MAX_CONCURRENT_OPERATIONS`
+
+### Resource Errors (Errno 11)
+
+1. The system now automatically handles this with better cleanup
+2. Monitor with `python monitor.py` to track resource usage
+3. If persistent, trigger emergency cleanup
+
+### Server Hanging
+
+1. Check for stuck operations with monitoring script
+2. Trigger emergency cleanup
+3. Restart the container if needed
+
+## Performance Tips
+
+1. **Use caching**: The API caches results for 36 hours by default
+2. **Monitor resources**: Use the monitoring script to track usage
+3. **Adjust limits**: Tune `MAX_BROWSERS` and `MAX_CONCURRENT_OPERATIONS` based on your server capacity
+4. **Regular cleanup**: The system automatically recycles browsers every 30 minutes
+
+## Logs
+
+The server provides detailed logging for:
+
+- Browser creation and cleanup
+- Resource usage warnings
+- Error conditions
+- Cache operations
+
+Monitor logs to identify patterns and adjust configuration accordingly.