cleanup

Dockers/puppeteer-api/README.md

@@ -52,9 +52,17 @@ This project has been successfully migrated from Puppeteer (pyppeteer) to Playwr
 - **System status endpoint**: `/status` - Monitor CPU, memory, file descriptors
 - **Browser pool monitoring**: Track active browsers and their ages
 - **Emergency cleanup endpoint**: `/emergency-cleanup` - Force cleanup when needed
+- **Timeout-based cleanup**: `/force-cleanup-old` - Force close old browser/page instances
 - **Monitoring script**: `monitor.py` - Real-time system monitoring
 
-#### 4. Error Handling
+#### 4. Timeout Management
+
+- **Configurable timeout**: `BROWSER_INSTANCE_TIMEOUT_MINUTES` (default: 10 minutes)
+- **Automatic cleanup**: Old browser and page instances are automatically force closed
+- **Page tracking**: All page instances are tracked with creation timestamps
+- **Health check integration**: Timeout cleanup runs every 2 minutes as part of health checks
+
+#### 5. Error Handling
 
 - **Timeout handling**: 30s timeout for getting browsers from pool
 - **Emergency recovery**: Auto-cleanup when timeouts occur
@@ -70,6 +78,7 @@ API_KEY=your-api-key-here
 MAX_BROWSERS=3                    # Maximum browser instances (default: 3)
 BROWSER_TTL=1800                  # Browser time-to-live in seconds (default: 1800 = 30min)
 MAX_CONCURRENT_OPERATIONS=5       # Max concurrent operations (default: 5)
+BROWSER_INSTANCE_TIMEOUT_MINUTES=10  # Force close old browser/page instances after N minutes (default: 10)
 CACHE_EXPIRY_HOURS=36            # Cache expiry in hours (default: 36)
 CLEANUP_CRON=0 3 * * *           # Cache cleanup schedule (default: daily at 3 AM)
 RATE_LIMIT_MINUTE=60             # Requests per minute (default: 60)
@@ -82,12 +91,22 @@ RATE_LIMIT_MINUTE=60             # Requests per minute (default: 60)
 - `GET /` - Visit URL and get HTML content
 - `GET /seo` - Extract SEO information
 - `GET /meta` - Extract meta tags and Open Graph data
+- `GET /json` - Fetch JSON content from a website
+
+The `/json` endpoint intelligently extracts JSON data from websites by:
+
+- Directly parsing JSON responses (when Content-Type is application/json)
+- Extracting JSON from `<script type="application/json">` tags
+- Finding JSON in `data-json` attributes
+- Searching for JSON-like structures in page content
+- Parsing the entire page content as JSON if possible
 
 ### Management Endpoints
 
 - `HEAD /` - Health check
 - `GET /status` - System status and browser pool information
 - `POST /emergency-cleanup` - Force cleanup all browsers
+- `POST /force-cleanup-old` - Force cleanup old browser/page instances (based on timeout)
 - `GET /cache/stats` - Cache statistics
 - `GET /cache/clear` - Clear all cache
 
@@ -102,8 +121,14 @@ curl -H "X-API-Key: your-api-key" "http://localhost:8000/?url=https://example.co
 # Extract SEO data
 curl -H "X-API-Key: your-api-key" "http://localhost:8000/seo?url=https://example.com"
 
+# Fetch JSON content
+curl -H "X-API-Key: your-api-key" "http://localhost:8000/json?url=https://api.example.com/data"
+
 # Get system status
 curl -H "X-API-Key: your-api-key" "http://localhost:8000/status"
+
+# Force cleanup old browser/page instances
+curl -X POST -H "X-API-Key: your-api-key" "http://localhost:8000/force-cleanup-old"
 ```
 
 ### Monitoring
@@ -157,7 +182,9 @@ This will test:
 
 1. Check system status: `GET /status`
 2. If browser pool is at capacity, trigger cleanup: `POST /emergency-cleanup`
-3. Consider reducing `MAX_BROWSERS` or `MAX_CONCURRENT_OPERATIONS`
+3. Force cleanup old instances: `POST /force-cleanup-old`
+4. Consider reducing `MAX_BROWSERS` or `MAX_CONCURRENT_OPERATIONS`
+5. Adjust `BROWSER_INSTANCE_TIMEOUT_MINUTES` to a lower value (e.g., 5 minutes)
 
 ### Resource Errors (Errno 11)