# Playwright API Server

A FastAPI-based server that uses Playwright to scrape websites and extract various types of data. **Now with advanced Cloudflare bypass capabilities!**

## 🛡️ Cloudflare Bypass Features (v4.0)

This API now includes sophisticated Cloudflare bypass techniques based on the latest anti-detection methods:

### Key Bypass Features

- **🔄 Dynamic User Agents**: Rotates between modern, realistic user agents
- **🎭 Stealth Mode**: Comprehensive browser fingerprint spoofing
- **🤖 Human-like Behavior**: Simulates real user interactions
- **🔒 Anti-Detection Scripts**: Injects scripts to bypass automation detection
- **🌐 Proxy Support**: Optional proxy configuration for additional stealth
- **⏱️ Challenge Handling**: Automatic Cloudflare challenge detection and waiting
- **📱 Realistic Viewports**: Random viewport sizes to appear more human
- **🌍 Geolocation Spoofing**: Simulates realistic location data

### Bypass Techniques Implemented

1. **Browser Fingerprinting Protection**

   - Removes `navigator.webdriver` property
   - Overrides automation detection methods
   - Spoofs browser plugins and languages
   - Masks Chrome automation indicators

2. **Request Header Spoofing**

   - Modern `Sec-Ch-Ua` headers
   - Realistic `Accept` and `Accept-Language` headers
   - Proper `Sec-Fetch-*` headers
   - Cache control headers

3. **Human-like Behavior**

   - Random mouse movements
   - Natural scrolling patterns
   - Realistic timing delays
   - Page interaction simulation

4. **Challenge Detection & Handling**
   - Automatic Cloudflare challenge detection
   - Intelligent waiting for challenge completion
   - Multiple retry attempts with backoff
   - Success/failure reporting

### Testing Cloudflare Bypass

Use the new `/test-cloudflare` endpoint to test bypass effectiveness:

```bash
# Test Cloudflare bypass on a protected site
curl -H "X-API-Key: your-api-key" \
  "http://localhost:8000/test-cloudflare?url=https://cloudflare-protected-site.com"
```

Response includes detailed bypass status:

```json
{
  "status": "success",
  "url": "https://example.com",
  "final_url": "https://example.com",
  "title": "Example Domain",
  "cloudflare_bypassed": true,
  "cloudflare_indicators": {
    "has_cloudflare_title": false,
    "has_challenge_form": false,
    "has_cf_wrapper": false,
    "has_please_wait": false,
    "has_browser_verification": false
  },
  "content_length": 12345
}
```

## Recent Migration (v3.0)

### Migration from Puppeteer to Playwright

This project has been successfully migrated from Puppeteer (pyppeteer) to Playwright for better performance, reliability, and maintainability.

#### Key Benefits of Playwright

- **Better performance**: More efficient browser management and faster page loads
- **Improved stability**: Better handling of modern web applications
- **Enhanced features**: Better support for modern web standards
- **Active development**: More frequent updates and better community support
- **Multi-browser support**: Can easily switch between Chromium, Firefox, and WebKit

#### Migration Changes

- **Dependencies**: Updated from `pyppeteer` to `playwright`
- **Browser management**: Improved browser pool with better resource management
- **API compatibility**: All existing endpoints remain the same
- **Docker image**: Updated to use Playwright browsers instead of Chrome

## Recent Fixes (v2.0)

### Issues Fixed

- **Resource temporarily unavailable (Errno 11)** - Fixed by implementing proper browser cleanup and resource limits
- **100% CPU usage** - Fixed by reducing browser pool size, adding operation limits, and improving cleanup
- **Server hanging** - Fixed by adding timeouts, emergency cleanup, and better error handling

### Key Improvements

#### 1. Resource Management

- **Reduced browser pool size**: From 5 to 3 browsers maximum
- **Shorter browser TTL**: From 1 hour to 30 minutes
- **Concurrent operation limits**: Maximum 5 concurrent operations
- **Queue size limits**: Prevent unbounded growth

#### 2. Better Cleanup

- **Automatic browser recycling**: Browsers are recycled every 30 minutes
- **Emergency cleanup**: Force cleanup all browsers when needed
- **Proper page cleanup**: Ensure pages are closed after each operation
- **Signal handlers**: Graceful shutdown on SIGTERM/SIGINT

#### 3. Monitoring & Debugging

- **System status endpoint**: `/status` - Monitor CPU, memory, file descriptors
- **Browser pool monitoring**: Track active browsers and their ages
- **Emergency cleanup endpoint**: `/emergency-cleanup` - Force cleanup when needed
- **Timeout-based cleanup**: `/force-cleanup-old` - Force close old browser/page instances
- **Monitoring script**: `monitor.py` - Real-time system monitoring

#### 4. Timeout Management

- **Configurable timeout**: `BROWSER_INSTANCE_TIMEOUT_MINUTES` (default: 10 minutes)
- **Automatic cleanup**: Old browser and page instances are automatically force closed
- **Page tracking**: All page instances are tracked with creation timestamps
- **Health check integration**: Timeout cleanup runs every 2 minutes as part of health checks

#### 5. Error Handling

- **Timeout handling**: 30s timeout for getting browsers from pool
- **Emergency recovery**: Auto-cleanup when timeouts occur
- **Better exception handling**: More detailed error logging

## Environment Variables

```bash
# Required
API_KEY=your-api-key-here

# Optional (with defaults)
MAX_BROWSERS=3                    # Maximum browser instances (default: 3)
BROWSER_TTL=1800                  # Browser time-to-live in seconds (default: 1800 = 30min)
MAX_CONCURRENT_OPERATIONS=5       # Max concurrent operations (default: 5)
BROWSER_INSTANCE_TIMEOUT_MINUTES=10  # Force close old browser/page instances after N minutes (default: 10)
CACHE_EXPIRY_HOURS=36            # Cache expiry in hours (default: 36)
CLEANUP_CRON=0 3 * * *           # Cache cleanup schedule (default: daily at 3 AM)
RATE_LIMIT_MINUTE=60             # Requests per minute (default: 60)

# Cloudflare Bypass Configuration
ENABLE_CLOUDFLARE_BYPASS=true   # Enable Cloudflare bypass (default: true)
PROXY_URL=                      # Optional proxy URL for additional stealth (e.g., http://proxy:8080)

# Database Configuration
# If all PostgreSQL credentials are provided, PostgreSQL will be used
# Otherwise, SQLite will be used as fallback
POSTGRES_HOST=                   # PostgreSQL host (optional)
POSTGRES_PORT=5432              # PostgreSQL port (default: 5432)
POSTGRES_DB=                    # PostgreSQL database name (optional)
POSTGRES_USER=                  # PostgreSQL username (optional)
POSTGRES_PASSWORD=              # PostgreSQL password (optional)
```

## API Endpoints

### Core Endpoints

- `GET /` - Visit URL and get HTML content (with Cloudflare bypass)
- `GET /screenshot` - Capture a screenshot of a URL (PNG output)
- `GET /seo` - Extract SEO information (with Cloudflare bypass)
- `GET /meta` - Extract meta tags and Open Graph data (with Cloudflare bypass)
- `GET /test-cloudflare` - Test Cloudflare bypass functionality on a specific URL
- `GET /resulting-url` - Get the final URL after navigation (handles redirects)

### Management Endpoints

- `HEAD /` - Health check
- `GET /status` - System status and browser pool information
- `POST /emergency-cleanup` - Force cleanup all browsers
- `POST /force-cleanup-old` - Force cleanup old browser/page instances (based on timeout)
- `GET /cache/stats` - Cache statistics
- `GET /cache/clear` - Clear all cache

## Usage Examples

### Basic Usage

```bash
# Visit a URL
curl -H "X-API-Key: your-api-key" "http://localhost:8000/?url=https://example.com"

# Capture a full-page screenshot (PNG)
curl -H "X-API-Key: your-api-key" "http://localhost:8000/screenshot?url=https://example.com" --output screenshot.png

# Extract SEO data
curl -H "X-API-Key: your-api-key" "http://localhost:8000/seo?url=https://example.com"

# Get resulting URL (handles redirects)
curl -H "X-API-Key: your-api-key" "http://localhost:8000/resulting-url?url=https://example.com"

# Get system status
curl -H "X-API-Key: your-api-key" "http://localhost:8000/status"

# Test Cloudflare bypass
curl -H "X-API-Key: your-api-key" "http://localhost:8000/test-cloudflare?url=https://cloudflare-protected-site.com"

# Force cleanup old browser/page instances
curl -X POST -H "X-API-Key: your-api-key" "http://localhost:8000/force-cleanup-old"
```

### Monitoring

```bash
# Monitor system every 30 seconds
python monitor.py your-api-key 30

# Monitor with custom URL
python monitor.py your-api-key 30 http://your-server:8000

# Trigger emergency cleanup
python monitor.py your-api-key 30 http://localhost:8000 cleanup
```

## Docker Usage

### Using SQLite (Default)

```bash
# Build the image
docker build -t playwright-api .

# Run with SQLite (default)
docker run -d \
  --name playwright-api \
  -p 8000:8000 \
  -e API_KEY=your-api-key \
  -e MAX_BROWSERS=3 \
  -e BROWSER_TTL=1800 \
  -v /path/to/cache:/db \
  playwright-api
```

### Using PostgreSQL

```bash
# Run with PostgreSQL
docker run -d \
  --name playwright-api \
  -p 8000:8000 \
  -e API_KEY=your-api-key \
  -e MAX_BROWSERS=3 \
  -e BROWSER_TTL=1800 \
  -e POSTGRES_HOST=your-postgres-host \
  -e POSTGRES_PORT=5432 \
  -e POSTGRES_DB=your-database-name \
  -e POSTGRES_USER=your-username \
  -e POSTGRES_PASSWORD=your-password \
  playwright-api
```

### Using Docker Compose with PostgreSQL

```yaml
version: "3.8"
services:
  postgres:
    image: postgres:15
    environment:
      POSTGRES_DB: playwright_cache
      POSTGRES_USER: playwright_user
      POSTGRES_PASSWORD: your_password
    volumes:
      - postgres_data:/var/lib/postgresql/data
    ports:
      - "5432:5432"

  playwright-api:
    build: .
    ports:
      - "8000:8000"
    environment:
      API_KEY: your-api-key
      POSTGRES_HOST: postgres
      POSTGRES_PORT: 5432
      POSTGRES_DB: playwright_cache
      POSTGRES_USER: playwright_user
      POSTGRES_PASSWORD: your_password
    depends_on:
      - postgres

volumes:
  postgres_data:
```

## Testing the Migration

To verify that the Playwright migration works correctly:

```bash
# Run the migration test script
python test_playwright_migration.py
```

This will test:

- Playwright installation and basic functionality
- Browser utilities module
- Browser service functionality

## Troubleshooting

### High CPU Usage

1. Check system status: `GET /status`
2. If browser pool is at capacity, trigger cleanup: `POST /emergency-cleanup`
3. Force cleanup old instances: `POST /force-cleanup-old`
4. Consider reducing `MAX_BROWSERS` or `MAX_CONCURRENT_OPERATIONS`
5. Adjust `BROWSER_INSTANCE_TIMEOUT_MINUTES` to a lower value (e.g., 5 minutes)

### Resource Errors (Errno 11)

1. The system now automatically handles this with better cleanup
2. Monitor with `python monitor.py` to track resource usage
3. If persistent, trigger emergency cleanup

### Server Hanging

1. Check for stuck operations with monitoring script
2. Trigger emergency cleanup
3. Restart the container if needed

### Playwright Installation Issues

1. Ensure Playwright browsers are installed: `playwright install chromium`
2. Check Docker build logs for browser installation
3. Verify system dependencies are installed

## Database Configuration

The API supports both SQLite and PostgreSQL for caching:

### SQLite (Default)

- **Automatic**: Used when no PostgreSQL credentials are provided
- **File-based**: Database stored in `/db/cache.db` (or `cache.db` as fallback)
- **Simple setup**: No additional services required
- **Suitable for**: Development, testing, and small deployments

### PostgreSQL

- **Configured via environment variables**: Set all PostgreSQL credentials to enable
- **Better performance**: For high-traffic applications
- **Scalable**: Can handle concurrent connections better
- **Suitable for**: Production deployments, high-traffic scenarios

### Migration

- **Automatic detection**: The system automatically chooses the database based on configuration
- **No data migration needed**: Each database type maintains its own cache
- **Backward compatible**: Existing SQLite setups continue to work unchanged

## Performance Tips

1. **Use caching**: The API caches results for 36 hours by default
2. **Monitor resources**: Use the monitoring script to track usage
3. **Adjust limits**: Tune `MAX_BROWSERS` and `MAX_CONCURRENT_OPERATIONS` based on your server capacity
4. **Regular cleanup**: The system automatically recycles browsers every 30 minutes
5. **Database choice**: Use PostgreSQL for high-traffic production deployments

## Logs

The server provides detailed logging for:

- Browser creation and cleanup
- Resource usage warnings
- Error conditions
- Cache operations

Monitor logs to identify patterns and adjust configuration accordingly.