A FastAPI-based server that uses Playwright to scrape websites and extract various types of data. Now with advanced Cloudflare bypass capabilities!

🛡️ Cloudflare Bypass Features (v4.0)

This API now includes sophisticated Cloudflare bypass techniques based on the latest anti-detection methods:

Key Bypass Features

🔄 Dynamic User Agents: Rotates between modern, realistic user agents
🎭 Stealth Mode: Comprehensive browser fingerprint spoofing
🤖 Human-like Behavior: Simulates real user interactions
🔒 Anti-Detection Scripts: Injects scripts to bypass automation detection
🌐 Proxy Support: Optional proxy configuration for additional stealth
⏱️ Challenge Handling: Automatic Cloudflare challenge detection and waiting
📱 Realistic Viewports: Random viewport sizes to appear more human
🌍 Geolocation Spoofing: Simulates realistic location data

Bypass Techniques Implemented

Browser Fingerprinting Protection
- Removes navigator.webdriver property
- Overrides automation detection methods
- Spoofs browser plugins and languages
- Masks Chrome automation indicators
Request Header Spoofing
- Modern Sec-Ch-Ua headers
- Realistic Accept and Accept-Language headers
- Proper Sec-Fetch-* headers
- Cache control headers
Human-like Behavior
- Random mouse movements
- Natural scrolling patterns
- Realistic timing delays
- Page interaction simulation
Challenge Detection & Handling
- Automatic Cloudflare challenge detection
- Intelligent waiting for challenge completion
- Multiple retry attempts with backoff
- Success/failure reporting

Testing Cloudflare Bypass

Use the new /test-cloudflare endpoint to test bypass effectiveness:

# Test Cloudflare bypass on a protected site
curl -H "X-API-Key: your-api-key" \
  "http://localhost:8000/test-cloudflare?url=https://cloudflare-protected-site.com"

Response includes detailed bypass status:

{
  "status": "success",
  "url": "https://example.com",
  "final_url": "https://example.com",
  "title": "Example Domain",
  "cloudflare_bypassed": true,
  "cloudflare_indicators": {
    "has_cloudflare_title": false,
    "has_challenge_form": false,
    "has_cf_wrapper": false,
    "has_please_wait": false,
    "has_browser_verification": false
  },
  "content_length": 12345
}

Recent Migration (v3.0)

Migration from Puppeteer to Playwright

This project has been successfully migrated from Puppeteer (pyppeteer) to Playwright for better performance, reliability, and maintainability.

Key Benefits of Playwright

Better performance: More efficient browser management and faster page loads
Improved stability: Better handling of modern web applications
Enhanced features: Better support for modern web standards
Active development: More frequent updates and better community support
Multi-browser support: Can easily switch between Chromium, Firefox, and WebKit

Migration Changes

Dependencies: Updated from pyppeteer to playwright
Browser management: Improved browser pool with better resource management
API compatibility: All existing endpoints remain the same
Docker image: Updated to use Playwright browsers instead of Chrome

Recent Fixes (v2.0)

Issues Fixed

Resource temporarily unavailable (Errno 11) - Fixed by implementing proper browser cleanup and resource limits
100% CPU usage - Fixed by reducing browser pool size, adding operation limits, and improving cleanup
Server hanging - Fixed by adding timeouts, emergency cleanup, and better error handling

Key Improvements

1. Resource Management

Reduced browser pool size: From 5 to 3 browsers maximum
Shorter browser TTL: From 1 hour to 30 minutes
Concurrent operation limits: Maximum 5 concurrent operations
Queue size limits: Prevent unbounded growth

2. Better Cleanup

Automatic browser recycling: Browsers are recycled every 30 minutes
Emergency cleanup: Force cleanup all browsers when needed
Proper page cleanup: Ensure pages are closed after each operation
Signal handlers: Graceful shutdown on SIGTERM/SIGINT

3. Monitoring & Debugging

System status endpoint: /status - Monitor CPU, memory, file descriptors
Browser pool monitoring: Track active browsers and their ages
Emergency cleanup endpoint: /emergency-cleanup - Force cleanup when needed
Timeout-based cleanup: /force-cleanup-old - Force close old browser/page instances
Monitoring script: monitor.py - Real-time system monitoring

4. Timeout Management

Configurable timeout: BROWSER_INSTANCE_TIMEOUT_MINUTES (default: 10 minutes)
Automatic cleanup: Old browser and page instances are automatically force closed
Page tracking: All page instances are tracked with creation timestamps
Health check integration: Timeout cleanup runs every 2 minutes as part of health checks

5. Error Handling

Timeout handling: 30s timeout for getting browsers from pool
Emergency recovery: Auto-cleanup when timeouts occur
Better exception handling: More detailed error logging

Environment Variables

# Required
API_KEY=your-api-key-here

# Optional (with defaults)
MAX_BROWSERS=3                    # Maximum browser instances (default: 3)
BROWSER_TTL=1800                  # Browser time-to-live in seconds (default: 1800 = 30min)
MAX_CONCURRENT_OPERATIONS=5       # Max concurrent operations (default: 5)
BROWSER_INSTANCE_TIMEOUT_MINUTES=10  # Force close old browser/page instances after N minutes (default: 10)
CACHE_EXPIRY_HOURS=36            # Cache expiry in hours (default: 36)
CLEANUP_CRON=0 3 * * *           # Cache cleanup schedule (default: daily at 3 AM)
RATE_LIMIT_MINUTE=60             # Requests per minute (default: 60)

# Cloudflare Bypass Configuration
ENABLE_CLOUDFLARE_BYPASS=true   # Enable Cloudflare bypass (default: true)
PROXY_URL=                      # Optional proxy URL for additional stealth (e.g., http://proxy:8080)

# Database Configuration
# If all PostgreSQL credentials are provided, PostgreSQL will be used
# Otherwise, SQLite will be used as fallback
POSTGRES_HOST=                   # PostgreSQL host (optional)
POSTGRES_PORT=5432              # PostgreSQL port (default: 5432)
POSTGRES_DB=                    # PostgreSQL database name (optional)
POSTGRES_USER=                  # PostgreSQL username (optional)
POSTGRES_PASSWORD=              # PostgreSQL password (optional)

API Endpoints

Core Endpoints

GET / - Visit URL and get HTML content (with Cloudflare bypass)
GET /seo - Extract SEO information (with Cloudflare bypass)
GET /meta - Extract meta tags and Open Graph data (with Cloudflare bypass)
GET /test-cloudflare - Test Cloudflare bypass functionality on a specific URL
GET /resulting-url - Get the final URL after navigation (handles redirects)

Management Endpoints

HEAD / - Health check
GET /status - System status and browser pool information
POST /emergency-cleanup - Force cleanup all browsers
POST /force-cleanup-old - Force cleanup old browser/page instances (based on timeout)
GET /cache/stats - Cache statistics
GET /cache/clear - Clear all cache

Usage Examples

Basic Usage

# Visit a URL
curl -H "X-API-Key: your-api-key" "http://localhost:8000/?url=https://example.com"

# Extract SEO data
curl -H "X-API-Key: your-api-key" "http://localhost:8000/seo?url=https://example.com"

# Get resulting URL (handles redirects)
curl -H "X-API-Key: your-api-key" "http://localhost:8000/resulting-url?url=https://example.com"

# Get system status
curl -H "X-API-Key: your-api-key" "http://localhost:8000/status"

# Test Cloudflare bypass
curl -H "X-API-Key: your-api-key" "http://localhost:8000/test-cloudflare?url=https://cloudflare-protected-site.com"

# Force cleanup old browser/page instances
curl -X POST -H "X-API-Key: your-api-key" "http://localhost:8000/force-cleanup-old"

Monitoring

# Monitor system every 30 seconds
python monitor.py your-api-key 30

# Monitor with custom URL
python monitor.py your-api-key 30 http://your-server:8000

# Trigger emergency cleanup
python monitor.py your-api-key 30 http://localhost:8000 cleanup

Docker Usage

Using SQLite (Default)

# Build the image
docker build -t playwright-api .

# Run with SQLite (default)
docker run -d \
  --name playwright-api \
  -p 8000:8000 \
  -e API_KEY=your-api-key \
  -e MAX_BROWSERS=3 \
  -e BROWSER_TTL=1800 \
  -v /path/to/cache:/db \
  playwright-api

Using PostgreSQL

# Run with PostgreSQL
docker run -d \
  --name playwright-api \
  -p 8000:8000 \
  -e API_KEY=your-api-key \
  -e MAX_BROWSERS=3 \
  -e BROWSER_TTL=1800 \
  -e POSTGRES_HOST=your-postgres-host \
  -e POSTGRES_PORT=5432 \
  -e POSTGRES_DB=your-database-name \
  -e POSTGRES_USER=your-username \
  -e POSTGRES_PASSWORD=your-password \
  playwright-api

Using Docker Compose with PostgreSQL

version: "3.8"
services:
  postgres:
    image: postgres:15
    environment:
      POSTGRES_DB: playwright_cache
      POSTGRES_USER: playwright_user
      POSTGRES_PASSWORD: your_password
    volumes:
      - postgres_data:/var/lib/postgresql/data
    ports:
      - "5432:5432"

  playwright-api:
    build: .
    ports:
      - "8000:8000"
    environment:
      API_KEY: your-api-key
      POSTGRES_HOST: postgres
      POSTGRES_PORT: 5432
      POSTGRES_DB: playwright_cache
      POSTGRES_USER: playwright_user
      POSTGRES_PASSWORD: your_password
    depends_on:
      - postgres

volumes:
  postgres_data:

Testing the Migration

To verify that the Playwright migration works correctly:

# Run the migration test script
python test_playwright_migration.py

This will test:

Playwright installation and basic functionality
Browser utilities module
Browser service functionality

Troubleshooting

High CPU Usage

Check system status: GET /status
If browser pool is at capacity, trigger cleanup: POST /emergency-cleanup
Force cleanup old instances: POST /force-cleanup-old
Consider reducing MAX_BROWSERS or MAX_CONCURRENT_OPERATIONS
Adjust BROWSER_INSTANCE_TIMEOUT_MINUTES to a lower value (e.g., 5 minutes)

Resource Errors (Errno 11)

The system now automatically handles this with better cleanup
Monitor with python monitor.py to track resource usage
If persistent, trigger emergency cleanup

Server Hanging

Check for stuck operations with monitoring script
Trigger emergency cleanup
Restart the container if needed

Playwright Installation Issues

Ensure Playwright browsers are installed: playwright install chromium
Check Docker build logs for browser installation
Verify system dependencies are installed

Database Configuration

The API supports both SQLite and PostgreSQL for caching:

SQLite (Default)

Automatic: Used when no PostgreSQL credentials are provided
File-based: Database stored in /db/cache.db (or cache.db as fallback)
Simple setup: No additional services required
Suitable for: Development, testing, and small deployments

PostgreSQL

Configured via environment variables: Set all PostgreSQL credentials to enable
Better performance: For high-traffic applications
Scalable: Can handle concurrent connections better
Suitable for: Production deployments, high-traffic scenarios

Migration

Automatic detection: The system automatically chooses the database based on configuration
No data migration needed: Each database type maintains its own cache
Backward compatible: Existing SQLite setups continue to work unchanged

Performance Tips

Use caching: The API caches results for 36 hours by default
Monitor resources: Use the monitoring script to track usage
Adjust limits: Tune MAX_BROWSERS and MAX_CONCURRENT_OPERATIONS based on your server capacity
Regular cleanup: The system automatically recycles browsers every 30 minutes
Database choice: Use PostgreSQL for high-traffic production deployments

Logs

The server provides detailed logging for:

Browser creation and cleanup
Resource usage warnings
Error conditions
Cache operations

Monitor logs to identify patterns and adjust configuration accordingly.