projects/Dockers/puppeteer-api/CLOUDFLARE_BYPASS_EXAMPLES.md

Cloudflare Bypass Examples & Usage Guide

This guide demonstrates how to use the advanced Cloudflare bypass features in the Playwright API.

🚀 Quick Start

Basic Usage

All existing endpoints now automatically include Cloudflare bypass:

# Visit a Cloudflare-protected site
curl -H "X-API-Key: your-api-key" \
  "http://localhost:8000/?url=https://cloudflare-protected-site.com"

# Extract SEO from protected site
curl -H "X-API-Key: your-api-key" \
  "http://localhost:8000/seo?url=https://cloudflare-protected-site.com"

# Extract meta tags from protected site
curl -H "X-API-Key: your-api-key" \
  "http://localhost:8000/meta?url=https://cloudflare-protected-site.com"

Test Cloudflare Bypass

Use the dedicated test endpoint to verify bypass effectiveness:

# Test bypass on a specific URL
curl -H "X-API-Key: your-api-key" \
  "http://localhost:8000/test-cloudflare?url=https://cloudflare-protected-site.com"

🔧 Configuration Options

Environment Variables

# Enable/disable Cloudflare bypass (default: true)
ENABLE_CLOUDFLARE_BYPASS=true

# Optional proxy for additional stealth
PROXY_URL=http://proxy-server:8080

# Other existing variables still work
API_KEY=your-api-key
MAX_BROWSERS=3
BROWSER_TTL=1800

Docker Example with Proxy

docker run -d \
  --name playwright-api \
  -p 8000:8000 \
  -e API_KEY=your-api-key \
  -e ENABLE_CLOUDFLARE_BYPASS=true \
  -e PROXY_URL=http://your-proxy:8080 \
  -v /path/to/cache:/db \
  playwright-api

📊 Response Examples

Successful Bypass

{
  "status": "success",
  "url": "https://cloudflare-protected-site.com",
  "final_url": "https://cloudflare-protected-site.com",
  "title": "Protected Site - Home",
  "cloudflare_bypassed": true,
  "cloudflare_indicators": {
    "has_cloudflare_title": false,
    "has_challenge_form": false,
    "has_cf_wrapper": false,
    "has_please_wait": false,
    "has_browser_verification": false
  },
  "content_length": 45678
}

Failed Bypass

{
  "status": "error",
  "error": "Failed to bypass Cloudflare protection",
  "url": "https://cloudflare-protected-site.com",
  "cloudflare_detected": true
}

🧪 Testing Examples

Using the Test Script

# Test a known Cloudflare-protected site
python test_cloudflare_bypass.py \
  http://localhost:8000 \
  your-api-key \
  https://example-cloudflare-site.com

# Test multiple sites
for site in "site1.com" "site2.com" "site3.com"; do
  python test_cloudflare_bypass.py \
    http://localhost:8000 \
    your-api-key \
    "https://$site"
done

Manual Testing with curl

# Test bypass endpoint
curl -H "X-API-Key: your-api-key" \
  "http://localhost:8000/test-cloudflare?url=https://example.com" | jq

# Compare with regular endpoint
curl -H "X-API-Key: your-api-key" \
  "http://localhost:8000/?url=https://example.com" | jq '.content | length'

🔍 Bypass Techniques Explained

1. Browser Fingerprinting Protection

The API automatically:

• Removes navigator.webdriver property
• Overrides automation detection methods
• Spoofs browser plugins and languages
• Masks Chrome automation indicators

2. Request Header Spoofing

Headers automatically set:

Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
Accept-Language: en-US,en;q=0.9
Accept-Encoding: gzip, deflate, br
Sec-Ch-Ua: "Not_A Brand";v="8", "Chromium";v="120", "Google Chrome";v="120"
Sec-Ch-Ua-Mobile: ?0
Sec-Ch-Ua-Platform: "Windows"
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: none
Sec-Fetch-User: ?1
Upgrade-Insecure-Requests: 1

3. Human-like Behavior

• Random mouse movements
• Natural scrolling patterns
• Realistic timing delays
• Page interaction simulation

4. Challenge Detection

Automatically detects and handles:

• Cloudflare challenge forms
• Browser verification pages
• "Please wait" screens
• JavaScript challenges

🛠️ Advanced Usage

Custom User Agents

The system rotates between modern user agents:

• Chrome 120 on Windows
• Chrome 119 on Windows
• Chrome 120 on macOS
• Chrome 119 on macOS
• Chrome 120 on Linux
• Chrome 119 on Linux

Viewport Randomization

Random viewport sizes to appear more human:

• 1920x1080 (Full HD)
• 1366x768 (HD)
• 1536x864 (HD+)
• 1440x900 (WXGA+)
• 1280x720 (HD)

Proxy Integration

For additional stealth, configure a proxy:

# HTTP proxy
PROXY_URL=http://proxy-server:8080

# HTTPS proxy
PROXY_URL=https://proxy-server:8443

# SOCKS proxy
PROXY_URL=socks5://proxy-server:1080

🚨 Troubleshooting

Common Issues

1. Still getting blocked

   • Try using a proxy: PROXY_URL=http://your-proxy:8080
   • Increase delays by modifying the bypass module
   • Check if the site has additional protection layers

2. Timeout errors

   • Increase browser timeout: BROWSER_TTL=3600
   • Check network connectivity
   • Verify proxy configuration

3. Memory issues

   • Reduce browser pool: MAX_BROWSERS=2
   • Increase cleanup frequency
   • Monitor system resources

Debug Mode

Enable detailed logging by checking the server logs:

# View real-time logs
docker logs -f playwright-api

# Check specific bypass attempts
docker logs playwright-api | grep -i cloudflare

📈 Performance Tips

Optimization

1. Use caching: All bypassed content is cached
2. Batch requests: Process multiple URLs efficiently
3. Monitor resources: Use /status endpoint
4. Cleanup regularly: Use /force-cleanup-old

Monitoring

# Check system status
curl -H "X-API-Key: your-api-key" \
  "http://localhost:8000/status" | jq

# Monitor cache
curl -H "X-API-Key: your-api-key" \
  "http://localhost:8000/cache/stats" | jq

🔐 Security Considerations

Best Practices

1. Use HTTPS: Always use HTTPS for API communication
2. Secure API keys: Keep API keys secure and rotate regularly
3. Rate limiting: Respect rate limits to avoid detection
4. Proxy rotation: Use multiple proxies for high-volume scraping
5. User agent rotation: The system does this automatically

Legal Compliance

• Always respect robots.txt
• Follow website terms of service
• Implement appropriate delays between requests
• Use for legitimate purposes only

📚 Additional Resources

• Kameleo Cloudflare Bypass Guide
• Playwright Documentation
• Cloudflare Detection Methods

🤝 Contributing

To improve the bypass techniques:

1. Test with different Cloudflare configurations
2. Report successful/failed bypass attempts
3. Suggest new detection methods to counter
4. Contribute to the stealth scripts

---

Note: Cloudflare bypass techniques are constantly evolving. This implementation includes the latest known methods, but Cloudflare may update their detection systems. Regular updates and testing are recommended.