# Cloudflare Bypass Examples & Usage Guide

This guide demonstrates how to use the advanced Cloudflare bypass features in the Playwright API.

## 🚀 Quick Start

### Basic Usage

All existing endpoints now automatically include Cloudflare bypass:

```bash
# Visit a Cloudflare-protected site
curl -H "X-API-Key: your-api-key" \
  "http://localhost:8000/?url=https://cloudflare-protected-site.com"

# Extract SEO from protected site
curl -H "X-API-Key: your-api-key" \
  "http://localhost:8000/seo?url=https://cloudflare-protected-site.com"

# Extract meta tags from protected site
curl -H "X-API-Key: your-api-key" \
  "http://localhost:8000/meta?url=https://cloudflare-protected-site.com"
```

### Test Cloudflare Bypass

Use the dedicated test endpoint to verify bypass effectiveness:

```bash
# Test bypass on a specific URL
curl -H "X-API-Key: your-api-key" \
  "http://localhost:8000/test-cloudflare?url=https://cloudflare-protected-site.com"
```

## 🔧 Configuration Options

### Environment Variables

```bash
# Enable/disable Cloudflare bypass (default: true)
ENABLE_CLOUDFLARE_BYPASS=true

# Optional proxy for additional stealth
PROXY_URL=http://proxy-server:8080

# Other existing variables still work
API_KEY=your-api-key
MAX_BROWSERS=3
BROWSER_TTL=1800
```

### Docker Example with Proxy

```bash
docker run -d \
  --name playwright-api \
  -p 8000:8000 \
  -e API_KEY=your-api-key \
  -e ENABLE_CLOUDFLARE_BYPASS=true \
  -e PROXY_URL=http://your-proxy:8080 \
  -v /path/to/cache:/db \
  playwright-api
```

## 📊 Response Examples

### Successful Bypass

```json
{
  "status": "success",
  "url": "https://cloudflare-protected-site.com",
  "final_url": "https://cloudflare-protected-site.com",
  "title": "Protected Site - Home",
  "cloudflare_bypassed": true,
  "cloudflare_indicators": {
    "has_cloudflare_title": false,
    "has_challenge_form": false,
    "has_cf_wrapper": false,
    "has_please_wait": false,
    "has_browser_verification": false
  },
  "content_length": 45678
}
```

### Failed Bypass

```json
{
  "status": "error",
  "error": "Failed to bypass Cloudflare protection",
  "url": "https://cloudflare-protected-site.com",
  "cloudflare_detected": true
}
```

## 🧪 Testing Examples

### Using the Test Script

```bash
# Test a known Cloudflare-protected site
python test_cloudflare_bypass.py \
  http://localhost:8000 \
  your-api-key \
  https://example-cloudflare-site.com

# Test multiple sites
for site in "site1.com" "site2.com" "site3.com"; do
  python test_cloudflare_bypass.py \
    http://localhost:8000 \
    your-api-key \
    "https://$site"
done
```

### Manual Testing with curl

```bash
# Test bypass endpoint
curl -H "X-API-Key: your-api-key" \
  "http://localhost:8000/test-cloudflare?url=https://example.com" | jq

# Compare with regular endpoint
curl -H "X-API-Key: your-api-key" \
  "http://localhost:8000/?url=https://example.com" | jq '.content | length'
```

## 🔍 Bypass Techniques Explained

### 1. Browser Fingerprinting Protection

The API automatically:

- Removes `navigator.webdriver` property
- Overrides automation detection methods
- Spoofs browser plugins and languages
- Masks Chrome automation indicators

### 2. Request Header Spoofing

Headers automatically set:

```http
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
Accept-Language: en-US,en;q=0.9
Accept-Encoding: gzip, deflate, br
Sec-Ch-Ua: "Not_A Brand";v="8", "Chromium";v="120", "Google Chrome";v="120"
Sec-Ch-Ua-Mobile: ?0
Sec-Ch-Ua-Platform: "Windows"
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: none
Sec-Fetch-User: ?1
Upgrade-Insecure-Requests: 1
```

### 3. Human-like Behavior

- Random mouse movements
- Natural scrolling patterns
- Realistic timing delays
- Page interaction simulation

### 4. Challenge Detection

Automatically detects and handles:

- Cloudflare challenge forms
- Browser verification pages
- "Please wait" screens
- JavaScript challenges

## 🛠️ Advanced Usage

### Custom User Agents

The system rotates between modern user agents:

- Chrome 120 on Windows
- Chrome 119 on Windows
- Chrome 120 on macOS
- Chrome 119 on macOS
- Chrome 120 on Linux
- Chrome 119 on Linux

### Viewport Randomization

Random viewport sizes to appear more human:

- 1920x1080 (Full HD)
- 1366x768 (HD)
- 1536x864 (HD+)
- 1440x900 (WXGA+)
- 1280x720 (HD)

### Proxy Integration

For additional stealth, configure a proxy:

```bash
# HTTP proxy
PROXY_URL=http://proxy-server:8080

# HTTPS proxy
PROXY_URL=https://proxy-server:8443

# SOCKS proxy
PROXY_URL=socks5://proxy-server:1080
```

## 🚨 Troubleshooting

### Common Issues

1. **Still getting blocked**

   - Try using a proxy: `PROXY_URL=http://your-proxy:8080`
   - Increase delays by modifying the bypass module
   - Check if the site has additional protection layers

2. **Timeout errors**

   - Increase browser timeout: `BROWSER_TTL=3600`
   - Check network connectivity
   - Verify proxy configuration

3. **Memory issues**
   - Reduce browser pool: `MAX_BROWSERS=2`
   - Increase cleanup frequency
   - Monitor system resources

### Debug Mode

Enable detailed logging by checking the server logs:

```bash
# View real-time logs
docker logs -f playwright-api

# Check specific bypass attempts
docker logs playwright-api | grep -i cloudflare
```

## 📈 Performance Tips

### Optimization

1. **Use caching**: All bypassed content is cached
2. **Batch requests**: Process multiple URLs efficiently
3. **Monitor resources**: Use `/status` endpoint
4. **Cleanup regularly**: Use `/force-cleanup-old`

### Monitoring

```bash
# Check system status
curl -H "X-API-Key: your-api-key" \
  "http://localhost:8000/status" | jq

# Monitor cache
curl -H "X-API-Key: your-api-key" \
  "http://localhost:8000/cache/stats" | jq
```

## 🔐 Security Considerations

### Best Practices

1. **Use HTTPS**: Always use HTTPS for API communication
2. **Secure API keys**: Keep API keys secure and rotate regularly
3. **Rate limiting**: Respect rate limits to avoid detection
4. **Proxy rotation**: Use multiple proxies for high-volume scraping
5. **User agent rotation**: The system does this automatically

### Legal Compliance

- Always respect robots.txt
- Follow website terms of service
- Implement appropriate delays between requests
- Use for legitimate purposes only

## 📚 Additional Resources

- [Kameleo Cloudflare Bypass Guide](https://kameleo.io/blog/how-to-bypass-cloudflare-with-playwright)
- [Playwright Documentation](https://playwright.dev/)
- [Cloudflare Detection Methods](https://developers.cloudflare.com/bots/)

## 🤝 Contributing

To improve the bypass techniques:

1. Test with different Cloudflare configurations
2. Report successful/failed bypass attempts
3. Suggest new detection methods to counter
4. Contribute to the stealth scripts

---

**Note**: Cloudflare bypass techniques are constantly evolving. This implementation includes the latest known methods, but Cloudflare may update their detection systems. Regular updates and testing are recommended.