302 lines
6.9 KiB
Markdown
302 lines
6.9 KiB
Markdown
# Cloudflare Bypass Examples & Usage Guide
|
|
|
|
This guide demonstrates how to use the advanced Cloudflare bypass features in the Playwright API.
|
|
|
|
## 🚀 Quick Start
|
|
|
|
### Basic Usage
|
|
|
|
All existing endpoints now automatically include Cloudflare bypass:
|
|
|
|
```bash
|
|
# Visit a Cloudflare-protected site
|
|
curl -H "X-API-Key: your-api-key" \
|
|
"http://localhost:8000/?url=https://cloudflare-protected-site.com"
|
|
|
|
# Extract SEO from protected site
|
|
curl -H "X-API-Key: your-api-key" \
|
|
"http://localhost:8000/seo?url=https://cloudflare-protected-site.com"
|
|
|
|
# Extract meta tags from protected site
|
|
curl -H "X-API-Key: your-api-key" \
|
|
"http://localhost:8000/meta?url=https://cloudflare-protected-site.com"
|
|
```
|
|
|
|
### Test Cloudflare Bypass
|
|
|
|
Use the dedicated test endpoint to verify bypass effectiveness:
|
|
|
|
```bash
|
|
# Test bypass on a specific URL
|
|
curl -H "X-API-Key: your-api-key" \
|
|
"http://localhost:8000/test-cloudflare?url=https://cloudflare-protected-site.com"
|
|
```
|
|
|
|
## 🔧 Configuration Options
|
|
|
|
### Environment Variables
|
|
|
|
```bash
|
|
# Enable/disable Cloudflare bypass (default: true)
|
|
ENABLE_CLOUDFLARE_BYPASS=true
|
|
|
|
# Optional proxy for additional stealth
|
|
PROXY_URL=http://proxy-server:8080
|
|
|
|
# Other existing variables still work
|
|
API_KEY=your-api-key
|
|
MAX_BROWSERS=3
|
|
BROWSER_TTL=1800
|
|
```
|
|
|
|
### Docker Example with Proxy
|
|
|
|
```bash
|
|
docker run -d \
|
|
--name playwright-api \
|
|
-p 8000:8000 \
|
|
-e API_KEY=your-api-key \
|
|
-e ENABLE_CLOUDFLARE_BYPASS=true \
|
|
-e PROXY_URL=http://your-proxy:8080 \
|
|
-v /path/to/cache:/db \
|
|
playwright-api
|
|
```
|
|
|
|
## 📊 Response Examples
|
|
|
|
### Successful Bypass
|
|
|
|
```json
|
|
{
|
|
"status": "success",
|
|
"url": "https://cloudflare-protected-site.com",
|
|
"final_url": "https://cloudflare-protected-site.com",
|
|
"title": "Protected Site - Home",
|
|
"cloudflare_bypassed": true,
|
|
"cloudflare_indicators": {
|
|
"has_cloudflare_title": false,
|
|
"has_challenge_form": false,
|
|
"has_cf_wrapper": false,
|
|
"has_please_wait": false,
|
|
"has_browser_verification": false
|
|
},
|
|
"content_length": 45678
|
|
}
|
|
```
|
|
|
|
### Failed Bypass
|
|
|
|
```json
|
|
{
|
|
"status": "error",
|
|
"error": "Failed to bypass Cloudflare protection",
|
|
"url": "https://cloudflare-protected-site.com",
|
|
"cloudflare_detected": true
|
|
}
|
|
```
|
|
|
|
## 🧪 Testing Examples
|
|
|
|
### Using the Test Script
|
|
|
|
```bash
|
|
# Test a known Cloudflare-protected site
|
|
python test_cloudflare_bypass.py \
|
|
http://localhost:8000 \
|
|
your-api-key \
|
|
https://example-cloudflare-site.com
|
|
|
|
# Test multiple sites
|
|
for site in "site1.com" "site2.com" "site3.com"; do
|
|
python test_cloudflare_bypass.py \
|
|
http://localhost:8000 \
|
|
your-api-key \
|
|
"https://$site"
|
|
done
|
|
```
|
|
|
|
### Manual Testing with curl
|
|
|
|
```bash
|
|
# Test bypass endpoint
|
|
curl -H "X-API-Key: your-api-key" \
|
|
"http://localhost:8000/test-cloudflare?url=https://example.com" | jq
|
|
|
|
# Compare with regular endpoint
|
|
curl -H "X-API-Key: your-api-key" \
|
|
"http://localhost:8000/?url=https://example.com" | jq '.content | length'
|
|
```
|
|
|
|
## 🔍 Bypass Techniques Explained
|
|
|
|
### 1. Browser Fingerprinting Protection
|
|
|
|
The API automatically:
|
|
|
|
- Removes `navigator.webdriver` property
|
|
- Overrides automation detection methods
|
|
- Spoofs browser plugins and languages
|
|
- Masks Chrome automation indicators
|
|
|
|
### 2. Request Header Spoofing
|
|
|
|
Headers automatically set:
|
|
|
|
```http
|
|
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
|
|
Accept-Language: en-US,en;q=0.9
|
|
Accept-Encoding: gzip, deflate, br
|
|
Sec-Ch-Ua: "Not_A Brand";v="8", "Chromium";v="120", "Google Chrome";v="120"
|
|
Sec-Ch-Ua-Mobile: ?0
|
|
Sec-Ch-Ua-Platform: "Windows"
|
|
Sec-Fetch-Dest: document
|
|
Sec-Fetch-Mode: navigate
|
|
Sec-Fetch-Site: none
|
|
Sec-Fetch-User: ?1
|
|
Upgrade-Insecure-Requests: 1
|
|
```
|
|
|
|
### 3. Human-like Behavior
|
|
|
|
- Random mouse movements
|
|
- Natural scrolling patterns
|
|
- Realistic timing delays
|
|
- Page interaction simulation
|
|
|
|
### 4. Challenge Detection
|
|
|
|
Automatically detects and handles:
|
|
|
|
- Cloudflare challenge forms
|
|
- Browser verification pages
|
|
- "Please wait" screens
|
|
- JavaScript challenges
|
|
|
|
## 🛠️ Advanced Usage
|
|
|
|
### Custom User Agents
|
|
|
|
The system rotates between modern user agents:
|
|
|
|
- Chrome 120 on Windows
|
|
- Chrome 119 on Windows
|
|
- Chrome 120 on macOS
|
|
- Chrome 119 on macOS
|
|
- Chrome 120 on Linux
|
|
- Chrome 119 on Linux
|
|
|
|
### Viewport Randomization
|
|
|
|
Random viewport sizes to appear more human:
|
|
|
|
- 1920x1080 (Full HD)
|
|
- 1366x768 (HD)
|
|
- 1536x864 (HD+)
|
|
- 1440x900 (WXGA+)
|
|
- 1280x720 (HD)
|
|
|
|
### Proxy Integration
|
|
|
|
For additional stealth, configure a proxy:
|
|
|
|
```bash
|
|
# HTTP proxy
|
|
PROXY_URL=http://proxy-server:8080
|
|
|
|
# HTTPS proxy
|
|
PROXY_URL=https://proxy-server:8443
|
|
|
|
# SOCKS proxy
|
|
PROXY_URL=socks5://proxy-server:1080
|
|
```
|
|
|
|
## 🚨 Troubleshooting
|
|
|
|
### Common Issues
|
|
|
|
1. **Still getting blocked**
|
|
|
|
- Try using a proxy: `PROXY_URL=http://your-proxy:8080`
|
|
- Increase delays by modifying the bypass module
|
|
- Check if the site has additional protection layers
|
|
|
|
2. **Timeout errors**
|
|
|
|
- Increase browser timeout: `BROWSER_TTL=3600`
|
|
- Check network connectivity
|
|
- Verify proxy configuration
|
|
|
|
3. **Memory issues**
|
|
- Reduce browser pool: `MAX_BROWSERS=2`
|
|
- Increase cleanup frequency
|
|
- Monitor system resources
|
|
|
|
### Debug Mode
|
|
|
|
Enable detailed logging by checking the server logs:
|
|
|
|
```bash
|
|
# View real-time logs
|
|
docker logs -f playwright-api
|
|
|
|
# Check specific bypass attempts
|
|
docker logs playwright-api | grep -i cloudflare
|
|
```
|
|
|
|
## 📈 Performance Tips
|
|
|
|
### Optimization
|
|
|
|
1. **Use caching**: All bypassed content is cached
|
|
2. **Batch requests**: Process multiple URLs efficiently
|
|
3. **Monitor resources**: Use `/status` endpoint
|
|
4. **Cleanup regularly**: Use `/force-cleanup-old`
|
|
|
|
### Monitoring
|
|
|
|
```bash
|
|
# Check system status
|
|
curl -H "X-API-Key: your-api-key" \
|
|
"http://localhost:8000/status" | jq
|
|
|
|
# Monitor cache
|
|
curl -H "X-API-Key: your-api-key" \
|
|
"http://localhost:8000/cache/stats" | jq
|
|
```
|
|
|
|
## 🔐 Security Considerations
|
|
|
|
### Best Practices
|
|
|
|
1. **Use HTTPS**: Always use HTTPS for API communication
|
|
2. **Secure API keys**: Keep API keys secure and rotate regularly
|
|
3. **Rate limiting**: Respect rate limits to avoid detection
|
|
4. **Proxy rotation**: Use multiple proxies for high-volume scraping
|
|
5. **User agent rotation**: The system does this automatically
|
|
|
|
### Legal Compliance
|
|
|
|
- Always respect robots.txt
|
|
- Follow website terms of service
|
|
- Implement appropriate delays between requests
|
|
- Use for legitimate purposes only
|
|
|
|
## 📚 Additional Resources
|
|
|
|
- [Kameleo Cloudflare Bypass Guide](https://kameleo.io/blog/how-to-bypass-cloudflare-with-playwright)
|
|
- [Playwright Documentation](https://playwright.dev/)
|
|
- [Cloudflare Detection Methods](https://developers.cloudflare.com/bots/)
|
|
|
|
## 🤝 Contributing
|
|
|
|
To improve the bypass techniques:
|
|
|
|
1. Test with different Cloudflare configurations
|
|
2. Report successful/failed bypass attempts
|
|
3. Suggest new detection methods to counter
|
|
4. Contribute to the stealth scripts
|
|
|
|
---
|
|
|
|
**Note**: Cloudflare bypass techniques are constantly evolving. This implementation includes the latest known methods, but Cloudflare may update their detection systems. Regular updates and testing are recommended.
|