potential bypass cloudflare

2025-07-18 11:32:13 +02:00
parent a97392f421
commit 085f447e49
8 changed files with 1022 additions and 61 deletions
@@ -0,0 +1,301 @@
+# Cloudflare Bypass Examples & Usage Guide
+
+This guide demonstrates how to use the advanced Cloudflare bypass features in the Playwright API.
+
+## 🚀 Quick Start
+
+### Basic Usage
+
+All existing endpoints now automatically include Cloudflare bypass:
+
+```bash
+# Visit a Cloudflare-protected site
+curl -H "X-API-Key: your-api-key" \
+  "http://localhost:8000/?url=https://cloudflare-protected-site.com"
+
+# Extract SEO from protected site
+curl -H "X-API-Key: your-api-key" \
+  "http://localhost:8000/seo?url=https://cloudflare-protected-site.com"
+
+# Extract meta tags from protected site
+curl -H "X-API-Key: your-api-key" \
+  "http://localhost:8000/meta?url=https://cloudflare-protected-site.com"
+```
+
+### Test Cloudflare Bypass
+
+Use the dedicated test endpoint to verify bypass effectiveness:
+
+```bash
+# Test bypass on a specific URL
+curl -H "X-API-Key: your-api-key" \
+  "http://localhost:8000/test-cloudflare?url=https://cloudflare-protected-site.com"
+```
+
+## 🔧 Configuration Options
+
+### Environment Variables
+
+```bash
+# Enable/disable Cloudflare bypass (default: true)
+ENABLE_CLOUDFLARE_BYPASS=true
+
+# Optional proxy for additional stealth
+PROXY_URL=http://proxy-server:8080
+
+# Other existing variables still work
+API_KEY=your-api-key
+MAX_BROWSERS=3
+BROWSER_TTL=1800
+```
+
+### Docker Example with Proxy
+
+```bash
+docker run -d \
+  --name playwright-api \
+  -p 8000:8000 \
+  -e API_KEY=your-api-key \
+  -e ENABLE_CLOUDFLARE_BYPASS=true \
+  -e PROXY_URL=http://your-proxy:8080 \
+  -v /path/to/cache:/db \
+  playwright-api
+```
+
+## 📊 Response Examples
+
+### Successful Bypass
+
+```json
+{
+  "status": "success",
+  "url": "https://cloudflare-protected-site.com",
+  "final_url": "https://cloudflare-protected-site.com",
+  "title": "Protected Site - Home",
+  "cloudflare_bypassed": true,
+  "cloudflare_indicators": {
+    "has_cloudflare_title": false,
+    "has_challenge_form": false,
+    "has_cf_wrapper": false,
+    "has_please_wait": false,
+    "has_browser_verification": false
+  },
+  "content_length": 45678
+}
+```
+
+### Failed Bypass
+
+```json
+{
+  "status": "error",
+  "error": "Failed to bypass Cloudflare protection",
+  "url": "https://cloudflare-protected-site.com",
+  "cloudflare_detected": true
+}
+```
+
+## 🧪 Testing Examples
+
+### Using the Test Script
+
+```bash
+# Test a known Cloudflare-protected site
+python test_cloudflare_bypass.py \
+  http://localhost:8000 \
+  your-api-key \
+  https://example-cloudflare-site.com
+
+# Test multiple sites
+for site in "site1.com" "site2.com" "site3.com"; do
+  python test_cloudflare_bypass.py \
+    http://localhost:8000 \
+    your-api-key \
+    "https://$site"
+done
+```
+
+### Manual Testing with curl
+
+```bash
+# Test bypass endpoint
+curl -H "X-API-Key: your-api-key" \
+  "http://localhost:8000/test-cloudflare?url=https://example.com" | jq
+
+# Compare with regular endpoint
+curl -H "X-API-Key: your-api-key" \
+  "http://localhost:8000/?url=https://example.com" | jq '.content | length'
+```
+
+## 🔍 Bypass Techniques Explained
+
+### 1. Browser Fingerprinting Protection
+
+The API automatically:
+
+- Removes `navigator.webdriver` property
+- Overrides automation detection methods
+- Spoofs browser plugins and languages
+- Masks Chrome automation indicators
+
+### 2. Request Header Spoofing
+
+Headers automatically set:
+
+```http
+Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
+Accept-Language: en-US,en;q=0.9
+Accept-Encoding: gzip, deflate, br
+Sec-Ch-Ua: "Not_A Brand";v="8", "Chromium";v="120", "Google Chrome";v="120"
+Sec-Ch-Ua-Mobile: ?0
+Sec-Ch-Ua-Platform: "Windows"
+Sec-Fetch-Dest: document
+Sec-Fetch-Mode: navigate
+Sec-Fetch-Site: none
+Sec-Fetch-User: ?1
+Upgrade-Insecure-Requests: 1
+```
+
+### 3. Human-like Behavior
+
+- Random mouse movements
+- Natural scrolling patterns
+- Realistic timing delays
+- Page interaction simulation
+
+### 4. Challenge Detection
+
+Automatically detects and handles:
+
+- Cloudflare challenge forms
+- Browser verification pages
+- "Please wait" screens
+- JavaScript challenges
+
+## 🛠️ Advanced Usage
+
+### Custom User Agents
+
+The system rotates between modern user agents:
+
+- Chrome 120 on Windows
+- Chrome 119 on Windows
+- Chrome 120 on macOS
+- Chrome 119 on macOS
+- Chrome 120 on Linux
+- Chrome 119 on Linux
+
+### Viewport Randomization
+
+Random viewport sizes to appear more human:
+
+- 1920x1080 (Full HD)
+- 1366x768 (HD)
+- 1536x864 (HD+)
+- 1440x900 (WXGA+)
+- 1280x720 (HD)
+
+### Proxy Integration
+
+For additional stealth, configure a proxy:
+
+```bash
+# HTTP proxy
+PROXY_URL=http://proxy-server:8080
+
+# HTTPS proxy
+PROXY_URL=https://proxy-server:8443
+
+# SOCKS proxy
+PROXY_URL=socks5://proxy-server:1080
+```
+
+## 🚨 Troubleshooting
+
+### Common Issues
+
+1. **Still getting blocked**
+
+   - Try using a proxy: `PROXY_URL=http://your-proxy:8080`
+   - Increase delays by modifying the bypass module
+   - Check if the site has additional protection layers
+
+2. **Timeout errors**
+
+   - Increase browser timeout: `BROWSER_TTL=3600`
+   - Check network connectivity
+   - Verify proxy configuration
+
+3. **Memory issues**
+   - Reduce browser pool: `MAX_BROWSERS=2`
+   - Increase cleanup frequency
+   - Monitor system resources
+
+### Debug Mode
+
+Enable detailed logging by checking the server logs:
+
+```bash
+# View real-time logs
+docker logs -f playwright-api
+
+# Check specific bypass attempts
+docker logs playwright-api | grep -i cloudflare
+```
+
+## 📈 Performance Tips
+
+### Optimization
+
+1. **Use caching**: All bypassed content is cached
+2. **Batch requests**: Process multiple URLs efficiently
+3. **Monitor resources**: Use `/status` endpoint
+4. **Cleanup regularly**: Use `/force-cleanup-old`
+
+### Monitoring
+
+```bash
+# Check system status
+curl -H "X-API-Key: your-api-key" \
+  "http://localhost:8000/status" | jq
+
+# Monitor cache
+curl -H "X-API-Key: your-api-key" \
+  "http://localhost:8000/cache/stats" | jq
+```
+
+## 🔐 Security Considerations
+
+### Best Practices
+
+1. **Use HTTPS**: Always use HTTPS for API communication
+2. **Secure API keys**: Keep API keys secure and rotate regularly
+3. **Rate limiting**: Respect rate limits to avoid detection
+4. **Proxy rotation**: Use multiple proxies for high-volume scraping
+5. **User agent rotation**: The system does this automatically
+
+### Legal Compliance
+
+- Always respect robots.txt
+- Follow website terms of service
+- Implement appropriate delays between requests
+- Use for legitimate purposes only
+
+## 📚 Additional Resources
+
+- [Kameleo Cloudflare Bypass Guide](https://kameleo.io/blog/how-to-bypass-cloudflare-with-playwright)
+- [Playwright Documentation](https://playwright.dev/)
+- [Cloudflare Detection Methods](https://developers.cloudflare.com/bots/)
+
+## 🤝 Contributing
+
+To improve the bypass techniques:
+
+1. Test with different Cloudflare configurations
+2. Report successful/failed bypass attempts
+3. Suggest new detection methods to counter
+4. Contribute to the stealth scripts
+
+---
+
+**Note**: Cloudflare bypass techniques are constantly evolving. This implementation includes the latest known methods, but Cloudflare may update their detection systems. Regular updates and testing are recommended.