potential bypass cloudflare

Dockers/puppeteer-api/README.md

@@ -1,6 +1,80 @@
 # Playwright API Server
 
-A FastAPI-based server that uses Playwright to scrape websites and extract various types of data.
+A FastAPI-based server that uses Playwright to scrape websites and extract various types of data. **Now with advanced Cloudflare bypass capabilities!**
+
+## 🛡️ Cloudflare Bypass Features (v4.0)
+
+This API now includes sophisticated Cloudflare bypass techniques based on the latest anti-detection methods:
+
+### Key Bypass Features
+
+- **🔄 Dynamic User Agents**: Rotates between modern, realistic user agents
+- **🎭 Stealth Mode**: Comprehensive browser fingerprint spoofing
+- **🤖 Human-like Behavior**: Simulates real user interactions
+- **🔒 Anti-Detection Scripts**: Injects scripts to bypass automation detection
+- **🌐 Proxy Support**: Optional proxy configuration for additional stealth
+- **⏱️ Challenge Handling**: Automatic Cloudflare challenge detection and waiting
+- **📱 Realistic Viewports**: Random viewport sizes to appear more human
+- **🌍 Geolocation Spoofing**: Simulates realistic location data
+
+### Bypass Techniques Implemented
+
+1. **Browser Fingerprinting Protection**
+
+   - Removes `navigator.webdriver` property
+   - Overrides automation detection methods
+   - Spoofs browser plugins and languages
+   - Masks Chrome automation indicators
+
+2. **Request Header Spoofing**
+
+   - Modern `Sec-Ch-Ua` headers
+   - Realistic `Accept` and `Accept-Language` headers
+   - Proper `Sec-Fetch-*` headers
+   - Cache control headers
+
+3. **Human-like Behavior**
+
+   - Random mouse movements
+   - Natural scrolling patterns
+   - Realistic timing delays
+   - Page interaction simulation
+
+4. **Challenge Detection & Handling**
+   - Automatic Cloudflare challenge detection
+   - Intelligent waiting for challenge completion
+   - Multiple retry attempts with backoff
+   - Success/failure reporting
+
+### Testing Cloudflare Bypass
+
+Use the new `/test-cloudflare` endpoint to test bypass effectiveness:
+
+```bash
+# Test Cloudflare bypass on a protected site
+curl -H "X-API-Key: your-api-key" \
+  "http://localhost:8000/test-cloudflare?url=https://cloudflare-protected-site.com"
+```
+
+Response includes detailed bypass status:
+
+```json
+{
+  "status": "success",
+  "url": "https://example.com",
+  "final_url": "https://example.com",
+  "title": "Example Domain",
+  "cloudflare_bypassed": true,
+  "cloudflare_indicators": {
+    "has_cloudflare_title": false,
+    "has_challenge_form": false,
+    "has_cf_wrapper": false,
+    "has_please_wait": false,
+    "has_browser_verification": false
+  },
+  "content_length": 12345
+}
+```
 
 ## Recent Migration (v3.0)
 
@@ -83,6 +157,10 @@ CACHE_EXPIRY_HOURS=36            # Cache expiry in hours (default: 36)
 CLEANUP_CRON=0 3 * * *           # Cache cleanup schedule (default: daily at 3 AM)
 RATE_LIMIT_MINUTE=60             # Requests per minute (default: 60)
 
+# Cloudflare Bypass Configuration
+ENABLE_CLOUDFLARE_BYPASS=true   # Enable Cloudflare bypass (default: true)
+PROXY_URL=                      # Optional proxy URL for additional stealth (e.g., http://proxy:8080)
+
 # Database Configuration
 # If all PostgreSQL credentials are provided, PostgreSQL will be used
 # Otherwise, SQLite will be used as fallback
@@ -97,9 +175,10 @@ POSTGRES_PASSWORD=              # PostgreSQL password (optional)
 
 ### Core Endpoints
 
-- `GET /` - Visit URL and get HTML content
-- `GET /seo` - Extract SEO information
-- `GET /meta` - Extract meta tags and Open Graph data
+- `GET /` - Visit URL and get HTML content (with Cloudflare bypass)
+- `GET /seo` - Extract SEO information (with Cloudflare bypass)
+- `GET /meta` - Extract meta tags and Open Graph data (with Cloudflare bypass)
+- `GET /test-cloudflare` - Test Cloudflare bypass functionality on a specific URL
 - `GET /resulting-url` - Get the final URL after navigation (handles redirects)
 
 ### Management Endpoints
@@ -128,6 +207,9 @@ curl -H "X-API-Key: your-api-key" "http://localhost:8000/resulting-url?url=https
 # Get system status
 curl -H "X-API-Key: your-api-key" "http://localhost:8000/status"
 
+# Test Cloudflare bypass
+curl -H "X-API-Key: your-api-key" "http://localhost:8000/test-cloudflare?url=https://cloudflare-protected-site.com"
+
 # Force cleanup old browser/page instances
 curl -X POST -H "X-API-Key: your-api-key" "http://localhost:8000/force-cleanup-old"
 ```