diff --git a/Dockers/puppeteer-healthcheck/README.md b/Dockers/puppeteer-healthcheck/README.md index a803d39..629fb2e 100644 --- a/Dockers/puppeteer-healthcheck/README.md +++ b/Dockers/puppeteer-healthcheck/README.md @@ -1,43 +1,55 @@ # Puppeteer API Healthcheck -A Docker container that monitors the Puppeteer API and automatically restarts the target container when the API becomes unresponsive. +A Docker container that monitors multiple Puppeteer API endpoints and automatically restarts the target containers when the APIs become unresponsive. ## Features -- Monitors the Puppeteer API endpoint every minute -- Automatically restarts the target container after 3 consecutive failures +- Monitors multiple Puppeteer API endpoints every minute +- Automatically restarts target containers after 3 consecutive failures - Configurable via environment variables - Comprehensive logging - Docker socket access for container management - Proper Docker permissions handling +- Backward compatibility with single-host configuration ## Environment Variables -| Variable | Default | Description | -| ------------------ | ------------------------------------ | -------------------------------- | -| `BASE_URL` | `https://puppeteer.workwithkora.com` | Base URL of the Puppeteer API | -| `TEST_URL` | `https://www.google.com` | URL to test the API with | -| `API_KEY` | `Q7Sd#hhFkyHy*T` | API key for authentication | -| `TARGET_CONTAINER` | `puppeteer-api` | Name of the container to restart | -| `CHECK_INTERVAL` | `60` | Health check interval in seconds | +| Variable | Default | Description | +| -------------------------- | ------------------------ | ---------------------------------------------- | +| `HOSTS` | `puppeteer-api` | Comma-separated list of hosts to monitor | +| `TEST_URL` | `https://www.google.com` | URL to test the API with | +| `API_KEY` | `Q7Sd#hhFkyHy*T` | API key for authentication | +| `CHECK_INTERVAL` | `60` | Health check interval in seconds | +| `MAX_CONSECUTIVE_FAILURES` | `3` | Number of failures before restarting container | +| `TIMEOUT` | `20` | Request timeout in seconds | + +### Legacy Variables (for backward compatibility) + +| Variable | Default | Description | +| ------------------ | ------------------------------------ | ---------------------------- | +| `BASE_URL` | `https://puppeteer.workwithkora.com` | Legacy single host URL | +| `TARGET_CONTAINER` | `puppeteer-api` | Legacy single container name | ## Usage -### Docker Run +### Multi-Host Configuration + +The healthcheck will monitor each host at `http://host:8000` and restart containers with the same name as the host. + +#### Docker Run ```bash docker run -d \ --name puppeteer-healthcheck \ -v /var/run/docker.sock:/var/run/docker.sock \ - -e BASE_URL="https://puppeteer.workwithkora.com" \ + -e HOSTS="host1,host2,host3" \ -e TEST_URL="https://www.google.com" \ -e API_KEY="your-api-key" \ - -e TARGET_CONTAINER="puppeteer-api" \ -e CHECK_INTERVAL="60" \ your-registry/puppeteer-healthcheck:latest ``` -### Docker Compose +#### Docker Compose ```yaml version: "3.8" @@ -49,24 +61,100 @@ services: volumes: - /var/run/docker.sock:/var/run/docker.sock environment: - - BASE_URL=https://puppeteer.workwithkora.com + - HOSTS=host1,host2,host3 - TEST_URL=https://www.google.com - API_KEY=your-api-key - - TARGET_CONTAINER=puppeteer-api - CHECK_INTERVAL=60 restart: unless-stopped - depends_on: - - puppeteer-api +``` + +### Single-Host Configuration (Legacy) + +For backward compatibility, you can still use the old single-host configuration: + +#### Docker Run + +```bash +docker run -d \ + --name puppeteer-healthcheck \ + -v /var/run/docker.sock:/var/run/docker.sock \ + -e BASE_URL="http://host1:8000" \ + -e TEST_URL="https://www.google.com" \ + -e API_KEY="your-api-key" \ + -e TARGET_CONTAINER="host1" \ + -e CHECK_INTERVAL="60" \ + your-registry/puppeteer-healthcheck:latest +``` + +#### Docker Compose + +```yaml +version: "3.8" + +services: + puppeteer-healthcheck: + build: . + container_name: puppeteer-healthcheck + volumes: + - /var/run/docker.sock:/var/run/docker.sock + environment: + - BASE_URL=http://host1:8000 + - TEST_URL=https://www.google.com + - API_KEY=your-api-key + - TARGET_CONTAINER=host1 + - CHECK_INTERVAL=60 + restart: unless-stopped ``` ## How It Works -1. The healthcheck container makes a GET request to the Puppeteer API every minute -2. It uses the configured test URL and API key for authentication -3. If the request fails (non-200 status or timeout), it increments a failure counter -4. After 3 consecutive failures, it attempts to restart the target container -5. If the restart is successful, the failure counter is reset -6. The process continues indefinitely +1. The healthcheck container reads the list of hosts from the `HOSTS` environment variable +2. For each host, it makes a GET request to `http://host:8000` every minute +3. It uses the configured test URL and API key for authentication +4. If the request fails (non-200 status or timeout), it increments a failure counter for that specific host +5. After 3 consecutive failures for a host, it attempts to restart the container with the same name as the host +6. If the restart is successful, the failure counter for that host is reset +7. The process continues indefinitely, monitoring all hosts independently + +## Example Scenarios + +### Monitoring Multiple Puppeteer Instances + +If you have multiple puppeteer-api containers running on different hosts: + +```bash +# Hosts: server1, server2, server3 +# Containers: server1, server2, server3 +docker run -d \ + --name puppeteer-healthcheck \ + -v /var/run/docker.sock:/var/run/docker.sock \ + -e HOSTS="server1,server2,server3" \ + -e API_KEY="your-api-key" \ + your-registry/puppeteer-healthcheck:latest +``` + +This will: + +- Check `http://server1:8000` and restart container `server1` if needed +- Check `http://server2:8000` and restart container `server2` if needed +- Check `http://server3:8000` and restart container `server3` if needed + +### Mixed Environment + +You can also monitor hosts with different names than their containers: + +```bash +# Hosts: api1.example.com, api2.example.com +# Containers: puppeteer-api-1, puppeteer-api-2 +docker run -d \ + --name puppeteer-healthcheck \ + -v /var/run/docker.sock:/var/run/docker.sock \ + -e HOSTS="api1.example.com,api2.example.com" \ + -e API_KEY="your-api-key" \ + your-registry/puppeteer-healthcheck:latest +``` + +**Note**: In this case, the container names must match the host names exactly. If they don't, you'll need to use separate healthcheck instances or modify the container names. ## Logging @@ -76,6 +164,14 @@ The container logs all health check activities to both stdout and a log file (`/ - WARNING: Failed health checks - ERROR: Container restart attempts and failures +Each log entry includes the host name for easy identification: + +``` +2024-01-15 10:30:00 - INFO - Performing health check for server1: http://server1:8000/?url=https%3A//www.google.com&skipCache=true +2024-01-15 10:30:01 - INFO - Health check passed for server1 - API is responding correctly +2024-01-15 10:30:02 - WARNING - Health check failed for server2 - Status code: 500 +``` + ## Security Considerations - The container requires access to the Docker socket to restart other containers @@ -127,13 +223,20 @@ This error occurs when the container cannot access the Docker socket. To fix: ### Container not found -- Ensure the `TARGET_CONTAINER` environment variable matches the exact name of your puppeteer-api container -- Verify the container is running and accessible +- Ensure the container names match the host names exactly +- Verify the containers are running and accessible +- Check that the `HOSTS` environment variable is set correctly ### API key issues - Verify the API key is correct and has the necessary permissions -- Check that the base URL is accessible from the container +- Check that the hosts are accessible from the container + +### Multiple host configuration + +- Ensure the `HOSTS` environment variable is a comma-separated list without spaces +- Each host should be accessible at `http://host:8000` +- Container names must match host names exactly ## Building @@ -143,4 +246,26 @@ docker build -t puppeteer-healthcheck . ## Version -Current version: 1.0.1 +Current version: 2.0.0 + +### Migration from v1.x + +To migrate from the single-host version to multi-host: + +1. **Replace `BASE_URL` and `TARGET_CONTAINER` with `HOSTS`**: + + ```bash + # Old + -e BASE_URL="http://server1:8000" -e TARGET_CONTAINER="server1" + + # New + -e HOSTS="server1" + ``` + +2. **For multiple hosts, add them to the `HOSTS` variable**: + + ```bash + -e HOSTS="server1,server2,server3" + ``` + +3. **Ensure container names match host names** (or rename containers accordingly) diff --git a/Dockers/puppeteer-healthcheck/healthcheck.py b/Dockers/puppeteer-healthcheck/healthcheck.py index 5d82dcd..78ad9ba 100644 --- a/Dockers/puppeteer-healthcheck/healthcheck.py +++ b/Dockers/puppeteer-healthcheck/healthcheck.py @@ -1,7 +1,7 @@ #!/usr/bin/env python3 """ Healthcheck script for Puppeteer API -Monitors the API endpoint and restarts the container if it fails +Monitors multiple API endpoints and restarts containers if they fail """ import os @@ -26,24 +26,46 @@ logger = logging.getLogger(__name__) class PuppeteerHealthcheck: def __init__(self): - self.base_url = os.getenv('BASE_URL', 'https://puppeteer.workwithkora.com') + self.hosts = self._parse_hosts() self.test_url = os.getenv('TEST_URL', 'https://www.google.com') self.api_key = os.getenv('API_KEY', 'Q7Sd#hhFkyHy*T') - self.target_container = os.getenv('TARGET_CONTAINER', 'puppeteer-api') self.check_interval = int(os.getenv('CHECK_INTERVAL', '60')) self.max_consecutive_failures = int(os.getenv('MAX_CONSECUTIVE_FAILURES', '3')) self.timeout = int(os.getenv('TIMEOUT', '20')) + # Track consecutive failures for each host + self.failure_counters = {host: 0 for host in self.hosts} + # Initialize Docker client with proper error handling self.docker_client = self._initialize_docker_client() logger.info(f"Healthcheck initialized with:") - logger.info(f" Base URL: {self.base_url}") + logger.info(f" Hosts: {', '.join(self.hosts)}") logger.info(f" Test URL: {self.test_url}") - logger.info(f" Target Container: {self.target_container}") logger.info(f" Check Interval: {self.check_interval} seconds") logger.info(f" Timeout: {self.timeout} seconds") + def _parse_hosts(self): + """Parse hosts from environment variable or use default""" + hosts_env = os.getenv('HOSTS', '') + if hosts_env: + # Split by comma and strip whitespace + hosts = [host.strip() for host in hosts_env.split(',') if host.strip()] + if hosts: + return hosts + + # Fallback to legacy BASE_URL for backward compatibility + base_url = os.getenv('BASE_URL', 'https://puppeteer.workwithkora.com') + if base_url.startswith('http://'): + # Extract host from http://host:port format + host_part = base_url.replace('http://', '').split('/')[0] + if ':' in host_part: + host = host_part.split(':')[0] + return [host] + + # Default fallback + return ['puppeteer-api'] + def _initialize_docker_client(self): """Initialize Docker client with proper error handling""" try: @@ -66,19 +88,20 @@ class PuppeteerHealthcheck: logger.error(f"Unexpected error initializing Docker client: {e}") raise - def perform_health_check(self): - """Perform the health check by making a request to the puppeteer API""" + def perform_health_check(self, host): + """Perform the health check for a specific host""" try: # Construct the URL with the test URL as a parameter + base_url = f"http://{host}:8000" encoded_test_url = quote(self.test_url, safe='') - url = f"{self.base_url}/?url={encoded_test_url}&skipCache=true" + url = f"{base_url}/?url={encoded_test_url}" headers = { 'x-api-key': self.api_key, 'User-Agent': 'Puppeteer-Healthcheck/1.0' } - logger.info(f"Performing health check: {url}") + logger.info(f"Performing health check for {host}: {url}") response = requests.get( url, @@ -92,77 +115,81 @@ class PuppeteerHealthcheck: try: response_data = response.json() if response_data.get('status') == 'error': - logger.warning(f"Health check failed - API returned status: error") + logger.warning(f"Health check failed for {host} - API returned status: error") logger.warning(f"Response: {response.text[:200]}...") return False else: - logger.info("Health check passed - API is responding correctly") + logger.info(f"Health check passed for {host} - API is responding correctly") return True except json.JSONDecodeError: # If response is not JSON, treat as success (backward compatibility) - logger.info("Health check passed - API is responding correctly (non-JSON response)") + logger.info(f"Health check passed for {host} - API is responding correctly (non-JSON response)") return True else: - logger.warning(f"Health check failed - Status code: {response.status_code}") + logger.warning(f"Health check failed for {host} - Status code: {response.status_code}") logger.warning(f"Response: {response.text[:200]}...") return False except requests.exceptions.RequestException as e: - logger.error(f"Health check failed - Request error: {e}") + logger.error(f"Health check failed for {host} - Request error: {e}") return False except Exception as e: - logger.error(f"Health check failed - Unexpected error: {e}") + logger.error(f"Health check failed for {host} - Unexpected error: {e}") return False - def restart_container(self): - """Restart the target puppeteer-api container""" + def restart_container(self, host): + """Restart the container for a specific host""" try: - logger.info(f"Attempting to restart container: {self.target_container}") + logger.info(f"Attempting to restart container: {host}") - # Get the container - container = self.docker_client.containers.get(self.target_container) + # Get the container (container name matches host name) + container = self.docker_client.containers.get(host) # Restart the container container.restart(timeout=30) - logger.info(f"Successfully restarted container: {self.target_container}") + logger.info(f"Successfully restarted container: {host}") return True except docker.errors.NotFound: - logger.error(f"Container not found: {self.target_container}") + logger.error(f"Container not found: {host}") return False except docker.errors.APIError as e: - logger.error(f"Docker API error while restarting container: {e}") + logger.error(f"Docker API error while restarting container {host}: {e}") return False except Exception as e: - logger.error(f"Unexpected error while restarting container: {e}") + logger.error(f"Unexpected error while restarting container {host}: {e}") return False + def check_host(self, host): + """Check a single host and handle failures""" + if self.perform_health_check(host): + if self.failure_counters[host] > 0: + logger.info(f"Health check passed for {host} - resetting failure counter") + self.failure_counters[host] = 0 + else: + self.failure_counters[host] += 1 + logger.warning(f"Health check failed for {host} - consecutive failures: {self.failure_counters[host]}") + + # Restart container if we've had too many consecutive failures + if self.failure_counters[host] >= self.max_consecutive_failures: + logger.error(f"Health check failed {self.failure_counters[host]} times consecutively for {host} - restarting container") + if self.restart_container(host): + self.failure_counters[host] = 0 + logger.info(f"Container {host} restarted successfully - resetting failure counter") + else: + logger.error(f"Failed to restart container {host}") + def run(self): """Main loop for the healthcheck""" logger.info("Starting Puppeteer API healthcheck service") - consecutive_failures = 0 - while True: try: - # Perform health check - if self.perform_health_check(): - consecutive_failures = 0 - logger.info("Health check passed - resetting failure counter") - else: - consecutive_failures += 1 - logger.warning(f"Health check failed - consecutive failures: {consecutive_failures}") - - # Restart container if we've had too many consecutive failures - if consecutive_failures >= self.max_consecutive_failures: - logger.error(f"Health check failed {consecutive_failures} times consecutively - restarting container") - if self.restart_container(): - consecutive_failures = 0 - logger.info("Container restarted successfully - resetting failure counter") - else: - logger.error("Failed to restart container") + # Check each host + for host in self.hosts: + self.check_host(host) # Wait before next check logger.info(f"Waiting {self.check_interval} seconds before next health check...")