Bram/projects
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 3 Wiki Activity

allow multiple hosts for healthcheck
Build and Push Docker Images / build-and-push (push) Successful in 23s
Details

Browse Source
This commit is contained in:
Bram
2025-07-16 10:13:53 +02:00
parent 22b06f775d
commit 55e6c2237f
2 changed files with 222 additions and 70 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Dockers/puppeteer-healthcheck/README.md
+153 -28
View File
@@ -1,43 +1,55 @@
# Puppeteer API Healthcheck # Puppeteer API Healthcheck
A Docker container that monitors the Puppeteer API and automatically restarts the target container when the API becomes unresponsive. A Docker container that monitors multiple Puppeteer API endpoints and automatically restarts the target containers when the APIs become unresponsive.
## Features ## Features
- Monitors the Puppeteer API endpoint every minute - Monitors multiple Puppeteer API endpoints every minute
- Automatically restarts the target container after 3 consecutive failures - Automatically restarts target containers after 3 consecutive failures
- Configurable via environment variables - Configurable via environment variables
- Comprehensive logging - Comprehensive logging
- Docker socket access for container management - Docker socket access for container management
- Proper Docker permissions handling - Proper Docker permissions handling
- Backward compatibility with single-host configuration
## Environment Variables ## Environment Variables
| Variable | Default | Description | | Variable | Default | Description |
| ------------------ | ------------------------------------ | -------------------------------- | | -------------------------- | ------------------------ | ---------------------------------------------- |
| `BASE_URL` | `https://puppeteer.workwithkora.com` | Base URL of the Puppeteer API | | `HOSTS` | `puppeteer-api` | Comma-separated list of hosts to monitor |
| `TEST_URL` | `https://www.google.com` | URL to test the API with | | `TEST_URL` | `https://www.google.com` | URL to test the API with |
| `API_KEY` | `Q7Sd#hhFkyHy*T` | API key for authentication | | `API_KEY` | `Q7Sd#hhFkyHy*T` | API key for authentication |
| `TARGET_CONTAINER` | `puppeteer-api` | Name of the container to restart | | `CHECK_INTERVAL` | `60` | Health check interval in seconds |
| `CHECK_INTERVAL` | `60` | Health check interval in seconds | | `MAX_CONSECUTIVE_FAILURES` | `3` | Number of failures before restarting container |
| `TIMEOUT` | `20` | Request timeout in seconds |
### Legacy Variables (for backward compatibility)
| Variable | Default | Description |
| ------------------ | ------------------------------------ | ---------------------------- |
| `BASE_URL` | `https://puppeteer.workwithkora.com` | Legacy single host URL |
| `TARGET_CONTAINER` | `puppeteer-api` | Legacy single container name |
## Usage ## Usage
### Docker Run ### Multi-Host Configuration
The healthcheck will monitor each host at `http://host:8000` and restart containers with the same name as the host.
#### Docker Run
```bash ```bash
docker run -d \ docker run -d \
--name puppeteer-healthcheck \ --name puppeteer-healthcheck \
-v /var/run/docker.sock:/var/run/docker.sock \ -v /var/run/docker.sock:/var/run/docker.sock \
-e BASE_URL="https://puppeteer.workwithkora.com" \ -e HOSTS="host1,host2,host3" \
-e TEST_URL="https://www.google.com" \ -e TEST_URL="https://www.google.com" \
-e API_KEY="your-api-key" \ -e API_KEY="your-api-key" \
-e TARGET_CONTAINER="puppeteer-api" \
-e CHECK_INTERVAL="60" \ -e CHECK_INTERVAL="60" \
your-registry/puppeteer-healthcheck:latest your-registry/puppeteer-healthcheck:latest
``` ```
### Docker Compose #### Docker Compose
```yaml ```yaml
version: "3.8" version: "3.8"
@@ -49,24 +61,100 @@ services:
volumes: volumes:
- /var/run/docker.sock:/var/run/docker.sock - /var/run/docker.sock:/var/run/docker.sock
environment: environment:
- BASE_URL=https://puppeteer.workwithkora.com - HOSTS=host1,host2,host3
- TEST_URL=https://www.google.com - TEST_URL=https://www.google.com
- API_KEY=your-api-key - API_KEY=your-api-key
- TARGET_CONTAINER=puppeteer-api
- CHECK_INTERVAL=60 - CHECK_INTERVAL=60
restart: unless-stopped restart: unless-stopped
depends_on: ```
- puppeteer-api
### Single-Host Configuration (Legacy)
For backward compatibility, you can still use the old single-host configuration:
#### Docker Run
```bash
docker run -d \
--name puppeteer-healthcheck \
-v /var/run/docker.sock:/var/run/docker.sock \
-e BASE_URL="http://host1:8000" \
-e TEST_URL="https://www.google.com" \
-e API_KEY="your-api-key" \
-e TARGET_CONTAINER="host1" \
-e CHECK_INTERVAL="60" \
your-registry/puppeteer-healthcheck:latest
```
#### Docker Compose
```yaml
version: "3.8"
services:
puppeteer-healthcheck:
build: .
container_name: puppeteer-healthcheck
volumes:
- /var/run/docker.sock:/var/run/docker.sock
environment:
- BASE_URL=http://host1:8000
- TEST_URL=https://www.google.com
- API_KEY=your-api-key
- TARGET_CONTAINER=host1
- CHECK_INTERVAL=60
restart: unless-stopped
``` ```
## How It Works ## How It Works
1. The healthcheck container makes a GET request to the Puppeteer API every minute 1. The healthcheck container reads the list of hosts from the `HOSTS` environment variable
2. It uses the configured test URL and API key for authentication 2. For each host, it makes a GET request to `http://host:8000` every minute
3. If the request fails (non-200 status or timeout), it increments a failure counter 3. It uses the configured test URL and API key for authentication
4. After 3 consecutive failures, it attempts to restart the target container 4. If the request fails (non-200 status or timeout), it increments a failure counter for that specific host
5. If the restart is successful, the failure counter is reset 5. After 3 consecutive failures for a host, it attempts to restart the container with the same name as the host
6. The process continues indefinitely 6. If the restart is successful, the failure counter for that host is reset
7. The process continues indefinitely, monitoring all hosts independently
## Example Scenarios
### Monitoring Multiple Puppeteer Instances
If you have multiple puppeteer-api containers running on different hosts:
```bash
# Hosts: server1, server2, server3
# Containers: server1, server2, server3
docker run -d \
--name puppeteer-healthcheck \
-v /var/run/docker.sock:/var/run/docker.sock \
-e HOSTS="server1,server2,server3" \
-e API_KEY="your-api-key" \
your-registry/puppeteer-healthcheck:latest
```
This will:
- Check `http://server1:8000` and restart container `server1` if needed
- Check `http://server2:8000` and restart container `server2` if needed
- Check `http://server3:8000` and restart container `server3` if needed
### Mixed Environment
You can also monitor hosts with different names than their containers:
```bash
# Hosts: api1.example.com, api2.example.com
# Containers: puppeteer-api-1, puppeteer-api-2
docker run -d \
--name puppeteer-healthcheck \
-v /var/run/docker.sock:/var/run/docker.sock \
-e HOSTS="api1.example.com,api2.example.com" \
-e API_KEY="your-api-key" \
your-registry/puppeteer-healthcheck:latest
```
**Note**: In this case, the container names must match the host names exactly. If they don't, you'll need to use separate healthcheck instances or modify the container names.
## Logging ## Logging
@@ -76,6 +164,14 @@ The container logs all health check activities to both stdout and a log file (`/
- WARNING: Failed health checks - WARNING: Failed health checks
- ERROR: Container restart attempts and failures - ERROR: Container restart attempts and failures
Each log entry includes the host name for easy identification:
```
2024-01-15 10:30:00 - INFO - Performing health check for server1: http://server1:8000/?url=https%3A//www.google.com&skipCache=true
2024-01-15 10:30:01 - INFO - Health check passed for server1 - API is responding correctly
2024-01-15 10:30:02 - WARNING - Health check failed for server2 - Status code: 500
```
## Security Considerations ## Security Considerations
- The container requires access to the Docker socket to restart other containers - The container requires access to the Docker socket to restart other containers
@@ -127,13 +223,20 @@ This error occurs when the container cannot access the Docker socket. To fix:
### Container not found ### Container not found
- Ensure the `TARGET_CONTAINER` environment variable matches the exact name of your puppeteer-api container - Ensure the container names match the host names exactly
- Verify the container is running and accessible - Verify the containers are running and accessible
- Check that the `HOSTS` environment variable is set correctly
### API key issues ### API key issues
- Verify the API key is correct and has the necessary permissions - Verify the API key is correct and has the necessary permissions
- Check that the base URL is accessible from the container - Check that the hosts are accessible from the container
### Multiple host configuration
- Ensure the `HOSTS` environment variable is a comma-separated list without spaces
- Each host should be accessible at `http://host:8000`
- Container names must match host names exactly
## Building ## Building
@@ -143,4 +246,26 @@ docker build -t puppeteer-healthcheck .
## Version ## Version
Current version: 1.0.1 Current version: 2.0.0
### Migration from v1.x
To migrate from the single-host version to multi-host:
1. **Replace `BASE_URL` and `TARGET_CONTAINER` with `HOSTS`**:
```bash
# Old
-e BASE_URL="http://server1:8000" -e TARGET_CONTAINER="server1"
# New
-e HOSTS="server1"
```
2. **For multiple hosts, add them to the `HOSTS` variable**:
```bash
-e HOSTS="server1,server2,server3"
```
3. **Ensure container names match host names** (or rename containers accordingly)
Dockers/puppeteer-healthcheck/healthcheck.py
+69 -42
View File
@@ -1,7 +1,7 @@
#!/usr/bin/env python3 #!/usr/bin/env python3
""" """
Healthcheck script for Puppeteer API Healthcheck script for Puppeteer API
Monitors the API endpoint and restarts the container if it fails Monitors multiple API endpoints and restarts containers if they fail
""" """
import os import os
@@ -26,24 +26,46 @@ logger = logging.getLogger(__name__)
class PuppeteerHealthcheck: class PuppeteerHealthcheck:
def __init__(self): def __init__(self):
self.base_url = os.getenv('BASE_URL', 'https://puppeteer.workwithkora.com') self.hosts = self._parse_hosts()
self.test_url = os.getenv('TEST_URL', 'https://www.google.com') self.test_url = os.getenv('TEST_URL', 'https://www.google.com')
self.api_key = os.getenv('API_KEY', 'Q7Sd#hhFkyHy*T') self.api_key = os.getenv('API_KEY', 'Q7Sd#hhFkyHy*T')
self.target_container = os.getenv('TARGET_CONTAINER', 'puppeteer-api')
self.check_interval = int(os.getenv('CHECK_INTERVAL', '60')) self.check_interval = int(os.getenv('CHECK_INTERVAL', '60'))
self.max_consecutive_failures = int(os.getenv('MAX_CONSECUTIVE_FAILURES', '3')) self.max_consecutive_failures = int(os.getenv('MAX_CONSECUTIVE_FAILURES', '3'))
self.timeout = int(os.getenv('TIMEOUT', '20')) self.timeout = int(os.getenv('TIMEOUT', '20'))
# Track consecutive failures for each host
self.failure_counters = {host: 0 for host in self.hosts}
# Initialize Docker client with proper error handling # Initialize Docker client with proper error handling
self.docker_client = self._initialize_docker_client() self.docker_client = self._initialize_docker_client()
logger.info(f"Healthcheck initialized with:") logger.info(f"Healthcheck initialized with:")
logger.info(f" Base URL: {self.base_url}") logger.info(f" Hosts: {', '.join(self.hosts)}")
logger.info(f" Test URL: {self.test_url}") logger.info(f" Test URL: {self.test_url}")
logger.info(f" Target Container: {self.target_container}")
logger.info(f" Check Interval: {self.check_interval} seconds") logger.info(f" Check Interval: {self.check_interval} seconds")
logger.info(f" Timeout: {self.timeout} seconds") logger.info(f" Timeout: {self.timeout} seconds")
def _parse_hosts(self):
"""Parse hosts from environment variable or use default"""
hosts_env = os.getenv('HOSTS', '')
if hosts_env:
# Split by comma and strip whitespace
hosts = [host.strip() for host in hosts_env.split(',') if host.strip()]
if hosts:
return hosts
# Fallback to legacy BASE_URL for backward compatibility
base_url = os.getenv('BASE_URL', 'https://puppeteer.workwithkora.com')
if base_url.startswith('http://'):
# Extract host from http://host:port format
host_part = base_url.replace('http://', '').split('/')[0]
if ':' in host_part:
host = host_part.split(':')[0]
return [host]
# Default fallback
return ['puppeteer-api']
def _initialize_docker_client(self): def _initialize_docker_client(self):
"""Initialize Docker client with proper error handling""" """Initialize Docker client with proper error handling"""
try: try:
@@ -66,19 +88,20 @@ class PuppeteerHealthcheck:
logger.error(f"Unexpected error initializing Docker client: {e}") logger.error(f"Unexpected error initializing Docker client: {e}")
raise raise
def perform_health_check(self): def perform_health_check(self, host):
"""Perform the health check by making a request to the puppeteer API""" """Perform the health check for a specific host"""
try: try:
# Construct the URL with the test URL as a parameter # Construct the URL with the test URL as a parameter
base_url = f"http://{host}:8000"
encoded_test_url = quote(self.test_url, safe='') encoded_test_url = quote(self.test_url, safe='')
url = f"{self.base_url}/?url={encoded_test_url}&skipCache=true" url = f"{base_url}/?url={encoded_test_url}"
headers = { headers = {
'x-api-key': self.api_key, 'x-api-key': self.api_key,
'User-Agent': 'Puppeteer-Healthcheck/1.0' 'User-Agent': 'Puppeteer-Healthcheck/1.0'
} }
logger.info(f"Performing health check: {url}") logger.info(f"Performing health check for {host}: {url}")
response = requests.get( response = requests.get(
url, url,
@@ -92,77 +115,81 @@ class PuppeteerHealthcheck:
try: try:
response_data = response.json() response_data = response.json()
if response_data.get('status') == 'error': if response_data.get('status') == 'error':
logger.warning(f"Health check failed - API returned status: error") logger.warning(f"Health check failed for {host} - API returned status: error")
logger.warning(f"Response: {response.text[:200]}...") logger.warning(f"Response: {response.text[:200]}...")
return False return False
else: else:
logger.info("Health check passed - API is responding correctly") logger.info(f"Health check passed for {host} - API is responding correctly")
return True return True
except json.JSONDecodeError: except json.JSONDecodeError:
# If response is not JSON, treat as success (backward compatibility) # If response is not JSON, treat as success (backward compatibility)
logger.info("Health check passed - API is responding correctly (non-JSON response)") logger.info(f"Health check passed for {host} - API is responding correctly (non-JSON response)")
return True return True
else: else:
logger.warning(f"Health check failed - Status code: {response.status_code}") logger.warning(f"Health check failed for {host} - Status code: {response.status_code}")
logger.warning(f"Response: {response.text[:200]}...") logger.warning(f"Response: {response.text[:200]}...")
return False return False
except requests.exceptions.RequestException as e: except requests.exceptions.RequestException as e:
logger.error(f"Health check failed - Request error: {e}") logger.error(f"Health check failed for {host} - Request error: {e}")
return False return False
except Exception as e: except Exception as e:
logger.error(f"Health check failed - Unexpected error: {e}") logger.error(f"Health check failed for {host} - Unexpected error: {e}")
return False return False
def restart_container(self): def restart_container(self, host):
"""Restart the target puppeteer-api container""" """Restart the container for a specific host"""
try: try:
logger.info(f"Attempting to restart container: {self.target_container}") logger.info(f"Attempting to restart container: {host}")
# Get the container # Get the container (container name matches host name)
container = self.docker_client.containers.get(self.target_container) container = self.docker_client.containers.get(host)
# Restart the container # Restart the container
container.restart(timeout=30) container.restart(timeout=30)
logger.info(f"Successfully restarted container: {self.target_container}") logger.info(f"Successfully restarted container: {host}")
return True return True
except docker.errors.NotFound: except docker.errors.NotFound:
logger.error(f"Container not found: {self.target_container}") logger.error(f"Container not found: {host}")
return False return False
except docker.errors.APIError as e: except docker.errors.APIError as e:
logger.error(f"Docker API error while restarting container: {e}") logger.error(f"Docker API error while restarting container {host}: {e}")
return False return False
except Exception as e: except Exception as e:
logger.error(f"Unexpected error while restarting container: {e}") logger.error(f"Unexpected error while restarting container {host}: {e}")
return False return False
def check_host(self, host):
"""Check a single host and handle failures"""
if self.perform_health_check(host):
if self.failure_counters[host] > 0:
logger.info(f"Health check passed for {host} - resetting failure counter")
self.failure_counters[host] = 0
else:
self.failure_counters[host] += 1
logger.warning(f"Health check failed for {host} - consecutive failures: {self.failure_counters[host]}")
# Restart container if we've had too many consecutive failures
if self.failure_counters[host] >= self.max_consecutive_failures:
logger.error(f"Health check failed {self.failure_counters[host]} times consecutively for {host} - restarting container")
if self.restart_container(host):
self.failure_counters[host] = 0
logger.info(f"Container {host} restarted successfully - resetting failure counter")
else:
logger.error(f"Failed to restart container {host}")
def run(self): def run(self):
"""Main loop for the healthcheck""" """Main loop for the healthcheck"""
logger.info("Starting Puppeteer API healthcheck service") logger.info("Starting Puppeteer API healthcheck service")
consecutive_failures = 0
while True: while True:
try: try:
# Perform health check # Check each host
if self.perform_health_check(): for host in self.hosts:
consecutive_failures = 0 self.check_host(host)
logger.info("Health check passed - resetting failure counter")
else:
consecutive_failures += 1
logger.warning(f"Health check failed - consecutive failures: {consecutive_failures}")
# Restart container if we've had too many consecutive failures
if consecutive_failures >= self.max_consecutive_failures:
logger.error(f"Health check failed {consecutive_failures} times consecutively - restarting container")
if self.restart_container():
consecutive_failures = 0
logger.info("Container restarted successfully - resetting failure counter")
else:
logger.error("Failed to restart container")
# Wait before next check # Wait before next check
logger.info(f"Waiting {self.check_interval} seconds before next health check...") logger.info(f"Waiting {self.check_interval} seconds before next health check...")