Bram/projects
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 3 Wiki Activity

allow multiple hosts for healthcheck
Build and Push Docker Images / build-and-push (push) Successful in 23s
Details

Browse Source
This commit is contained in:
Bram
2025-07-16 10:13:53 +02:00
parent 22b06f775d
commit 55e6c2237f
2 changed files with 222 additions and 70 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Dockers/puppeteer-healthcheck/README.md
+153 -28
View File
@@ -1,43 +1,55 @@
# Puppeteer API Healthcheck
A Docker container that monitors the Puppeteer API and automatically restarts the target container when the API becomes unresponsive.
A Docker container that monitors multiple Puppeteer API endpoints and automatically restarts the target containers when the APIs become unresponsive.
## Features
- Monitors the Puppeteer API endpoint every minute
- Automatically restarts the target container after 3 consecutive failures
- Monitors multiple Puppeteer API endpoints every minute
- Automatically restarts target containers after 3 consecutive failures
- Configurable via environment variables
- Comprehensive logging
- Docker socket access for container management
- Proper Docker permissions handling
- Backward compatibility with single-host configuration
## Environment Variables
| Variable | Default | Description |
| ------------------ | ------------------------------------ | -------------------------------- |
| `BASE_URL` | `https://puppeteer.workwithkora.com` | Base URL of the Puppeteer API |
| `TEST_URL` | `https://www.google.com` | URL to test the API with |
| `API_KEY` | `Q7Sd#hhFkyHy*T` | API key for authentication |
| `TARGET_CONTAINER` | `puppeteer-api` | Name of the container to restart |
| `CHECK_INTERVAL` | `60` | Health check interval in seconds |
| Variable | Default | Description |
| -------------------------- | ------------------------ | ---------------------------------------------- |
| `HOSTS` | `puppeteer-api` | Comma-separated list of hosts to monitor |
| `TEST_URL` | `https://www.google.com` | URL to test the API with |
| `API_KEY` | `Q7Sd#hhFkyHy*T` | API key for authentication |
| `CHECK_INTERVAL` | `60` | Health check interval in seconds |
| `MAX_CONSECUTIVE_FAILURES` | `3` | Number of failures before restarting container |
| `TIMEOUT` | `20` | Request timeout in seconds |
### Legacy Variables (for backward compatibility)
| Variable | Default | Description |
| ------------------ | ------------------------------------ | ---------------------------- |
| `BASE_URL` | `https://puppeteer.workwithkora.com` | Legacy single host URL |
| `TARGET_CONTAINER` | `puppeteer-api` | Legacy single container name |
## Usage
### Docker Run
### Multi-Host Configuration
The healthcheck will monitor each host at `http://host:8000` and restart containers with the same name as the host.
#### Docker Run
```bash
docker run -d \
--name puppeteer-healthcheck \
-v /var/run/docker.sock:/var/run/docker.sock \
-e BASE_URL="https://puppeteer.workwithkora.com" \
-e HOSTS="host1,host2,host3" \
-e TEST_URL="https://www.google.com" \
-e API_KEY="your-api-key" \
-e TARGET_CONTAINER="puppeteer-api" \
-e CHECK_INTERVAL="60" \
your-registry/puppeteer-healthcheck:latest
```
### Docker Compose
#### Docker Compose
```yaml
version: "3.8"
@@ -49,24 +61,100 @@ services:
volumes:
- /var/run/docker.sock:/var/run/docker.sock
environment:
- BASE_URL=https://puppeteer.workwithkora.com
- HOSTS=host1,host2,host3
- TEST_URL=https://www.google.com
- API_KEY=your-api-key
- TARGET_CONTAINER=puppeteer-api
- CHECK_INTERVAL=60
restart: unless-stopped
depends_on:
- puppeteer-api
```
### Single-Host Configuration (Legacy)
For backward compatibility, you can still use the old single-host configuration:
#### Docker Run
```bash
docker run -d \
--name puppeteer-healthcheck \
-v /var/run/docker.sock:/var/run/docker.sock \
-e BASE_URL="http://host1:8000" \
-e TEST_URL="https://www.google.com" \
-e API_KEY="your-api-key" \
-e TARGET_CONTAINER="host1" \
-e CHECK_INTERVAL="60" \
your-registry/puppeteer-healthcheck:latest
```
#### Docker Compose
```yaml
version: "3.8"
services:
puppeteer-healthcheck:
build: .
container_name: puppeteer-healthcheck
volumes:
- /var/run/docker.sock:/var/run/docker.sock
environment:
- BASE_URL=http://host1:8000
- TEST_URL=https://www.google.com
- API_KEY=your-api-key
- TARGET_CONTAINER=host1
- CHECK_INTERVAL=60
restart: unless-stopped
```
## How It Works
1. The healthcheck container makes a GET request to the Puppeteer API every minute
2. It uses the configured test URL and API key for authentication
3. If the request fails (non-200 status or timeout), it increments a failure counter
4. After 3 consecutive failures, it attempts to restart the target container
5. If the restart is successful, the failure counter is reset
6. The process continues indefinitely
1. The healthcheck container reads the list of hosts from the `HOSTS` environment variable
2. For each host, it makes a GET request to `http://host:8000` every minute
3. It uses the configured test URL and API key for authentication
4. If the request fails (non-200 status or timeout), it increments a failure counter for that specific host
5. After 3 consecutive failures for a host, it attempts to restart the container with the same name as the host
6. If the restart is successful, the failure counter for that host is reset
7. The process continues indefinitely, monitoring all hosts independently
## Example Scenarios
### Monitoring Multiple Puppeteer Instances
If you have multiple puppeteer-api containers running on different hosts:
```bash
# Hosts: server1, server2, server3
# Containers: server1, server2, server3
docker run -d \
--name puppeteer-healthcheck \
-v /var/run/docker.sock:/var/run/docker.sock \
-e HOSTS="server1,server2,server3" \
-e API_KEY="your-api-key" \
your-registry/puppeteer-healthcheck:latest
```
This will:
- Check `http://server1:8000` and restart container `server1` if needed
- Check `http://server2:8000` and restart container `server2` if needed
- Check `http://server3:8000` and restart container `server3` if needed
### Mixed Environment
You can also monitor hosts with different names than their containers:
```bash
# Hosts: api1.example.com, api2.example.com
# Containers: puppeteer-api-1, puppeteer-api-2
docker run -d \
--name puppeteer-healthcheck \
-v /var/run/docker.sock:/var/run/docker.sock \
-e HOSTS="api1.example.com,api2.example.com" \
-e API_KEY="your-api-key" \
your-registry/puppeteer-healthcheck:latest
```
**Note**: In this case, the container names must match the host names exactly. If they don't, you'll need to use separate healthcheck instances or modify the container names.
## Logging
@@ -76,6 +164,14 @@ The container logs all health check activities to both stdout and a log file (`/
- WARNING: Failed health checks
- ERROR: Container restart attempts and failures
Each log entry includes the host name for easy identification:
```
2024-01-15 10:30:00 - INFO - Performing health check for server1: http://server1:8000/?url=https%3A//www.google.com&skipCache=true
2024-01-15 10:30:01 - INFO - Health check passed for server1 - API is responding correctly
2024-01-15 10:30:02 - WARNING - Health check failed for server2 - Status code: 500
```
## Security Considerations
- The container requires access to the Docker socket to restart other containers
@@ -127,13 +223,20 @@ This error occurs when the container cannot access the Docker socket. To fix:
### Container not found
- Ensure the `TARGET_CONTAINER` environment variable matches the exact name of your puppeteer-api container
- Verify the container is running and accessible
- Ensure the container names match the host names exactly
- Verify the containers are running and accessible
- Check that the `HOSTS` environment variable is set correctly
### API key issues
- Verify the API key is correct and has the necessary permissions
- Check that the base URL is accessible from the container
- Check that the hosts are accessible from the container
### Multiple host configuration
- Ensure the `HOSTS` environment variable is a comma-separated list without spaces
- Each host should be accessible at `http://host:8000`
- Container names must match host names exactly
## Building
@@ -143,4 +246,26 @@ docker build -t puppeteer-healthcheck .
## Version
Current version: 1.0.1
Current version: 2.0.0
### Migration from v1.x
To migrate from the single-host version to multi-host:
1. **Replace `BASE_URL` and `TARGET_CONTAINER` with `HOSTS`**:
```bash
# Old
-e BASE_URL="http://server1:8000" -e TARGET_CONTAINER="server1"
# New
-e HOSTS="server1"
```
2. **For multiple hosts, add them to the `HOSTS` variable**:
```bash
-e HOSTS="server1,server2,server3"
```
3. **Ensure container names match host names** (or rename containers accordingly)
Dockers/puppeteer-healthcheck/healthcheck.py
+69 -42
View File
@@ -1,7 +1,7 @@
#!/usr/bin/env python3
"""
Healthcheck script for Puppeteer API
Monitors the API endpoint and restarts the container if it fails
Monitors multiple API endpoints and restarts containers if they fail
"""
import os
@@ -26,24 +26,46 @@ logger = logging.getLogger(__name__)
class PuppeteerHealthcheck:
def __init__(self):
self.base_url = os.getenv('BASE_URL', 'https://puppeteer.workwithkora.com')
self.hosts = self._parse_hosts()
self.test_url = os.getenv('TEST_URL', 'https://www.google.com')
self.api_key = os.getenv('API_KEY', 'Q7Sd#hhFkyHy*T')
self.target_container = os.getenv('TARGET_CONTAINER', 'puppeteer-api')
self.check_interval = int(os.getenv('CHECK_INTERVAL', '60'))
self.max_consecutive_failures = int(os.getenv('MAX_CONSECUTIVE_FAILURES', '3'))
self.timeout = int(os.getenv('TIMEOUT', '20'))
# Track consecutive failures for each host
self.failure_counters = {host: 0 for host in self.hosts}
# Initialize Docker client with proper error handling
self.docker_client = self._initialize_docker_client()
logger.info(f"Healthcheck initialized with:")
logger.info(f" Base URL: {self.base_url}")
logger.info(f" Hosts: {', '.join(self.hosts)}")
logger.info(f" Test URL: {self.test_url}")
logger.info(f" Target Container: {self.target_container}")
logger.info(f" Check Interval: {self.check_interval} seconds")
logger.info(f" Timeout: {self.timeout} seconds")
def _parse_hosts(self):
"""Parse hosts from environment variable or use default"""
hosts_env = os.getenv('HOSTS', '')
if hosts_env:
# Split by comma and strip whitespace
hosts = [host.strip() for host in hosts_env.split(',') if host.strip()]
if hosts:
return hosts
# Fallback to legacy BASE_URL for backward compatibility
base_url = os.getenv('BASE_URL', 'https://puppeteer.workwithkora.com')
if base_url.startswith('http://'):
# Extract host from http://host:port format
host_part = base_url.replace('http://', '').split('/')[0]
if ':' in host_part:
host = host_part.split(':')[0]
return [host]
# Default fallback
return ['puppeteer-api']
def _initialize_docker_client(self):
"""Initialize Docker client with proper error handling"""
try:
@@ -66,19 +88,20 @@ class PuppeteerHealthcheck:
logger.error(f"Unexpected error initializing Docker client: {e}")
raise
def perform_health_check(self):
"""Perform the health check by making a request to the puppeteer API"""
def perform_health_check(self, host):
"""Perform the health check for a specific host"""
try:
# Construct the URL with the test URL as a parameter
base_url = f"http://{host}:8000"
encoded_test_url = quote(self.test_url, safe='')
url = f"{self.base_url}/?url={encoded_test_url}&skipCache=true"
url = f"{base_url}/?url={encoded_test_url}"
headers = {
'x-api-key': self.api_key,
'User-Agent': 'Puppeteer-Healthcheck/1.0'
}
logger.info(f"Performing health check: {url}")
logger.info(f"Performing health check for {host}: {url}")
response = requests.get(
url,
@@ -92,77 +115,81 @@ class PuppeteerHealthcheck:
try:
response_data = response.json()
if response_data.get('status') == 'error':
logger.warning(f"Health check failed - API returned status: error")
logger.warning(f"Health check failed for {host} - API returned status: error")
logger.warning(f"Response: {response.text[:200]}...")
return False
else:
logger.info("Health check passed - API is responding correctly")
logger.info(f"Health check passed for {host} - API is responding correctly")
return True
except json.JSONDecodeError:
# If response is not JSON, treat as success (backward compatibility)
logger.info("Health check passed - API is responding correctly (non-JSON response)")
logger.info(f"Health check passed for {host} - API is responding correctly (non-JSON response)")
return True
else:
logger.warning(f"Health check failed - Status code: {response.status_code}")
logger.warning(f"Health check failed for {host} - Status code: {response.status_code}")
logger.warning(f"Response: {response.text[:200]}...")
return False
except requests.exceptions.RequestException as e:
logger.error(f"Health check failed - Request error: {e}")
logger.error(f"Health check failed for {host} - Request error: {e}")
return False
except Exception as e:
logger.error(f"Health check failed - Unexpected error: {e}")
logger.error(f"Health check failed for {host} - Unexpected error: {e}")
return False
def restart_container(self):
"""Restart the target puppeteer-api container"""
def restart_container(self, host):
"""Restart the container for a specific host"""
try:
logger.info(f"Attempting to restart container: {self.target_container}")
logger.info(f"Attempting to restart container: {host}")
# Get the container
container = self.docker_client.containers.get(self.target_container)
# Get the container (container name matches host name)
container = self.docker_client.containers.get(host)
# Restart the container
container.restart(timeout=30)
logger.info(f"Successfully restarted container: {self.target_container}")
logger.info(f"Successfully restarted container: {host}")
return True
except docker.errors.NotFound:
logger.error(f"Container not found: {self.target_container}")
logger.error(f"Container not found: {host}")
return False
except docker.errors.APIError as e:
logger.error(f"Docker API error while restarting container: {e}")
logger.error(f"Docker API error while restarting container {host}: {e}")
return False
except Exception as e:
logger.error(f"Unexpected error while restarting container: {e}")
logger.error(f"Unexpected error while restarting container {host}: {e}")
return False
def check_host(self, host):
"""Check a single host and handle failures"""
if self.perform_health_check(host):
if self.failure_counters[host] > 0:
logger.info(f"Health check passed for {host} - resetting failure counter")
self.failure_counters[host] = 0
else:
self.failure_counters[host] += 1
logger.warning(f"Health check failed for {host} - consecutive failures: {self.failure_counters[host]}")
# Restart container if we've had too many consecutive failures
if self.failure_counters[host] >= self.max_consecutive_failures:
logger.error(f"Health check failed {self.failure_counters[host]} times consecutively for {host} - restarting container")
if self.restart_container(host):
self.failure_counters[host] = 0
logger.info(f"Container {host} restarted successfully - resetting failure counter")
else:
logger.error(f"Failed to restart container {host}")
def run(self):
"""Main loop for the healthcheck"""
logger.info("Starting Puppeteer API healthcheck service")
consecutive_failures = 0
while True:
try:
# Perform health check
if self.perform_health_check():
consecutive_failures = 0
logger.info("Health check passed - resetting failure counter")
else:
consecutive_failures += 1
logger.warning(f"Health check failed - consecutive failures: {consecutive_failures}")
# Restart container if we've had too many consecutive failures
if consecutive_failures >= self.max_consecutive_failures:
logger.error(f"Health check failed {consecutive_failures} times consecutively - restarting container")
if self.restart_container():
consecutive_failures = 0
logger.info("Container restarted successfully - resetting failure counter")
else:
logger.error("Failed to restart container")
# Check each host
for host in self.hosts:
self.check_host(host)
# Wait before next check
logger.info(f"Waiting {self.check_interval} seconds before next health check...")