first version of gmail invoice extractor
Build and Push Docker Images / build-and-push (push) Failing after 1m37s
Build and Push Docker Images / build-and-push (push) Failing after 1m37s
This commit is contained in:
@@ -0,0 +1,5 @@
|
|||||||
|
__pycache__/
|
||||||
|
invoices/
|
||||||
|
token.json
|
||||||
|
credentials.json
|
||||||
|
invoice_extractor.log
|
||||||
@@ -0,0 +1,30 @@
|
|||||||
|
FROM python:3.12-slim
|
||||||
|
|
||||||
|
# Install system dependencies
|
||||||
|
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||||
|
build-essential \
|
||||||
|
libffi-dev \
|
||||||
|
libxml2 \
|
||||||
|
libxslt1-dev \
|
||||||
|
libjpeg-dev \
|
||||||
|
zlib1g-dev \
|
||||||
|
libpoppler-cpp-dev \
|
||||||
|
&& rm -rf /var/lib/apt/lists/*
|
||||||
|
|
||||||
|
# Set workdir
|
||||||
|
WORKDIR /app
|
||||||
|
|
||||||
|
# Copy requirements first (better caching)
|
||||||
|
COPY requirements.txt .
|
||||||
|
|
||||||
|
# Install Python dependencies
|
||||||
|
RUN pip install --no-cache-dir -r requirements.txt
|
||||||
|
|
||||||
|
# Copy source code
|
||||||
|
COPY . .
|
||||||
|
|
||||||
|
# Make invoices folder inside container
|
||||||
|
RUN mkdir -p /app/invoices
|
||||||
|
|
||||||
|
# Run script
|
||||||
|
CMD ["python", "main.py"]
|
||||||
@@ -0,0 +1,76 @@
|
|||||||
|
# Gmail Invoice Extractor
|
||||||
|
|
||||||
|
This application automatically scans Gmail for emails containing PDF attachments, checks if they contain your company's VAT number, and saves them to a designated folder.
|
||||||
|
|
||||||
|
## Features
|
||||||
|
|
||||||
|
- 🔍 Scans Gmail for emails with PDF attachments
|
||||||
|
- 📄 Extracts text from PDF files using OCR
|
||||||
|
- 💾 Automatically saves valid invoices to a local folder
|
||||||
|
- 📝 Comprehensive logging
|
||||||
|
- 🐳 Docker support
|
||||||
|
|
||||||
|
## Setup
|
||||||
|
|
||||||
|
### 1. Gmail API Setup
|
||||||
|
|
||||||
|
1. Go to the [Google Cloud Console](https://console.cloud.google.com/)
|
||||||
|
2. Create a new project or select an existing one
|
||||||
|
3. Enable the Gmail API
|
||||||
|
4. Create credentials (OAuth 2.0 Client ID)
|
||||||
|
5. Download the credentials file and save it as `credentials.json` in this directory
|
||||||
|
|
||||||
|
### 2. Configuration
|
||||||
|
|
||||||
|
1. Update the VAT number patterns in `config.yaml` to match your company's VAT numbers
|
||||||
|
2. Modify the Gmail search query if needed (default: emails with PDF attachments)
|
||||||
|
|
||||||
|
### 3. Running the Application
|
||||||
|
|
||||||
|
#### Option A: Direct Python execution
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pip install -r requirements.txt
|
||||||
|
python main.py
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Option B: Using Docker
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker build -t gmail-invoice-extractor .
|
||||||
|
docker run -v $(pwd)/invoices:/app/invoices -v $(pwd)/credentials.json:/app/credentials.json gmail-invoice-extractor
|
||||||
|
```
|
||||||
|
|
||||||
|
## First Run
|
||||||
|
|
||||||
|
On the first run, the application will:
|
||||||
|
|
||||||
|
1. Open a browser window for Gmail authentication
|
||||||
|
2. Ask you to authorize the application
|
||||||
|
3. Save the authentication token for future runs
|
||||||
|
|
||||||
|
## Configuration
|
||||||
|
|
||||||
|
Edit `config.yaml` to customize:
|
||||||
|
|
||||||
|
- VAT number patterns for your company
|
||||||
|
- Gmail search query
|
||||||
|
- Maximum messages to process
|
||||||
|
- Save folder location
|
||||||
|
- Logging level
|
||||||
|
|
||||||
|
## Logs
|
||||||
|
|
||||||
|
The application creates detailed logs in `invoice_extractor.log` and displays them in the console.
|
||||||
|
|
||||||
|
## Security Notes
|
||||||
|
|
||||||
|
- Keep your `credentials.json` and `token.json` files secure
|
||||||
|
- Never commit these files to version control
|
||||||
|
- The application only requests read-only access to Gmail
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
- **Authentication issues**: Delete `token.json` and run again to re-authenticate
|
||||||
|
- **No PDFs found**: Check your Gmail search query in `config.yaml`
|
||||||
|
- **VAT numbers not detected**: Verify your VAT number patterns in `config.yaml`
|
||||||
@@ -0,0 +1,39 @@
|
|||||||
|
import yaml
|
||||||
|
import os
|
||||||
|
from typing import List
|
||||||
|
|
||||||
|
class Config:
|
||||||
|
def __init__(self, config_file: str = "config.yaml"):
|
||||||
|
self.config_file = config_file
|
||||||
|
self.load_config()
|
||||||
|
|
||||||
|
def load_config(self):
|
||||||
|
"""Load configuration from YAML file."""
|
||||||
|
if os.path.exists(self.config_file):
|
||||||
|
with open(self.config_file, 'r') as f:
|
||||||
|
config = yaml.safe_load(f)
|
||||||
|
else:
|
||||||
|
# Default configuration
|
||||||
|
config = {
|
||||||
|
'vat_patterns': [
|
||||||
|
'BE0?\\d{9}',
|
||||||
|
'NL\\d{9}B\\d{2}',
|
||||||
|
'DE\\d{9}',
|
||||||
|
'FR[A-Z0-9]{2}\\d{9}',
|
||||||
|
'GB\\d{9,12}'
|
||||||
|
],
|
||||||
|
'gmail_query': 'has:attachment filename:pdf',
|
||||||
|
'max_messages': 100,
|
||||||
|
'save_folder': 'invoices',
|
||||||
|
'log_level': 'INFO'
|
||||||
|
}
|
||||||
|
|
||||||
|
self.vat_patterns = config.get('vat_patterns', [])
|
||||||
|
self.gmail_query = config.get('gmail_query', 'has:attachment filename:pdf')
|
||||||
|
self.max_messages = config.get('max_messages', 100)
|
||||||
|
self.save_folder = config.get('save_folder', 'invoices')
|
||||||
|
self.log_level = config.get('log_level', 'INFO')
|
||||||
|
|
||||||
|
def get_vat_patterns(self) -> List[str]:
|
||||||
|
"""Get VAT number patterns as compiled regex patterns."""
|
||||||
|
return self.vat_patterns
|
||||||
@@ -0,0 +1,23 @@
|
|||||||
|
# Gmail Invoice Extractor Configuration
|
||||||
|
|
||||||
|
# VAT number patterns for your company
|
||||||
|
# Add your actual VAT number patterns here
|
||||||
|
vat_patterns:
|
||||||
|
- "BE0?\\d{9}" # Belgium
|
||||||
|
- "NL\\d{9}B\\d{2}" # Netherlands
|
||||||
|
- "DE\\d{9}" # Germany
|
||||||
|
- "FR[A-Z0-9]{2}\\d{9}" # France
|
||||||
|
- "GB\\d{9,12}" # UK
|
||||||
|
# Add more patterns as needed
|
||||||
|
|
||||||
|
# Gmail search query (modify as needed)
|
||||||
|
gmail_query: "has:attachment filename:pdf"
|
||||||
|
|
||||||
|
# Maximum number of messages to process per run
|
||||||
|
max_messages: 100
|
||||||
|
|
||||||
|
# Folder to save invoices
|
||||||
|
save_folder: "invoices"
|
||||||
|
|
||||||
|
# Logging level (DEBUG, INFO, WARNING, ERROR)
|
||||||
|
log_level: "INFO"
|
||||||
@@ -0,0 +1,258 @@
|
|||||||
|
import os
|
||||||
|
import re
|
||||||
|
import base64
|
||||||
|
import pdfplumber
|
||||||
|
import logging
|
||||||
|
from datetime import datetime, timedelta
|
||||||
|
from typing import List, Tuple, Optional
|
||||||
|
|
||||||
|
from google.oauth2.credentials import Credentials
|
||||||
|
from google_auth_oauthlib.flow import InstalledAppFlow
|
||||||
|
from googleapiclient.discovery import build
|
||||||
|
from google.auth.transport.requests import Request
|
||||||
|
|
||||||
|
from config import Config
|
||||||
|
|
||||||
|
# Load configuration
|
||||||
|
config = Config()
|
||||||
|
|
||||||
|
# Configure logging
|
||||||
|
logging.basicConfig(
|
||||||
|
level=getattr(logging, config.log_level),
|
||||||
|
format='%(asctime)s - %(levelname)s - %(message)s',
|
||||||
|
handlers=[
|
||||||
|
logging.FileHandler('invoice_extractor.log'),
|
||||||
|
logging.StreamHandler()
|
||||||
|
]
|
||||||
|
)
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
# Gmail API scope for read-only with attachments
|
||||||
|
SCOPES = ['https://www.googleapis.com/auth/gmail.readonly']
|
||||||
|
|
||||||
|
# Folder to save invoices
|
||||||
|
SAVE_FOLDER = config.save_folder
|
||||||
|
os.makedirs(SAVE_FOLDER, exist_ok=True)
|
||||||
|
|
||||||
|
|
||||||
|
def get_last_month_date_range():
|
||||||
|
"""Get the date range for the last month in Gmail query format."""
|
||||||
|
today = datetime.now()
|
||||||
|
|
||||||
|
# Calculate first day of current month
|
||||||
|
first_day_current = today.replace(day=1)
|
||||||
|
|
||||||
|
# Calculate first day of last month
|
||||||
|
if first_day_current.month == 1:
|
||||||
|
first_day_last_month = first_day_current.replace(year=first_day_current.year - 1, month=12)
|
||||||
|
else:
|
||||||
|
first_day_last_month = first_day_current.replace(month=first_day_current.month - 1)
|
||||||
|
|
||||||
|
# Calculate last day of last month
|
||||||
|
last_day_last_month = first_day_current - timedelta(days=1)
|
||||||
|
|
||||||
|
# Format dates for Gmail query (YYYY/MM/DD format)
|
||||||
|
start_date = first_day_last_month.strftime('%Y/%m/%d')
|
||||||
|
end_date = last_day_last_month.strftime('%Y/%m/%d')
|
||||||
|
|
||||||
|
logger.info(f"Scanning emails from {start_date} to {end_date}")
|
||||||
|
|
||||||
|
return f"after:{start_date} before:{end_date}"
|
||||||
|
|
||||||
|
|
||||||
|
def gmail_authenticate():
|
||||||
|
"""Authenticate with Gmail API using OAuth2."""
|
||||||
|
creds = None
|
||||||
|
if os.path.exists("token.json"):
|
||||||
|
creds = Credentials.from_authorized_user_file("token.json", SCOPES)
|
||||||
|
|
||||||
|
if not creds or not creds.valid:
|
||||||
|
if creds and creds.expired and creds.refresh_token:
|
||||||
|
try:
|
||||||
|
creds.refresh(Request())
|
||||||
|
logger.info("Credentials refreshed successfully")
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Failed to refresh credentials: {e}")
|
||||||
|
creds = None
|
||||||
|
|
||||||
|
if not creds:
|
||||||
|
try:
|
||||||
|
flow = InstalledAppFlow.from_client_secrets_file("credentials.json", SCOPES)
|
||||||
|
creds = flow.run_local_server(port=0)
|
||||||
|
logger.info("New credentials obtained")
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Failed to obtain credentials: {e}")
|
||||||
|
raise
|
||||||
|
|
||||||
|
with open("token.json", "w") as token:
|
||||||
|
token.write(creds.to_json())
|
||||||
|
logger.info("Credentials saved to token.json")
|
||||||
|
|
||||||
|
return build("gmail", "v1", credentials=creds)
|
||||||
|
|
||||||
|
|
||||||
|
def search_messages(service, base_query: str, max_results: int = None) -> List[dict]:
|
||||||
|
"""Search for messages matching the query with date filtering."""
|
||||||
|
if max_results is None:
|
||||||
|
max_results = config.max_messages
|
||||||
|
|
||||||
|
# Add date filter for last month
|
||||||
|
date_filter = get_last_month_date_range()
|
||||||
|
full_query = f"{base_query} {date_filter}"
|
||||||
|
|
||||||
|
try:
|
||||||
|
results = service.users().messages().list(
|
||||||
|
userId="me",
|
||||||
|
q=full_query,
|
||||||
|
maxResults=max_results
|
||||||
|
).execute()
|
||||||
|
messages = results.get("messages", [])
|
||||||
|
logger.info(f"Found {len(messages)} messages matching query: {full_query}")
|
||||||
|
return messages
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Failed to search messages: {e}")
|
||||||
|
return []
|
||||||
|
|
||||||
|
|
||||||
|
def get_attachments(service, msg_id: str) -> List[Tuple[str, bytes]]:
|
||||||
|
"""Extract PDF attachments from a message."""
|
||||||
|
try:
|
||||||
|
message = service.users().messages().get(userId="me", id=msg_id).execute()
|
||||||
|
parts = message.get("payload", {}).get("parts", [])
|
||||||
|
attachments = []
|
||||||
|
|
||||||
|
for part in parts:
|
||||||
|
if part.get("filename") and part["filename"].lower().endswith(".pdf"):
|
||||||
|
att_id = part["body"].get("attachmentId")
|
||||||
|
if att_id:
|
||||||
|
try:
|
||||||
|
att = service.users().messages().attachments().get(
|
||||||
|
userId="me", messageId=msg_id, id=att_id
|
||||||
|
).execute()
|
||||||
|
data = base64.urlsafe_b64decode(att["data"])
|
||||||
|
attachments.append((part["filename"], data))
|
||||||
|
logger.debug(f"Extracted attachment: {part['filename']}")
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Failed to extract attachment {part['filename']}: {e}")
|
||||||
|
|
||||||
|
return attachments
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Failed to get attachments for message {msg_id}: {e}")
|
||||||
|
return []
|
||||||
|
|
||||||
|
|
||||||
|
def extract_text_from_pdf(pdf_bytes: bytes) -> str:
|
||||||
|
"""Extract text from PDF bytes."""
|
||||||
|
text = ""
|
||||||
|
temp_file = f"temp_{datetime.now().strftime('%Y%m%d_%H%M%S')}.pdf"
|
||||||
|
|
||||||
|
try:
|
||||||
|
with open(temp_file, "wb") as f:
|
||||||
|
f.write(pdf_bytes)
|
||||||
|
|
||||||
|
with pdfplumber.open(temp_file) as pdf:
|
||||||
|
for page_num, page in enumerate(pdf.pages):
|
||||||
|
page_text = page.extract_text()
|
||||||
|
if page_text:
|
||||||
|
text += page_text + "\n"
|
||||||
|
logger.debug(f"Extracted text from page {page_num + 1}")
|
||||||
|
|
||||||
|
return text
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Failed to extract text from PDF: {e}")
|
||||||
|
return ""
|
||||||
|
finally:
|
||||||
|
if os.path.exists(temp_file):
|
||||||
|
os.remove(temp_file)
|
||||||
|
|
||||||
|
|
||||||
|
def contains_vat_number(text: str) -> bool:
|
||||||
|
"""Check if text contains any of the configured VAT number patterns."""
|
||||||
|
if not text:
|
||||||
|
return False
|
||||||
|
|
||||||
|
text_upper = text.upper()
|
||||||
|
vat_patterns = config.get_vat_patterns()
|
||||||
|
|
||||||
|
for pattern in vat_patterns:
|
||||||
|
if re.search(pattern, text_upper):
|
||||||
|
logger.info(f"Found VAT number match with pattern: {pattern}")
|
||||||
|
return True
|
||||||
|
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def save_invoice(filename: str, pdf_bytes: bytes) -> bool:
|
||||||
|
"""Save invoice PDF to the designated folder."""
|
||||||
|
try:
|
||||||
|
# Create filename with timestamp to avoid conflicts
|
||||||
|
timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
|
||||||
|
name, ext = os.path.splitext(filename)
|
||||||
|
safe_filename = f"{name}_{timestamp}{ext}"
|
||||||
|
save_path = os.path.join(SAVE_FOLDER, safe_filename)
|
||||||
|
|
||||||
|
with open(save_path, "wb") as f:
|
||||||
|
f.write(pdf_bytes)
|
||||||
|
|
||||||
|
logger.info(f"✅ Saved invoice: {safe_filename}")
|
||||||
|
return True
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Failed to save invoice {filename}: {e}")
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
"""Main function to process Gmail messages and extract invoices."""
|
||||||
|
logger.info("Starting Gmail Invoice Extractor")
|
||||||
|
logger.info(f"Configuration: {config.gmail_query}, max_messages: {config.max_messages}")
|
||||||
|
|
||||||
|
try:
|
||||||
|
# Authenticate with Gmail
|
||||||
|
service = gmail_authenticate()
|
||||||
|
|
||||||
|
# Search for messages with PDF attachments from last month
|
||||||
|
messages = search_messages(service, config.gmail_query)
|
||||||
|
|
||||||
|
if not messages:
|
||||||
|
logger.info("No messages found with PDF attachments from last month")
|
||||||
|
return
|
||||||
|
|
||||||
|
processed_count = 0
|
||||||
|
saved_count = 0
|
||||||
|
|
||||||
|
for msg in messages:
|
||||||
|
try:
|
||||||
|
msg_id = msg["id"]
|
||||||
|
attachments = get_attachments(service, msg_id)
|
||||||
|
|
||||||
|
for filename, pdf_bytes in attachments:
|
||||||
|
processed_count += 1
|
||||||
|
logger.info(f"Processing attachment: {filename}")
|
||||||
|
|
||||||
|
# Extract text from PDF
|
||||||
|
text = extract_text_from_pdf(pdf_bytes)
|
||||||
|
|
||||||
|
if not text:
|
||||||
|
logger.warning(f"No text extracted from {filename}")
|
||||||
|
continue
|
||||||
|
|
||||||
|
# Check for VAT number
|
||||||
|
if contains_vat_number(text):
|
||||||
|
if save_invoice(filename, pdf_bytes):
|
||||||
|
saved_count += 1
|
||||||
|
else:
|
||||||
|
logger.info(f"❌ Skipped (no VAT match): {filename}")
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error processing message {msg.get('id', 'unknown')}: {e}")
|
||||||
|
continue
|
||||||
|
|
||||||
|
logger.info(f"Processing complete. Processed: {processed_count}, Saved: {saved_count}")
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Fatal error in main: {e}")
|
||||||
|
raise
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,5 @@
|
|||||||
|
google-api-python-client
|
||||||
|
google-auth-httplib2
|
||||||
|
google-auth-oauthlib
|
||||||
|
pdfplumber
|
||||||
|
PyYAML
|
||||||
@@ -1,39 +0,0 @@
|
|||||||
FROM alpine:latest
|
|
||||||
|
|
||||||
# Install necessary packages
|
|
||||||
RUN apk add --no-cache \
|
|
||||||
openvpn \
|
|
||||||
squid \
|
|
||||||
curl \
|
|
||||||
bash \
|
|
||||||
iptables \
|
|
||||||
ip6tables \
|
|
||||||
net-tools \
|
|
||||||
procps \
|
|
||||||
&& rm -rf /var/cache/apk/*
|
|
||||||
|
|
||||||
# Create directories
|
|
||||||
RUN mkdir -p /etc/openvpn /var/log/squid /var/cache/squid /var/run/squid
|
|
||||||
|
|
||||||
# Copy configuration files
|
|
||||||
COPY squid.conf /etc/squid/squid.conf
|
|
||||||
COPY start.sh /start.sh
|
|
||||||
COPY version /version
|
|
||||||
|
|
||||||
# Make scripts executable
|
|
||||||
RUN chmod +x /start.sh
|
|
||||||
|
|
||||||
# Expose proxy port
|
|
||||||
EXPOSE 3128
|
|
||||||
|
|
||||||
# Set environment variables
|
|
||||||
ENV PROXY_URL=""
|
|
||||||
ENV VPN_CONFIG=""
|
|
||||||
ENV VPN_TYPE="openvpn"
|
|
||||||
|
|
||||||
# Health check
|
|
||||||
HEALTHCHECK --interval=30s --timeout=10s --start-period=60s --retries=3 \
|
|
||||||
CMD curl -f http://localhost:3128/ || exit 1
|
|
||||||
|
|
||||||
# Start the service
|
|
||||||
CMD ["/start.sh"]
|
|
||||||
@@ -1,169 +0,0 @@
|
|||||||
# VPN Proxy Docker Container
|
|
||||||
|
|
||||||
A Docker container that creates a proxy server routed through a VPN connection (OpenVPN or WireGuard). This allows you to access websites through a VPN tunnel by using the container as a proxy.
|
|
||||||
|
|
||||||
## Features
|
|
||||||
|
|
||||||
- Supports both OpenVPN (.ovpn) and WireGuard (.conf) configurations
|
|
||||||
- Squid proxy server for HTTP/HTTPS traffic
|
|
||||||
- Automatic VPN connection and health monitoring
|
|
||||||
- Configurable target URL for testing
|
|
||||||
- Health checks and automatic restart on failure
|
|
||||||
|
|
||||||
## Usage
|
|
||||||
|
|
||||||
### Prerequisites
|
|
||||||
|
|
||||||
- Docker with `--cap-add=NET_ADMIN` capability
|
|
||||||
- Access to `/dev/net/tun` device (for OpenVPN)
|
|
||||||
- VPN configuration file (.ovpn or .conf)
|
|
||||||
|
|
||||||
### Basic Usage
|
|
||||||
|
|
||||||
1. **Build the container:**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
docker build -t vpn-proxy .
|
|
||||||
```
|
|
||||||
|
|
||||||
2. **Run with OpenVPN:**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
docker run -d \
|
|
||||||
--name vpn-proxy \
|
|
||||||
--cap-add=NET_ADMIN \
|
|
||||||
--device /dev/net/tun \
|
|
||||||
-p 3128:3128 \
|
|
||||||
-e PROXY_URL=https://example.com \
|
|
||||||
-e VPN_CONFIG=/vpn/config.ovpn \
|
|
||||||
-e VPN_TYPE=openvpn \
|
|
||||||
-v /path/to/your/config.ovpn:/vpn/config.ovpn:ro \
|
|
||||||
vpn-proxy
|
|
||||||
```
|
|
||||||
|
|
||||||
3. **Run with WireGuard:**
|
|
||||||
```bash
|
|
||||||
docker run -d \
|
|
||||||
--name vpn-proxy \
|
|
||||||
--cap-add=NET_ADMIN \
|
|
||||||
-p 3128:3128 \
|
|
||||||
-e PROXY_URL=https://example.com \
|
|
||||||
-e VPN_CONFIG=/vpn/wg0.conf \
|
|
||||||
-e VPN_TYPE=wireguard \
|
|
||||||
-v /path/to/your/wg0.conf:/vpn/wg0.conf:ro \
|
|
||||||
vpn-proxy
|
|
||||||
```
|
|
||||||
|
|
||||||
### Environment Variables
|
|
||||||
|
|
||||||
| Variable | Required | Default | Description |
|
|
||||||
| ------------ | -------- | --------- | ---------------------------------- |
|
|
||||||
| `PROXY_URL` | Yes | - | Target URL to proxy through VPN |
|
|
||||||
| `VPN_CONFIG` | Yes | - | Path to VPN configuration file |
|
|
||||||
| `VPN_TYPE` | No | `openvpn` | VPN type: `openvpn` or `wireguard` |
|
|
||||||
|
|
||||||
### Using the Proxy
|
|
||||||
|
|
||||||
Once the container is running, you can use it as a proxy:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Test the proxy
|
|
||||||
curl -x localhost:3128 https://httpbin.org/ip
|
|
||||||
|
|
||||||
# Use in applications
|
|
||||||
export http_proxy=http://localhost:3128
|
|
||||||
export https_proxy=http://localhost:3128
|
|
||||||
```
|
|
||||||
|
|
||||||
### Docker Compose Example
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
version: "3.8"
|
|
||||||
services:
|
|
||||||
vpn-proxy:
|
|
||||||
build: .
|
|
||||||
container_name: vpn-proxy
|
|
||||||
cap_add:
|
|
||||||
- NET_ADMIN
|
|
||||||
devices:
|
|
||||||
- /dev/net/tun
|
|
||||||
ports:
|
|
||||||
- "3128:3128"
|
|
||||||
environment:
|
|
||||||
- PROXY_URL=https://example.com
|
|
||||||
- VPN_CONFIG=/vpn/config.ovpn
|
|
||||||
- VPN_TYPE=openvpn
|
|
||||||
volumes:
|
|
||||||
- ./config.ovpn:/vpn/config.ovpn:ro
|
|
||||||
restart: unless-stopped
|
|
||||||
```
|
|
||||||
|
|
||||||
## Configuration Files
|
|
||||||
|
|
||||||
### OpenVPN Configuration
|
|
||||||
|
|
||||||
Your `.ovpn` file should contain all necessary connection details including:
|
|
||||||
|
|
||||||
- Server address and port
|
|
||||||
- Authentication credentials
|
|
||||||
- Certificate data
|
|
||||||
- Cipher settings
|
|
||||||
|
|
||||||
### WireGuard Configuration
|
|
||||||
|
|
||||||
Your `.conf` file should follow the standard WireGuard format:
|
|
||||||
|
|
||||||
```ini
|
|
||||||
[Interface]
|
|
||||||
PrivateKey = your_private_key
|
|
||||||
Address = 10.0.0.2/24
|
|
||||||
DNS = 8.8.8.8
|
|
||||||
|
|
||||||
[Peer]
|
|
||||||
PublicKey = server_public_key
|
|
||||||
Endpoint = server.example.com:51820
|
|
||||||
AllowedIPs = 0.0.0.0/0
|
|
||||||
```
|
|
||||||
|
|
||||||
## Monitoring
|
|
||||||
|
|
||||||
The container includes health checks and will automatically restart if:
|
|
||||||
|
|
||||||
- VPN connection is lost
|
|
||||||
- Proxy server stops responding
|
|
||||||
- Container receives SIGTERM/SIGINT
|
|
||||||
|
|
||||||
Check logs with:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
docker logs vpn-proxy
|
|
||||||
```
|
|
||||||
|
|
||||||
## Security Notes
|
|
||||||
|
|
||||||
- The container runs with `NET_ADMIN` capability to manage network interfaces
|
|
||||||
- VPN credentials are stored in mounted configuration files
|
|
||||||
- The proxy server is configured to forward all traffic through the VPN
|
|
||||||
- No caching is performed to ensure fresh data through VPN
|
|
||||||
|
|
||||||
## Troubleshooting
|
|
||||||
|
|
||||||
1. **VPN won't connect:**
|
|
||||||
|
|
||||||
- Check that the configuration file is properly mounted
|
|
||||||
- Verify VPN credentials and server availability
|
|
||||||
- Check container logs for specific error messages
|
|
||||||
|
|
||||||
2. **Proxy not working:**
|
|
||||||
|
|
||||||
- Ensure port 3128 is accessible
|
|
||||||
- Test with `curl -x localhost:3128 http://httpbin.org/ip`
|
|
||||||
- Check that the VPN connection is active
|
|
||||||
|
|
||||||
3. **Permission denied:**
|
|
||||||
- Ensure the container has `--cap-add=NET_ADMIN`
|
|
||||||
- For OpenVPN, ensure `/dev/net/tun` is accessible
|
|
||||||
|
|
||||||
## License
|
|
||||||
|
|
||||||
This project is open source and available under the MIT License.
|
|
||||||
@@ -1,25 +0,0 @@
|
|||||||
# VPN Proxy Environment Variables
|
|
||||||
|
|
||||||
# Target URL to proxy (required)
|
|
||||||
# This is the website you want to access through the VPN
|
|
||||||
PROXY_URL=https://example.com
|
|
||||||
|
|
||||||
# VPN Configuration file path (required)
|
|
||||||
# Mount your .ovpn or .conf file and specify the path here
|
|
||||||
VPN_CONFIG=/vpn/config.ovpn
|
|
||||||
|
|
||||||
# VPN Type (optional, defaults to openvpn)
|
|
||||||
# Options: openvpn, wireguard
|
|
||||||
VPN_TYPE=openvpn
|
|
||||||
|
|
||||||
# Example usage:
|
|
||||||
# docker run -d \
|
|
||||||
# --name vpn-proxy \
|
|
||||||
# --cap-add=NET_ADMIN \
|
|
||||||
# --device /dev/net/tun \
|
|
||||||
# -p 3128:3128 \
|
|
||||||
# -e PROXY_URL=https://example.com \
|
|
||||||
# -e VPN_CONFIG=/vpn/config.ovpn \
|
|
||||||
# -e VPN_TYPE=openvpn \
|
|
||||||
# -v /path/to/your/config.ovpn:/vpn/config.ovpn:ro \
|
|
||||||
# vpn-proxy
|
|
||||||
@@ -1,61 +0,0 @@
|
|||||||
# Squid configuration for VPN proxy
|
|
||||||
http_port 3128
|
|
||||||
|
|
||||||
# Allow access from any IP (since we're in a container)
|
|
||||||
acl localnet src 0.0.0.1-0.255.255.255
|
|
||||||
acl localnet src 10.0.0.0/8
|
|
||||||
acl localnet src 100.64.0.0/10
|
|
||||||
acl localnet src 169.254.0.0/16
|
|
||||||
acl localnet src 172.16.0.0/12
|
|
||||||
acl localnet src 192.168.0.0/16
|
|
||||||
acl localnet src fc00::/7
|
|
||||||
acl localnet src fe80::/10
|
|
||||||
|
|
||||||
# Allow all HTTP and HTTPS traffic
|
|
||||||
acl SSL_ports port 443
|
|
||||||
acl Safe_ports port 80
|
|
||||||
acl Safe_ports port 21
|
|
||||||
acl Safe_ports port 443
|
|
||||||
acl Safe_ports port 70
|
|
||||||
acl Safe_ports port 210
|
|
||||||
acl Safe_ports port 1025-65535
|
|
||||||
acl Safe_ports port 280
|
|
||||||
acl Safe_ports port 488
|
|
||||||
acl Safe_ports port 591
|
|
||||||
acl Safe_ports port 777
|
|
||||||
acl CONNECT method CONNECT
|
|
||||||
|
|
||||||
# Deny requests to certain unsafe ports
|
|
||||||
http_access deny !Safe_ports
|
|
||||||
|
|
||||||
# Deny CONNECT to other than secure SSL ports
|
|
||||||
http_access deny CONNECT !SSL_ports
|
|
||||||
|
|
||||||
# Allow localhost management
|
|
||||||
http_access allow localhost manager
|
|
||||||
http_access deny manager
|
|
||||||
|
|
||||||
# Allow access from local networks
|
|
||||||
http_access allow localnet
|
|
||||||
http_access allow localhost
|
|
||||||
|
|
||||||
# Allow all other traffic (since we're proxying through VPN)
|
|
||||||
http_access allow all
|
|
||||||
|
|
||||||
# Cache settings (minimal for proxy)
|
|
||||||
cache_dir ufs /var/cache/squid 100 16 256
|
|
||||||
maximum_object_size 1024 MB
|
|
||||||
|
|
||||||
# Logging
|
|
||||||
access_log /var/log/squid/access.log squid
|
|
||||||
cache_log /var/log/squid/cache.log
|
|
||||||
|
|
||||||
# Don't forward private IPs
|
|
||||||
never_direct allow all
|
|
||||||
|
|
||||||
# Forward all requests through the VPN interface
|
|
||||||
forwarded_for off
|
|
||||||
via off
|
|
||||||
|
|
||||||
# Hide client IP
|
|
||||||
forwarded_for delete
|
|
||||||
@@ -1,218 +0,0 @@
|
|||||||
#!/bin/bash
|
|
||||||
|
|
||||||
set -e
|
|
||||||
|
|
||||||
# Function to log messages
|
|
||||||
log() {
|
|
||||||
echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1"
|
|
||||||
}
|
|
||||||
|
|
||||||
# Function to check if VPN is connected
|
|
||||||
check_vpn() {
|
|
||||||
if [ "$VPN_TYPE" = "openvpn" ]; then
|
|
||||||
# Check if tun interface exists and has an IP
|
|
||||||
if ip addr show tun0 >/dev/null 2>&1; then
|
|
||||||
TUN_IP=$(ip addr show tun0 | grep 'inet ' | awk '{print $2}' | cut -d'/' -f1)
|
|
||||||
if [ -n "$TUN_IP" ]; then
|
|
||||||
return 0
|
|
||||||
fi
|
|
||||||
fi
|
|
||||||
elif [ "$VPN_TYPE" = "wireguard" ]; then
|
|
||||||
# Check if wg interface exists and has an IP
|
|
||||||
if ip addr show wg0 >/dev/null 2>&1; then
|
|
||||||
WG_IP=$(ip addr show wg0 | grep 'inet ' | awk '{print $2}' | cut -d'/' -f1)
|
|
||||||
if [ -n "$WG_IP" ]; then
|
|
||||||
return 0
|
|
||||||
fi
|
|
||||||
fi
|
|
||||||
fi
|
|
||||||
return 1
|
|
||||||
}
|
|
||||||
|
|
||||||
# Function to start OpenVPN
|
|
||||||
start_openvpn() {
|
|
||||||
log "Starting OpenVPN with config: $VPN_CONFIG"
|
|
||||||
|
|
||||||
if [ ! -f "$VPN_CONFIG" ]; then
|
|
||||||
log "ERROR: VPN config file not found: $VPN_CONFIG"
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Start OpenVPN in background
|
|
||||||
openvpn --config "$VPN_CONFIG" --daemon --log /var/log/openvpn.log
|
|
||||||
|
|
||||||
# Wait for VPN to connect
|
|
||||||
log "Waiting for VPN connection..."
|
|
||||||
for i in {1..30}; do
|
|
||||||
if check_vpn; then
|
|
||||||
log "VPN connected successfully"
|
|
||||||
return 0
|
|
||||||
fi
|
|
||||||
sleep 2
|
|
||||||
done
|
|
||||||
|
|
||||||
log "ERROR: VPN failed to connect within 60 seconds"
|
|
||||||
exit 1
|
|
||||||
}
|
|
||||||
|
|
||||||
# Function to start WireGuard
|
|
||||||
start_wireguard() {
|
|
||||||
log "Starting WireGuard with config: $VPN_CONFIG"
|
|
||||||
|
|
||||||
if [ ! -f "$VPN_CONFIG" ]; then
|
|
||||||
log "ERROR: WireGuard config file not found: $VPN_CONFIG"
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Start WireGuard
|
|
||||||
wg-quick up "$VPN_CONFIG"
|
|
||||||
|
|
||||||
# Wait for WireGuard to connect
|
|
||||||
log "Waiting for WireGuard connection..."
|
|
||||||
for i in {1..30}; do
|
|
||||||
if check_vpn; then
|
|
||||||
log "WireGuard connected successfully"
|
|
||||||
return 0
|
|
||||||
fi
|
|
||||||
sleep 2
|
|
||||||
done
|
|
||||||
|
|
||||||
log "ERROR: WireGuard failed to connect within 60 seconds"
|
|
||||||
exit 1
|
|
||||||
}
|
|
||||||
|
|
||||||
# Function to setup routing
|
|
||||||
setup_routing() {
|
|
||||||
log "Setting up routing for VPN proxy"
|
|
||||||
|
|
||||||
# Get the VPN interface IP
|
|
||||||
if [ "$VPN_TYPE" = "openvpn" ]; then
|
|
||||||
VPN_IP=$(ip addr show tun0 | grep 'inet ' | awk '{print $2}' | cut -d'/' -f1)
|
|
||||||
VPN_INTERFACE="tun0"
|
|
||||||
elif [ "$VPN_TYPE" = "wireguard" ]; then
|
|
||||||
VPN_IP=$(ip addr show wg0 | grep 'inet ' | awk '{print $2}' | cut -d'/' -f1)
|
|
||||||
VPN_INTERFACE="wg0"
|
|
||||||
fi
|
|
||||||
|
|
||||||
log "VPN Interface: $VPN_INTERFACE, IP: $VPN_IP"
|
|
||||||
|
|
||||||
# Set up iptables rules to route traffic through VPN
|
|
||||||
iptables -t nat -A OUTPUT -p tcp --dport 80 -j DNAT --to-destination $VPN_IP:80
|
|
||||||
iptables -t nat -A OUTPUT -p tcp --dport 443 -j DNAT --to-destination $VPN_IP:443
|
|
||||||
|
|
||||||
# Allow traffic through VPN interface
|
|
||||||
iptables -A OUTPUT -o $VPN_INTERFACE -j ACCEPT
|
|
||||||
iptables -A INPUT -i $VPN_INTERFACE -j ACCEPT
|
|
||||||
}
|
|
||||||
|
|
||||||
# Function to start Squid proxy
|
|
||||||
start_squid() {
|
|
||||||
log "Starting Squid proxy server"
|
|
||||||
|
|
||||||
# Initialize Squid cache
|
|
||||||
squid -z -N -d 1
|
|
||||||
|
|
||||||
# Start Squid
|
|
||||||
squid -N -d 1 &
|
|
||||||
SQUID_PID=$!
|
|
||||||
|
|
||||||
# Wait for Squid to start
|
|
||||||
sleep 5
|
|
||||||
|
|
||||||
if kill -0 $SQUID_PID 2>/dev/null; then
|
|
||||||
log "Squid proxy started successfully on port 3128"
|
|
||||||
else
|
|
||||||
log "ERROR: Failed to start Squid proxy"
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
}
|
|
||||||
|
|
||||||
# Function to test proxy
|
|
||||||
test_proxy() {
|
|
||||||
log "Testing proxy connection"
|
|
||||||
|
|
||||||
# Test if we can reach the target URL through the proxy
|
|
||||||
if [ -n "$PROXY_URL" ]; then
|
|
||||||
log "Testing connection to: $PROXY_URL"
|
|
||||||
if curl -x localhost:3128 --connect-timeout 10 --max-time 30 -s -o /dev/null "$PROXY_URL"; then
|
|
||||||
log "Proxy test successful - can reach $PROXY_URL"
|
|
||||||
else
|
|
||||||
log "WARNING: Proxy test failed - cannot reach $PROXY_URL"
|
|
||||||
fi
|
|
||||||
fi
|
|
||||||
}
|
|
||||||
|
|
||||||
# Main execution
|
|
||||||
main() {
|
|
||||||
log "Starting VPN Proxy container"
|
|
||||||
|
|
||||||
# Check required environment variables
|
|
||||||
if [ -z "$PROXY_URL" ]; then
|
|
||||||
log "ERROR: PROXY_URL environment variable is required"
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [ -z "$VPN_CONFIG" ]; then
|
|
||||||
log "ERROR: VPN_CONFIG environment variable is required"
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
log "Configuration:"
|
|
||||||
log " PROXY_URL: $PROXY_URL"
|
|
||||||
log " VPN_CONFIG: $VPN_CONFIG"
|
|
||||||
log " VPN_TYPE: ${VPN_TYPE:-openvpn}"
|
|
||||||
|
|
||||||
# Start VPN based on type
|
|
||||||
if [ "$VPN_TYPE" = "wireguard" ]; then
|
|
||||||
start_wireguard
|
|
||||||
else
|
|
||||||
start_openvpn
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Setup routing
|
|
||||||
setup_routing
|
|
||||||
|
|
||||||
# Start Squid proxy
|
|
||||||
start_squid
|
|
||||||
|
|
||||||
# Test proxy
|
|
||||||
test_proxy
|
|
||||||
|
|
||||||
log "VPN Proxy is ready and listening on port 3128"
|
|
||||||
log "Use this container as a proxy: http://localhost:3128"
|
|
||||||
|
|
||||||
# Keep container running and monitor
|
|
||||||
while true; do
|
|
||||||
if ! check_vpn; then
|
|
||||||
log "ERROR: VPN connection lost, restarting..."
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
if ! kill -0 $SQUID_PID 2>/dev/null; then
|
|
||||||
log "ERROR: Squid proxy died, restarting..."
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
sleep 30
|
|
||||||
done
|
|
||||||
}
|
|
||||||
|
|
||||||
# Handle shutdown
|
|
||||||
cleanup() {
|
|
||||||
log "Shutting down VPN Proxy"
|
|
||||||
if [ -n "$SQUID_PID" ]; then
|
|
||||||
kill $SQUID_PID 2>/dev/null || true
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [ "$VPN_TYPE" = "wireguard" ] && [ -f "$VPN_CONFIG" ]; then
|
|
||||||
wg-quick down "$VPN_CONFIG" 2>/dev/null || true
|
|
||||||
fi
|
|
||||||
|
|
||||||
exit 0
|
|
||||||
}
|
|
||||||
|
|
||||||
# Set up signal handlers
|
|
||||||
trap cleanup SIGTERM SIGINT
|
|
||||||
|
|
||||||
# Run main function
|
|
||||||
main "$@"
|
|
||||||
@@ -1 +0,0 @@
|
|||||||
1.0.0
|
|
||||||
Reference in New Issue
Block a user