ocr api
Build and Push Docker Images / build-and-push (push) Failing after 2m5s

This commit is contained in:
2025-07-17 16:00:42 +02:00
parent 7860aa3579
commit fed33b7623
9 changed files with 893 additions and 2 deletions
+149
View File
@@ -0,0 +1,149 @@
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class
# C extensions
*.so
# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
pip-wheel-metadata/
share/python-wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST
# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec
# Installer logs
pip-log.txt
pip-delete-this-directory.txt
# Unit test / coverage reports
htmlcov/
.tox/
.nox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
*.py,cover
.hypothesis/
.pytest_cache/
# Translations
*.mo
*.pot
# Django stuff:
*.log
local_settings.py
db.sqlite3
db.sqlite3-journal
# Flask stuff:
instance/
.webassets-cache
# Scrapy stuff:
.scrapy
# Sphinx documentation
docs/_build/
# PyBuilder
target/
# Jupyter Notebook
.ipynb_checkpoints
# IPython
profile_default/
ipython_config.py
# pyenv
.python-version
# pipenv
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
# However, in case of collaboration, if having platform-specific dependencies or dependencies
# having no cross-platform support, pipenv may install dependencies that don't work, or not
# install all needed dependencies.
#Pipfile.lock
# PEP 582; used by e.g. github.com/David-OConnor/pyflow
__pypackages__/
# Celery stuff
celerybeat-schedule
celerybeat.pid
# SageMath parsed files
*.sage.py
# Environments
.env
.venv
env/
venv/
ENV/
env.bak/
venv.bak/
# Spyder project settings
.spyderproject
.spyproject
# Rope project settings
.ropeproject
# mkdocs documentation
/site
# mypy
.mypy_cache/
.dmypy.json
dmypy.json
# Pyre type checker
.pyre/
# Test files
test_files/
*.pdf
*.png
*.jpg
*.jpeg
*.bmp
*.tiff
*.tif
# IDE
.vscode/
.idea/
*.swp
*.swo
# OS
.DS_Store
Thumbs.db
+46
View File
@@ -0,0 +1,46 @@
# Use Python 3.11 slim image as base
FROM python:3.11-slim
# Set environment variables
ENV PYTHONUNBUFFERED=1
ENV PYTHONDONTWRITEBYTECODE=1
# Install system dependencies
RUN apt-get update && apt-get install -y \
tesseract-ocr \
tesseract-ocr-eng \
tesseract-ocr-fra \
tesseract-ocr-deu \
tesseract-ocr-spa \
tesseract-ocr-ita \
poppler-utils \
libpoppler-cpp-dev \
pkg-config \
&& rm -rf /var/lib/apt/lists/*
# Set working directory
WORKDIR /app
# Copy requirements first for better caching
COPY requirements.txt .
# Install Python dependencies
RUN pip install --no-cache-dir -r requirements.txt
# Copy application code
COPY main.py .
# Create a non-root user
RUN useradd --create-home --shell /bin/bash app \
&& chown -R app:app /app
USER app
# Expose port
EXPOSE 8000
# Health check
HEALTHCHECK --interval=30s --timeout=30s --start-period=5s --retries=3 \
CMD curl -f http://localhost:8000/health || exit 1
# Run the application
CMD ["python", "main.py"]
+207
View File
@@ -0,0 +1,207 @@
# OCR API
A FastAPI-based OCR (Optical Character Recognition) service that can extract text from PDF files and images using Tesseract.
## Features
- **PDF Processing**: Convert PDF files to images and extract text using OCR
- **Image Processing**: Direct OCR processing of image files (PNG, JPG, JPEG, BMP, TIFF)
- **Multi-language Support**: Supports English, French, German, Spanish, and Italian
- **Confidence Scoring**: Returns confidence scores for OCR results
- **RESTful API**: Clean, documented API endpoints
- **Docker Support**: Easy deployment with Docker and Docker Compose
- **File Validation**: Built-in file type and size validation
- **Background Processing**: Efficient file cleanup and processing
## Quick Start
### Using Docker (Recommended)
1. **Clone and build the container:**
```bash
docker-compose up --build
```
2. **Access the API:**
- API Documentation: http://localhost:8000/docs
- Health Check: http://localhost:8000/health
### Local Development
1. **Install system dependencies:**
```bash
# Ubuntu/Debian
sudo apt-get update
sudo apt-get install tesseract-ocr tesseract-ocr-eng poppler-utils libpoppler-cpp-dev pkg-config
# macOS
brew install tesseract poppler
# Arch Linux
sudo pacman -S tesseract tesseract-data-eng poppler
```
2. **Install Python dependencies:**
```bash
pip install -r requirements.txt
```
3. **Run the application:**
```bash
python main.py
```
## API Endpoints
### Health Check
- **GET** `/health` - Check if the service is running
### PDF OCR
- **POST** `/ocr/pdf` - Extract text from PDF files
- **File**: PDF file (max 50MB)
- **Response**: Extracted text with metadata
### Image OCR
- **POST** `/ocr/image` - Extract text from image files
- **File**: Image file (PNG, JPG, JPEG, BMP, TIFF)
- **Response**: Extracted text with metadata
## Usage Examples
### Using curl
**Process a PDF file:**
```bash
curl -X POST "http://localhost:8000/ocr/pdf" \
-H "accept: application/json" \
-H "Content-Type: multipart/form-data" \
-F "file=@document.pdf"
```
**Process an image file:**
```bash
curl -X POST "http://localhost:8000/ocr/image" \
-H "accept: application/json" \
-H "Content-Type: multipart/form-data" \
-F "file=@image.png"
```
### Using Python
```python
import requests
# Process PDF
with open('document.pdf', 'rb') as f:
files = {'file': f}
response = requests.post('http://localhost:8000/ocr/pdf', files=files)
result = response.json()
print(f"Extracted text: {result['text']}")
print(f"Processing time: {result['processing_time']}s")
print(f"Confidence: {result['confidence']}%")
```
### Using JavaScript/Fetch
```javascript
// Process PDF
const formData = new FormData();
formData.append("file", fileInput.files[0]);
fetch("http://localhost:8000/ocr/pdf", {
method: "POST",
body: formData,
})
.then((response) => response.json())
.then((data) => {
console.log("Extracted text:", data.text);
console.log("Processing time:", data.processing_time);
console.log("Confidence:", data.confidence);
});
```
## Response Format
```json
{
"text": "Extracted text content...",
"pages": 3,
"processing_time": 2.45,
"confidence": 85.2
}
```
## Configuration
### Environment Variables
- `PYTHONUNBUFFERED=1` - Ensures Python output is not buffered
- `PYTHONDONTWRITEBYTECODE=1` - Prevents Python from writing .pyc files
### Tesseract Configuration
The Docker image includes multiple language packs:
- English (eng)
- French (fra)
- German (deu)
- Spanish (spa)
- Italian (ita)
To add more languages, modify the Dockerfile and add additional `tesseract-ocr-*` packages.
## Performance Tips
1. **Image Quality**: Higher DPI (300) provides better OCR accuracy but slower processing
2. **File Size**: Larger files take longer to process
3. **Text Quality**: Clear, high-contrast text yields better results
4. **Language**: Specify the correct language for better accuracy
## Troubleshooting
### Common Issues
1. **Tesseract not found**: Ensure Tesseract is installed on your system
2. **PDF processing errors**: Check if Poppler utilities are installed
3. **Memory issues**: Reduce DPI or process smaller files
4. **Poor OCR quality**: Ensure images are clear and have good contrast
### Logs
The application logs processing information. Check Docker logs:
```bash
docker-compose logs ocr-api
```
## Development
### Project Structure
```
ocr-api/
├── main.py # FastAPI application
├── requirements.txt # Python dependencies
├── Dockerfile # Docker configuration
├── docker-compose.yml # Docker Compose configuration
└── README.md # This file
```
### Adding Features
1. **New OCR engines**: Modify the `process_pdf_with_ocr` function
2. **Additional file formats**: Add new endpoints and processing logic
3. **Custom preprocessing**: Add image preprocessing steps before OCR
4. **Batch processing**: Implement queue-based processing for multiple files
## License
This project is open source and available under the MIT License.
+19
View File
@@ -0,0 +1,19 @@
version: "3.8"
services:
ocr-api:
build: .
ports:
- "8000:8000"
environment:
- PYTHONUNBUFFERED=1
volumes:
# Optional: mount a directory for testing with sample files
- ./test_files:/app/test_files:ro
restart: unless-stopped
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
interval: 30s
timeout: 10s
retries: 3
start_period: 40s
+225
View File
@@ -0,0 +1,225 @@
import os
import tempfile
import logging
from typing import List, Optional
from pathlib import Path
from fastapi import FastAPI, File, UploadFile, HTTPException, BackgroundTasks
from fastapi.responses import JSONResponse
from pydantic import BaseModel
import pytesseract
from PIL import Image
from pdf2image import convert_from_path
import uvicorn
# Configure logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
app = FastAPI(
title="OCR API",
description="API for extracting text from PDF files using OCR",
version="1.0.0"
)
class OCRResponse(BaseModel):
text: str
pages: int
processing_time: float
confidence: Optional[float] = None
class ErrorResponse(BaseModel):
error: str
detail: str
@app.get("/")
async def root():
return {"message": "OCR API is running. Use /docs for API documentation."}
@app.get("/health")
async def health_check():
return {"status": "healthy", "service": "OCR API"}
async def process_pdf_with_ocr(pdf_path: str) -> OCRResponse:
"""
Process a PDF file and extract text using OCR
"""
import time
start_time = time.time()
try:
# Convert PDF to images
logger.info(f"Converting PDF to images: {pdf_path}")
images = convert_from_path(pdf_path, dpi=300)
if not images:
raise HTTPException(status_code=400, detail="Could not extract images from PDF")
# Extract text from each page
all_text = []
total_confidence = 0
confidence_count = 0
for i, image in enumerate(images):
logger.info(f"Processing page {i+1}/{len(images)}")
# Perform OCR on the image
ocr_data = pytesseract.image_to_data(image, output_type=pytesseract.Output.DICT)
# Extract text and confidence scores
page_text = []
for j in range(len(ocr_data['text'])):
if int(ocr_data['conf'][j]) > 0: # Filter out low confidence results
page_text.append(ocr_data['text'][j])
total_confidence += float(ocr_data['conf'][j])
confidence_count += 1
page_text = ' '.join(page_text).strip()
if page_text:
all_text.append(f"--- Page {i+1} ---\n{page_text}")
# Calculate average confidence
avg_confidence = total_confidence / confidence_count if confidence_count > 0 else None
processing_time = time.time() - start_time
return OCRResponse(
text='\n\n'.join(all_text),
pages=len(images),
processing_time=round(processing_time, 2),
confidence=round(avg_confidence, 2) if avg_confidence else None
)
except Exception as e:
logger.error(f"Error processing PDF: {str(e)}")
raise HTTPException(status_code=500, detail=f"Error processing PDF: {str(e)}")
@app.post("/ocr/pdf", response_model=OCRResponse)
async def extract_text_from_pdf(
file: UploadFile = File(...),
background_tasks: BackgroundTasks = None
):
"""
Extract text from a PDF file using OCR
- **file**: PDF file to process
- Returns extracted text with metadata
"""
# Validate file type
if not file.filename.lower().endswith('.pdf'):
raise HTTPException(
status_code=400,
detail="File must be a PDF"
)
# Check file size (limit to 50MB)
if file.size and file.size > 50 * 1024 * 1024:
raise HTTPException(
status_code=400,
detail="File size too large. Maximum size is 50MB"
)
try:
# Create temporary file
with tempfile.NamedTemporaryFile(delete=False, suffix='.pdf') as temp_file:
# Write uploaded file to temporary file
content = await file.read()
temp_file.write(content)
temp_file_path = temp_file.name
try:
# Process the PDF
result = await process_pdf_with_ocr(temp_file_path)
# Clean up temporary file
background_tasks.add_task(os.unlink, temp_file_path)
return result
except Exception as e:
# Clean up temporary file on error
if os.path.exists(temp_file_path):
os.unlink(temp_file_path)
raise e
except Exception as e:
logger.error(f"Error handling file upload: {str(e)}")
raise HTTPException(status_code=500, detail=f"Error processing file: {str(e)}")
@app.post("/ocr/image", response_model=OCRResponse)
async def extract_text_from_image(
file: UploadFile = File(...),
background_tasks: BackgroundTasks = None
):
"""
Extract text from an image file using OCR
- **file**: Image file to process (PNG, JPG, JPEG, etc.)
- Returns extracted text with metadata
"""
# Validate file type
allowed_extensions = {'.png', '.jpg', '.jpeg', '.bmp', '.tiff', '.tif'}
file_extension = Path(file.filename).suffix.lower()
if file_extension not in allowed_extensions:
raise HTTPException(
status_code=400,
detail=f"Unsupported file type. Allowed types: {', '.join(allowed_extensions)}"
)
try:
# Create temporary file
with tempfile.NamedTemporaryFile(delete=False, suffix=file_extension) as temp_file:
content = await file.read()
temp_file.write(content)
temp_file_path = temp_file.name
try:
import time
start_time = time.time()
# Open and process image
image = Image.open(temp_file_path)
# Perform OCR
ocr_data = pytesseract.image_to_data(image, output_type=pytesseract.Output.DICT)
# Extract text and confidence
text_parts = []
total_confidence = 0
confidence_count = 0
for i in range(len(ocr_data['text'])):
if int(ocr_data['conf'][i]) > 0:
text_parts.append(ocr_data['text'][i])
total_confidence += float(ocr_data['conf'][i])
confidence_count += 1
text = ' '.join(text_parts).strip()
avg_confidence = total_confidence / confidence_count if confidence_count > 0 else None
processing_time = time.time() - start_time
# Clean up temporary file
background_tasks.add_task(os.unlink, temp_file_path)
return OCRResponse(
text=text,
pages=1,
processing_time=round(processing_time, 2),
confidence=round(avg_confidence, 2) if avg_confidence else None
)
except Exception as e:
# Clean up temporary file on error
if os.path.exists(temp_file_path):
os.unlink(temp_file_path)
raise e
except Exception as e:
logger.error(f"Error processing image: {str(e)}")
raise HTTPException(status_code=500, detail=f"Error processing image: {str(e)}")
if __name__ == "__main__":
uvicorn.run(app, host="0.0.0.0", port=8000)
+8
View File
@@ -0,0 +1,8 @@
fastapi==0.104.1
uvicorn[standard]==0.24.0
python-multipart==0.0.6
Pillow==10.1.0
pytesseract==0.3.10
pdf2image==1.16.3
pydantic==2.5.0
python-dotenv==1.0.0
+109
View File
@@ -0,0 +1,109 @@
#!/usr/bin/env python3
"""
Test client for the OCR API
This script demonstrates how to use the OCR API endpoints
"""
import requests
import json
import sys
from pathlib import Path
# API base URL
BASE_URL = "http://localhost:8000"
def test_health():
"""Test the health endpoint"""
try:
response = requests.get(f"{BASE_URL}/health")
print(f"Health check: {response.status_code} - {response.json()}")
return response.status_code == 200
except requests.exceptions.ConnectionError:
print("❌ Could not connect to the API. Make sure it's running on localhost:8000")
return False
def test_pdf_ocr(file_path):
"""Test PDF OCR endpoint"""
if not Path(file_path).exists():
print(f"❌ File not found: {file_path}")
return False
try:
with open(file_path, 'rb') as f:
files = {'file': f}
print(f"📄 Processing PDF: {file_path}")
response = requests.post(f"{BASE_URL}/ocr/pdf", files=files)
if response.status_code == 200:
result = response.json()
print(f"✅ PDF OCR successful!")
print(f" Pages: {result['pages']}")
print(f" Processing time: {result['processing_time']}s")
print(f" Confidence: {result['confidence']}%")
print(f" Text preview: {result['text'][:200]}...")
return True
else:
print(f"❌ PDF OCR failed: {response.status_code} - {response.text}")
return False
except Exception as e:
print(f"❌ Error processing PDF: {str(e)}")
return False
def test_image_ocr(file_path):
"""Test image OCR endpoint"""
if not Path(file_path).exists():
print(f"❌ File not found: {file_path}")
return False
try:
with open(file_path, 'rb') as f:
files = {'file': f}
print(f"🖼️ Processing image: {file_path}")
response = requests.post(f"{BASE_URL}/ocr/image", files=files)
if response.status_code == 200:
result = response.json()
print(f"✅ Image OCR successful!")
print(f" Processing time: {result['processing_time']}s")
print(f" Confidence: {result['confidence']}%")
print(f" Text: {result['text']}")
return True
else:
print(f"❌ Image OCR failed: {response.status_code} - {response.text}")
return False
except Exception as e:
print(f"❌ Error processing image: {str(e)}")
return False
def main():
"""Main test function"""
print("🧪 OCR API Test Client")
print("=" * 50)
# Test health endpoint
if not test_health():
sys.exit(1)
print("\n" + "=" * 50)
# Test PDF OCR if file provided
if len(sys.argv) > 1:
file_path = sys.argv[1]
file_ext = Path(file_path).suffix.lower()
if file_ext == '.pdf':
test_pdf_ocr(file_path)
elif file_ext in ['.png', '.jpg', '.jpeg', '.bmp', '.tiff', '.tif']:
test_image_ocr(file_path)
else:
print(f"❌ Unsupported file type: {file_ext}")
print("Supported types: .pdf, .png, .jpg, .jpeg, .bmp, .tiff, .tif")
else:
print("📝 Usage: python test_client.py <file_path>")
print(" Example: python test_client.py document.pdf")
print(" Example: python test_client.py image.png")
if __name__ == "__main__":
main()
@@ -0,0 +1,96 @@
#!/usr/bin/env python3
"""
Test script to verify PostgreSQL database creation and setup
"""
import os
import sys
# Add the app directory to the Python path
sys.path.insert(0, os.path.join(os.path.dirname(__file__), 'app'))
def test_postgres_setup():
"""Test PostgreSQL database creation and setup"""
print("=== Testing PostgreSQL Database Setup ===")
# Check if PostgreSQL credentials are available
required_vars = ['POSTGRES_HOST', 'POSTGRES_DB', 'POSTGRES_USER', 'POSTGRES_PASSWORD']
missing_vars = [var for var in required_vars if not os.getenv(var)]
if missing_vars:
print(f"⚠ PostgreSQL test skipped - missing environment variables: {', '.join(missing_vars)}")
print("Set these variables to test PostgreSQL:")
print(" POSTGRES_HOST=your-host")
print(" POSTGRES_DB=your-database")
print(" POSTGRES_USER=your-username")
print(" POSTGRES_PASSWORD=your-password")
return False
try:
from app.database import db_manager, init_db
print(f"Database type: {db_manager.db_type}")
print(f"PostgreSQL host: {os.getenv('POSTGRES_HOST')}")
print(f"PostgreSQL database: {os.getenv('POSTGRES_DB')}")
print(f"PostgreSQL user: {os.getenv('POSTGRES_USER')}")
# Initialize database (this will create the database if it doesn't exist)
init_db()
print("✓ Database initialized successfully")
# Test a simple cache operation
from app.database import save_to_cache, get_cached_data
test_url = "https://example.com"
test_route = "test"
test_data = {"title": "Test Page", "content": "Test content"}
# Save to cache
save_to_cache(test_url, test_route, test_data)
print("✓ Data saved to cache")
# Retrieve from cache
cached_data = get_cached_data(test_url, test_route)
if cached_data and cached_data == test_data:
print("✓ Data retrieved from cache successfully")
else:
print("✗ Failed to retrieve data from cache")
return False
# Test cache stats
from app.services.cache import get_cache_stats
stats = get_cache_stats()
if stats['status'] == 'success':
print("✓ Cache stats retrieved successfully")
print(f" Database type: {stats['stats']['database_type']}")
print(f" Total entries: {stats['stats']['total_entries']}")
else:
print("✗ Failed to get cache stats")
return False
print("✓ PostgreSQL database setup test passed!")
return True
except Exception as e:
print(f"✗ PostgreSQL setup test failed: {e}")
import traceback
traceback.print_exc()
return False
def main():
"""Run the PostgreSQL setup test"""
print("PostgreSQL Database Setup Test")
print("=" * 50)
success = test_postgres_setup()
print("\n" + "=" * 50)
if success:
print("🎉 PostgreSQL setup test passed! Database creation and connection working correctly.")
return 0
else:
print("❌ PostgreSQL setup test failed. Please check the configuration and PostgreSQL server.")
return 1
if __name__ == "__main__":
sys.exit(main())
+34 -2
View File
@@ -167,7 +167,7 @@ The container logs all health check activities to both stdout and a log file (`/
Each log entry includes the host name for easy identification: Each log entry includes the host name for easy identification:
``` ```
2024-01-15 10:30:00 - INFO - Performing health check for server1: http://server1:8000/?url=https%3A//www.google.com&skipCache=true 2024-01-15 10:30:00 - INFO - Performing health check for server1: http://server1:8000/?url=https%3A//www.google.com
2024-01-15 10:30:01 - INFO - Health check passed for server1 - API is responding correctly 2024-01-15 10:30:01 - INFO - Health check passed for server1 - API is responding correctly
2024-01-15 10:30:02 - WARNING - Health check failed for server2 - Status code: 500 2024-01-15 10:30:02 - WARNING - Health check failed for server2 - Status code: 500
``` ```
@@ -181,6 +181,38 @@ Each log entry includes the host name for easy identification:
## Troubleshooting ## Troubleshooting
### USB Device Mounting Error
If you encounter an error like:
```
error creating device nodes: mount src=/dev/bus/usb/003/003, dst=/var/lib/docker/overlay2/.../merged/dev/bus/usb/003/003: no such file or directory
```
**Solution**: Remove `privileged: true` from your Docker Compose configuration. The healthcheck container doesn't need privileged access.
**Correct configuration:**
```yaml
puppeteer-healthcheck:
image: bramkel/puppeteer-healthcheck:latest
container_name: puppeteer-healthcheck
# Remove this line: privileged: true
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
environment:
- HOSTS=puppeteer-api-1,puppeteer-api-2
- TEST_URL=https://www.google.com
- MAX_CONSECUTIVE_FAILURES=2
- API_KEY=${PUPPETEER_API_KEY}
- CHECK_INTERVAL=60
- TIMEOUT=15
restart: unless-stopped
depends_on:
- puppeteer-api-1
- puppeteer-api-2
```
### Docker Permission Issues ### Docker Permission Issues
If you see "Permission denied" errors when accessing the Docker socket: If you see "Permission denied" errors when accessing the Docker socket:
@@ -246,7 +278,7 @@ docker build -t puppeteer-healthcheck .
## Version ## Version
Current version: 2.0.0 Current version: latest
### Migration from v1.x ### Migration from v1.x