This commit is contained in:
@@ -0,0 +1,149 @@
|
|||||||
|
# Byte-compiled / optimized / DLL files
|
||||||
|
__pycache__/
|
||||||
|
*.py[cod]
|
||||||
|
*$py.class
|
||||||
|
|
||||||
|
# C extensions
|
||||||
|
*.so
|
||||||
|
|
||||||
|
# Distribution / packaging
|
||||||
|
.Python
|
||||||
|
build/
|
||||||
|
develop-eggs/
|
||||||
|
dist/
|
||||||
|
downloads/
|
||||||
|
eggs/
|
||||||
|
.eggs/
|
||||||
|
lib/
|
||||||
|
lib64/
|
||||||
|
parts/
|
||||||
|
sdist/
|
||||||
|
var/
|
||||||
|
wheels/
|
||||||
|
pip-wheel-metadata/
|
||||||
|
share/python-wheels/
|
||||||
|
*.egg-info/
|
||||||
|
.installed.cfg
|
||||||
|
*.egg
|
||||||
|
MANIFEST
|
||||||
|
|
||||||
|
# PyInstaller
|
||||||
|
# Usually these files are written by a python script from a template
|
||||||
|
# before PyInstaller builds the exe, so as to inject date/other infos into it.
|
||||||
|
*.manifest
|
||||||
|
*.spec
|
||||||
|
|
||||||
|
# Installer logs
|
||||||
|
pip-log.txt
|
||||||
|
pip-delete-this-directory.txt
|
||||||
|
|
||||||
|
# Unit test / coverage reports
|
||||||
|
htmlcov/
|
||||||
|
.tox/
|
||||||
|
.nox/
|
||||||
|
.coverage
|
||||||
|
.coverage.*
|
||||||
|
.cache
|
||||||
|
nosetests.xml
|
||||||
|
coverage.xml
|
||||||
|
*.cover
|
||||||
|
*.py,cover
|
||||||
|
.hypothesis/
|
||||||
|
.pytest_cache/
|
||||||
|
|
||||||
|
# Translations
|
||||||
|
*.mo
|
||||||
|
*.pot
|
||||||
|
|
||||||
|
# Django stuff:
|
||||||
|
*.log
|
||||||
|
local_settings.py
|
||||||
|
db.sqlite3
|
||||||
|
db.sqlite3-journal
|
||||||
|
|
||||||
|
# Flask stuff:
|
||||||
|
instance/
|
||||||
|
.webassets-cache
|
||||||
|
|
||||||
|
# Scrapy stuff:
|
||||||
|
.scrapy
|
||||||
|
|
||||||
|
# Sphinx documentation
|
||||||
|
docs/_build/
|
||||||
|
|
||||||
|
# PyBuilder
|
||||||
|
target/
|
||||||
|
|
||||||
|
# Jupyter Notebook
|
||||||
|
.ipynb_checkpoints
|
||||||
|
|
||||||
|
# IPython
|
||||||
|
profile_default/
|
||||||
|
ipython_config.py
|
||||||
|
|
||||||
|
# pyenv
|
||||||
|
.python-version
|
||||||
|
|
||||||
|
# pipenv
|
||||||
|
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
|
||||||
|
# However, in case of collaboration, if having platform-specific dependencies or dependencies
|
||||||
|
# having no cross-platform support, pipenv may install dependencies that don't work, or not
|
||||||
|
# install all needed dependencies.
|
||||||
|
#Pipfile.lock
|
||||||
|
|
||||||
|
# PEP 582; used by e.g. github.com/David-OConnor/pyflow
|
||||||
|
__pypackages__/
|
||||||
|
|
||||||
|
# Celery stuff
|
||||||
|
celerybeat-schedule
|
||||||
|
celerybeat.pid
|
||||||
|
|
||||||
|
# SageMath parsed files
|
||||||
|
*.sage.py
|
||||||
|
|
||||||
|
# Environments
|
||||||
|
.env
|
||||||
|
.venv
|
||||||
|
env/
|
||||||
|
venv/
|
||||||
|
ENV/
|
||||||
|
env.bak/
|
||||||
|
venv.bak/
|
||||||
|
|
||||||
|
# Spyder project settings
|
||||||
|
.spyderproject
|
||||||
|
.spyproject
|
||||||
|
|
||||||
|
# Rope project settings
|
||||||
|
.ropeproject
|
||||||
|
|
||||||
|
# mkdocs documentation
|
||||||
|
/site
|
||||||
|
|
||||||
|
# mypy
|
||||||
|
.mypy_cache/
|
||||||
|
.dmypy.json
|
||||||
|
dmypy.json
|
||||||
|
|
||||||
|
# Pyre type checker
|
||||||
|
.pyre/
|
||||||
|
|
||||||
|
# Test files
|
||||||
|
test_files/
|
||||||
|
*.pdf
|
||||||
|
*.png
|
||||||
|
*.jpg
|
||||||
|
*.jpeg
|
||||||
|
*.bmp
|
||||||
|
*.tiff
|
||||||
|
*.tif
|
||||||
|
|
||||||
|
# IDE
|
||||||
|
.vscode/
|
||||||
|
.idea/
|
||||||
|
*.swp
|
||||||
|
*.swo
|
||||||
|
|
||||||
|
# OS
|
||||||
|
.DS_Store
|
||||||
|
Thumbs.db
|
||||||
@@ -0,0 +1,46 @@
|
|||||||
|
# Use Python 3.11 slim image as base
|
||||||
|
FROM python:3.11-slim
|
||||||
|
|
||||||
|
# Set environment variables
|
||||||
|
ENV PYTHONUNBUFFERED=1
|
||||||
|
ENV PYTHONDONTWRITEBYTECODE=1
|
||||||
|
|
||||||
|
# Install system dependencies
|
||||||
|
RUN apt-get update && apt-get install -y \
|
||||||
|
tesseract-ocr \
|
||||||
|
tesseract-ocr-eng \
|
||||||
|
tesseract-ocr-fra \
|
||||||
|
tesseract-ocr-deu \
|
||||||
|
tesseract-ocr-spa \
|
||||||
|
tesseract-ocr-ita \
|
||||||
|
poppler-utils \
|
||||||
|
libpoppler-cpp-dev \
|
||||||
|
pkg-config \
|
||||||
|
&& rm -rf /var/lib/apt/lists/*
|
||||||
|
|
||||||
|
# Set working directory
|
||||||
|
WORKDIR /app
|
||||||
|
|
||||||
|
# Copy requirements first for better caching
|
||||||
|
COPY requirements.txt .
|
||||||
|
|
||||||
|
# Install Python dependencies
|
||||||
|
RUN pip install --no-cache-dir -r requirements.txt
|
||||||
|
|
||||||
|
# Copy application code
|
||||||
|
COPY main.py .
|
||||||
|
|
||||||
|
# Create a non-root user
|
||||||
|
RUN useradd --create-home --shell /bin/bash app \
|
||||||
|
&& chown -R app:app /app
|
||||||
|
USER app
|
||||||
|
|
||||||
|
# Expose port
|
||||||
|
EXPOSE 8000
|
||||||
|
|
||||||
|
# Health check
|
||||||
|
HEALTHCHECK --interval=30s --timeout=30s --start-period=5s --retries=3 \
|
||||||
|
CMD curl -f http://localhost:8000/health || exit 1
|
||||||
|
|
||||||
|
# Run the application
|
||||||
|
CMD ["python", "main.py"]
|
||||||
@@ -0,0 +1,207 @@
|
|||||||
|
# OCR API
|
||||||
|
|
||||||
|
A FastAPI-based OCR (Optical Character Recognition) service that can extract text from PDF files and images using Tesseract.
|
||||||
|
|
||||||
|
## Features
|
||||||
|
|
||||||
|
- **PDF Processing**: Convert PDF files to images and extract text using OCR
|
||||||
|
- **Image Processing**: Direct OCR processing of image files (PNG, JPG, JPEG, BMP, TIFF)
|
||||||
|
- **Multi-language Support**: Supports English, French, German, Spanish, and Italian
|
||||||
|
- **Confidence Scoring**: Returns confidence scores for OCR results
|
||||||
|
- **RESTful API**: Clean, documented API endpoints
|
||||||
|
- **Docker Support**: Easy deployment with Docker and Docker Compose
|
||||||
|
- **File Validation**: Built-in file type and size validation
|
||||||
|
- **Background Processing**: Efficient file cleanup and processing
|
||||||
|
|
||||||
|
## Quick Start
|
||||||
|
|
||||||
|
### Using Docker (Recommended)
|
||||||
|
|
||||||
|
1. **Clone and build the container:**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker-compose up --build
|
||||||
|
```
|
||||||
|
|
||||||
|
2. **Access the API:**
|
||||||
|
- API Documentation: http://localhost:8000/docs
|
||||||
|
- Health Check: http://localhost:8000/health
|
||||||
|
|
||||||
|
### Local Development
|
||||||
|
|
||||||
|
1. **Install system dependencies:**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Ubuntu/Debian
|
||||||
|
sudo apt-get update
|
||||||
|
sudo apt-get install tesseract-ocr tesseract-ocr-eng poppler-utils libpoppler-cpp-dev pkg-config
|
||||||
|
|
||||||
|
# macOS
|
||||||
|
brew install tesseract poppler
|
||||||
|
|
||||||
|
# Arch Linux
|
||||||
|
sudo pacman -S tesseract tesseract-data-eng poppler
|
||||||
|
```
|
||||||
|
|
||||||
|
2. **Install Python dependencies:**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pip install -r requirements.txt
|
||||||
|
```
|
||||||
|
|
||||||
|
3. **Run the application:**
|
||||||
|
```bash
|
||||||
|
python main.py
|
||||||
|
```
|
||||||
|
|
||||||
|
## API Endpoints
|
||||||
|
|
||||||
|
### Health Check
|
||||||
|
|
||||||
|
- **GET** `/health` - Check if the service is running
|
||||||
|
|
||||||
|
### PDF OCR
|
||||||
|
|
||||||
|
- **POST** `/ocr/pdf` - Extract text from PDF files
|
||||||
|
- **File**: PDF file (max 50MB)
|
||||||
|
- **Response**: Extracted text with metadata
|
||||||
|
|
||||||
|
### Image OCR
|
||||||
|
|
||||||
|
- **POST** `/ocr/image` - Extract text from image files
|
||||||
|
- **File**: Image file (PNG, JPG, JPEG, BMP, TIFF)
|
||||||
|
- **Response**: Extracted text with metadata
|
||||||
|
|
||||||
|
## Usage Examples
|
||||||
|
|
||||||
|
### Using curl
|
||||||
|
|
||||||
|
**Process a PDF file:**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -X POST "http://localhost:8000/ocr/pdf" \
|
||||||
|
-H "accept: application/json" \
|
||||||
|
-H "Content-Type: multipart/form-data" \
|
||||||
|
-F "file=@document.pdf"
|
||||||
|
```
|
||||||
|
|
||||||
|
**Process an image file:**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -X POST "http://localhost:8000/ocr/image" \
|
||||||
|
-H "accept: application/json" \
|
||||||
|
-H "Content-Type: multipart/form-data" \
|
||||||
|
-F "file=@image.png"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Using Python
|
||||||
|
|
||||||
|
```python
|
||||||
|
import requests
|
||||||
|
|
||||||
|
# Process PDF
|
||||||
|
with open('document.pdf', 'rb') as f:
|
||||||
|
files = {'file': f}
|
||||||
|
response = requests.post('http://localhost:8000/ocr/pdf', files=files)
|
||||||
|
result = response.json()
|
||||||
|
print(f"Extracted text: {result['text']}")
|
||||||
|
print(f"Processing time: {result['processing_time']}s")
|
||||||
|
print(f"Confidence: {result['confidence']}%")
|
||||||
|
```
|
||||||
|
|
||||||
|
### Using JavaScript/Fetch
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
// Process PDF
|
||||||
|
const formData = new FormData();
|
||||||
|
formData.append("file", fileInput.files[0]);
|
||||||
|
|
||||||
|
fetch("http://localhost:8000/ocr/pdf", {
|
||||||
|
method: "POST",
|
||||||
|
body: formData,
|
||||||
|
})
|
||||||
|
.then((response) => response.json())
|
||||||
|
.then((data) => {
|
||||||
|
console.log("Extracted text:", data.text);
|
||||||
|
console.log("Processing time:", data.processing_time);
|
||||||
|
console.log("Confidence:", data.confidence);
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
## Response Format
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"text": "Extracted text content...",
|
||||||
|
"pages": 3,
|
||||||
|
"processing_time": 2.45,
|
||||||
|
"confidence": 85.2
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Configuration
|
||||||
|
|
||||||
|
### Environment Variables
|
||||||
|
|
||||||
|
- `PYTHONUNBUFFERED=1` - Ensures Python output is not buffered
|
||||||
|
- `PYTHONDONTWRITEBYTECODE=1` - Prevents Python from writing .pyc files
|
||||||
|
|
||||||
|
### Tesseract Configuration
|
||||||
|
|
||||||
|
The Docker image includes multiple language packs:
|
||||||
|
|
||||||
|
- English (eng)
|
||||||
|
- French (fra)
|
||||||
|
- German (deu)
|
||||||
|
- Spanish (spa)
|
||||||
|
- Italian (ita)
|
||||||
|
|
||||||
|
To add more languages, modify the Dockerfile and add additional `tesseract-ocr-*` packages.
|
||||||
|
|
||||||
|
## Performance Tips
|
||||||
|
|
||||||
|
1. **Image Quality**: Higher DPI (300) provides better OCR accuracy but slower processing
|
||||||
|
2. **File Size**: Larger files take longer to process
|
||||||
|
3. **Text Quality**: Clear, high-contrast text yields better results
|
||||||
|
4. **Language**: Specify the correct language for better accuracy
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
### Common Issues
|
||||||
|
|
||||||
|
1. **Tesseract not found**: Ensure Tesseract is installed on your system
|
||||||
|
2. **PDF processing errors**: Check if Poppler utilities are installed
|
||||||
|
3. **Memory issues**: Reduce DPI or process smaller files
|
||||||
|
4. **Poor OCR quality**: Ensure images are clear and have good contrast
|
||||||
|
|
||||||
|
### Logs
|
||||||
|
|
||||||
|
The application logs processing information. Check Docker logs:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker-compose logs ocr-api
|
||||||
|
```
|
||||||
|
|
||||||
|
## Development
|
||||||
|
|
||||||
|
### Project Structure
|
||||||
|
|
||||||
|
```
|
||||||
|
ocr-api/
|
||||||
|
├── main.py # FastAPI application
|
||||||
|
├── requirements.txt # Python dependencies
|
||||||
|
├── Dockerfile # Docker configuration
|
||||||
|
├── docker-compose.yml # Docker Compose configuration
|
||||||
|
└── README.md # This file
|
||||||
|
```
|
||||||
|
|
||||||
|
### Adding Features
|
||||||
|
|
||||||
|
1. **New OCR engines**: Modify the `process_pdf_with_ocr` function
|
||||||
|
2. **Additional file formats**: Add new endpoints and processing logic
|
||||||
|
3. **Custom preprocessing**: Add image preprocessing steps before OCR
|
||||||
|
4. **Batch processing**: Implement queue-based processing for multiple files
|
||||||
|
|
||||||
|
## License
|
||||||
|
|
||||||
|
This project is open source and available under the MIT License.
|
||||||
@@ -0,0 +1,19 @@
|
|||||||
|
version: "3.8"
|
||||||
|
|
||||||
|
services:
|
||||||
|
ocr-api:
|
||||||
|
build: .
|
||||||
|
ports:
|
||||||
|
- "8000:8000"
|
||||||
|
environment:
|
||||||
|
- PYTHONUNBUFFERED=1
|
||||||
|
volumes:
|
||||||
|
# Optional: mount a directory for testing with sample files
|
||||||
|
- ./test_files:/app/test_files:ro
|
||||||
|
restart: unless-stopped
|
||||||
|
healthcheck:
|
||||||
|
test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
|
||||||
|
interval: 30s
|
||||||
|
timeout: 10s
|
||||||
|
retries: 3
|
||||||
|
start_period: 40s
|
||||||
@@ -0,0 +1,225 @@
|
|||||||
|
import os
|
||||||
|
import tempfile
|
||||||
|
import logging
|
||||||
|
from typing import List, Optional
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from fastapi import FastAPI, File, UploadFile, HTTPException, BackgroundTasks
|
||||||
|
from fastapi.responses import JSONResponse
|
||||||
|
from pydantic import BaseModel
|
||||||
|
import pytesseract
|
||||||
|
from PIL import Image
|
||||||
|
from pdf2image import convert_from_path
|
||||||
|
import uvicorn
|
||||||
|
|
||||||
|
# Configure logging
|
||||||
|
logging.basicConfig(level=logging.INFO)
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
app = FastAPI(
|
||||||
|
title="OCR API",
|
||||||
|
description="API for extracting text from PDF files using OCR",
|
||||||
|
version="1.0.0"
|
||||||
|
)
|
||||||
|
|
||||||
|
class OCRResponse(BaseModel):
|
||||||
|
text: str
|
||||||
|
pages: int
|
||||||
|
processing_time: float
|
||||||
|
confidence: Optional[float] = None
|
||||||
|
|
||||||
|
class ErrorResponse(BaseModel):
|
||||||
|
error: str
|
||||||
|
detail: str
|
||||||
|
|
||||||
|
@app.get("/")
|
||||||
|
async def root():
|
||||||
|
return {"message": "OCR API is running. Use /docs for API documentation."}
|
||||||
|
|
||||||
|
@app.get("/health")
|
||||||
|
async def health_check():
|
||||||
|
return {"status": "healthy", "service": "OCR API"}
|
||||||
|
|
||||||
|
async def process_pdf_with_ocr(pdf_path: str) -> OCRResponse:
|
||||||
|
"""
|
||||||
|
Process a PDF file and extract text using OCR
|
||||||
|
"""
|
||||||
|
import time
|
||||||
|
start_time = time.time()
|
||||||
|
|
||||||
|
try:
|
||||||
|
# Convert PDF to images
|
||||||
|
logger.info(f"Converting PDF to images: {pdf_path}")
|
||||||
|
images = convert_from_path(pdf_path, dpi=300)
|
||||||
|
|
||||||
|
if not images:
|
||||||
|
raise HTTPException(status_code=400, detail="Could not extract images from PDF")
|
||||||
|
|
||||||
|
# Extract text from each page
|
||||||
|
all_text = []
|
||||||
|
total_confidence = 0
|
||||||
|
confidence_count = 0
|
||||||
|
|
||||||
|
for i, image in enumerate(images):
|
||||||
|
logger.info(f"Processing page {i+1}/{len(images)}")
|
||||||
|
|
||||||
|
# Perform OCR on the image
|
||||||
|
ocr_data = pytesseract.image_to_data(image, output_type=pytesseract.Output.DICT)
|
||||||
|
|
||||||
|
# Extract text and confidence scores
|
||||||
|
page_text = []
|
||||||
|
for j in range(len(ocr_data['text'])):
|
||||||
|
if int(ocr_data['conf'][j]) > 0: # Filter out low confidence results
|
||||||
|
page_text.append(ocr_data['text'][j])
|
||||||
|
total_confidence += float(ocr_data['conf'][j])
|
||||||
|
confidence_count += 1
|
||||||
|
|
||||||
|
page_text = ' '.join(page_text).strip()
|
||||||
|
if page_text:
|
||||||
|
all_text.append(f"--- Page {i+1} ---\n{page_text}")
|
||||||
|
|
||||||
|
# Calculate average confidence
|
||||||
|
avg_confidence = total_confidence / confidence_count if confidence_count > 0 else None
|
||||||
|
|
||||||
|
processing_time = time.time() - start_time
|
||||||
|
|
||||||
|
return OCRResponse(
|
||||||
|
text='\n\n'.join(all_text),
|
||||||
|
pages=len(images),
|
||||||
|
processing_time=round(processing_time, 2),
|
||||||
|
confidence=round(avg_confidence, 2) if avg_confidence else None
|
||||||
|
)
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error processing PDF: {str(e)}")
|
||||||
|
raise HTTPException(status_code=500, detail=f"Error processing PDF: {str(e)}")
|
||||||
|
|
||||||
|
@app.post("/ocr/pdf", response_model=OCRResponse)
|
||||||
|
async def extract_text_from_pdf(
|
||||||
|
file: UploadFile = File(...),
|
||||||
|
background_tasks: BackgroundTasks = None
|
||||||
|
):
|
||||||
|
"""
|
||||||
|
Extract text from a PDF file using OCR
|
||||||
|
|
||||||
|
- **file**: PDF file to process
|
||||||
|
- Returns extracted text with metadata
|
||||||
|
"""
|
||||||
|
|
||||||
|
# Validate file type
|
||||||
|
if not file.filename.lower().endswith('.pdf'):
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=400,
|
||||||
|
detail="File must be a PDF"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Check file size (limit to 50MB)
|
||||||
|
if file.size and file.size > 50 * 1024 * 1024:
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=400,
|
||||||
|
detail="File size too large. Maximum size is 50MB"
|
||||||
|
)
|
||||||
|
|
||||||
|
try:
|
||||||
|
# Create temporary file
|
||||||
|
with tempfile.NamedTemporaryFile(delete=False, suffix='.pdf') as temp_file:
|
||||||
|
# Write uploaded file to temporary file
|
||||||
|
content = await file.read()
|
||||||
|
temp_file.write(content)
|
||||||
|
temp_file_path = temp_file.name
|
||||||
|
|
||||||
|
try:
|
||||||
|
# Process the PDF
|
||||||
|
result = await process_pdf_with_ocr(temp_file_path)
|
||||||
|
|
||||||
|
# Clean up temporary file
|
||||||
|
background_tasks.add_task(os.unlink, temp_file_path)
|
||||||
|
|
||||||
|
return result
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
# Clean up temporary file on error
|
||||||
|
if os.path.exists(temp_file_path):
|
||||||
|
os.unlink(temp_file_path)
|
||||||
|
raise e
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error handling file upload: {str(e)}")
|
||||||
|
raise HTTPException(status_code=500, detail=f"Error processing file: {str(e)}")
|
||||||
|
|
||||||
|
@app.post("/ocr/image", response_model=OCRResponse)
|
||||||
|
async def extract_text_from_image(
|
||||||
|
file: UploadFile = File(...),
|
||||||
|
background_tasks: BackgroundTasks = None
|
||||||
|
):
|
||||||
|
"""
|
||||||
|
Extract text from an image file using OCR
|
||||||
|
|
||||||
|
- **file**: Image file to process (PNG, JPG, JPEG, etc.)
|
||||||
|
- Returns extracted text with metadata
|
||||||
|
"""
|
||||||
|
|
||||||
|
# Validate file type
|
||||||
|
allowed_extensions = {'.png', '.jpg', '.jpeg', '.bmp', '.tiff', '.tif'}
|
||||||
|
file_extension = Path(file.filename).suffix.lower()
|
||||||
|
|
||||||
|
if file_extension not in allowed_extensions:
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=400,
|
||||||
|
detail=f"Unsupported file type. Allowed types: {', '.join(allowed_extensions)}"
|
||||||
|
)
|
||||||
|
|
||||||
|
try:
|
||||||
|
# Create temporary file
|
||||||
|
with tempfile.NamedTemporaryFile(delete=False, suffix=file_extension) as temp_file:
|
||||||
|
content = await file.read()
|
||||||
|
temp_file.write(content)
|
||||||
|
temp_file_path = temp_file.name
|
||||||
|
|
||||||
|
try:
|
||||||
|
import time
|
||||||
|
start_time = time.time()
|
||||||
|
|
||||||
|
# Open and process image
|
||||||
|
image = Image.open(temp_file_path)
|
||||||
|
|
||||||
|
# Perform OCR
|
||||||
|
ocr_data = pytesseract.image_to_data(image, output_type=pytesseract.Output.DICT)
|
||||||
|
|
||||||
|
# Extract text and confidence
|
||||||
|
text_parts = []
|
||||||
|
total_confidence = 0
|
||||||
|
confidence_count = 0
|
||||||
|
|
||||||
|
for i in range(len(ocr_data['text'])):
|
||||||
|
if int(ocr_data['conf'][i]) > 0:
|
||||||
|
text_parts.append(ocr_data['text'][i])
|
||||||
|
total_confidence += float(ocr_data['conf'][i])
|
||||||
|
confidence_count += 1
|
||||||
|
|
||||||
|
text = ' '.join(text_parts).strip()
|
||||||
|
avg_confidence = total_confidence / confidence_count if confidence_count > 0 else None
|
||||||
|
processing_time = time.time() - start_time
|
||||||
|
|
||||||
|
# Clean up temporary file
|
||||||
|
background_tasks.add_task(os.unlink, temp_file_path)
|
||||||
|
|
||||||
|
return OCRResponse(
|
||||||
|
text=text,
|
||||||
|
pages=1,
|
||||||
|
processing_time=round(processing_time, 2),
|
||||||
|
confidence=round(avg_confidence, 2) if avg_confidence else None
|
||||||
|
)
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
# Clean up temporary file on error
|
||||||
|
if os.path.exists(temp_file_path):
|
||||||
|
os.unlink(temp_file_path)
|
||||||
|
raise e
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error processing image: {str(e)}")
|
||||||
|
raise HTTPException(status_code=500, detail=f"Error processing image: {str(e)}")
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
uvicorn.run(app, host="0.0.0.0", port=8000)
|
||||||
@@ -0,0 +1,8 @@
|
|||||||
|
fastapi==0.104.1
|
||||||
|
uvicorn[standard]==0.24.0
|
||||||
|
python-multipart==0.0.6
|
||||||
|
Pillow==10.1.0
|
||||||
|
pytesseract==0.3.10
|
||||||
|
pdf2image==1.16.3
|
||||||
|
pydantic==2.5.0
|
||||||
|
python-dotenv==1.0.0
|
||||||
@@ -0,0 +1,109 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
Test client for the OCR API
|
||||||
|
This script demonstrates how to use the OCR API endpoints
|
||||||
|
"""
|
||||||
|
|
||||||
|
import requests
|
||||||
|
import json
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
# API base URL
|
||||||
|
BASE_URL = "http://localhost:8000"
|
||||||
|
|
||||||
|
def test_health():
|
||||||
|
"""Test the health endpoint"""
|
||||||
|
try:
|
||||||
|
response = requests.get(f"{BASE_URL}/health")
|
||||||
|
print(f"Health check: {response.status_code} - {response.json()}")
|
||||||
|
return response.status_code == 200
|
||||||
|
except requests.exceptions.ConnectionError:
|
||||||
|
print("❌ Could not connect to the API. Make sure it's running on localhost:8000")
|
||||||
|
return False
|
||||||
|
|
||||||
|
def test_pdf_ocr(file_path):
|
||||||
|
"""Test PDF OCR endpoint"""
|
||||||
|
if not Path(file_path).exists():
|
||||||
|
print(f"❌ File not found: {file_path}")
|
||||||
|
return False
|
||||||
|
|
||||||
|
try:
|
||||||
|
with open(file_path, 'rb') as f:
|
||||||
|
files = {'file': f}
|
||||||
|
print(f"📄 Processing PDF: {file_path}")
|
||||||
|
response = requests.post(f"{BASE_URL}/ocr/pdf", files=files)
|
||||||
|
|
||||||
|
if response.status_code == 200:
|
||||||
|
result = response.json()
|
||||||
|
print(f"✅ PDF OCR successful!")
|
||||||
|
print(f" Pages: {result['pages']}")
|
||||||
|
print(f" Processing time: {result['processing_time']}s")
|
||||||
|
print(f" Confidence: {result['confidence']}%")
|
||||||
|
print(f" Text preview: {result['text'][:200]}...")
|
||||||
|
return True
|
||||||
|
else:
|
||||||
|
print(f"❌ PDF OCR failed: {response.status_code} - {response.text}")
|
||||||
|
return False
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
print(f"❌ Error processing PDF: {str(e)}")
|
||||||
|
return False
|
||||||
|
|
||||||
|
def test_image_ocr(file_path):
|
||||||
|
"""Test image OCR endpoint"""
|
||||||
|
if not Path(file_path).exists():
|
||||||
|
print(f"❌ File not found: {file_path}")
|
||||||
|
return False
|
||||||
|
|
||||||
|
try:
|
||||||
|
with open(file_path, 'rb') as f:
|
||||||
|
files = {'file': f}
|
||||||
|
print(f"🖼️ Processing image: {file_path}")
|
||||||
|
response = requests.post(f"{BASE_URL}/ocr/image", files=files)
|
||||||
|
|
||||||
|
if response.status_code == 200:
|
||||||
|
result = response.json()
|
||||||
|
print(f"✅ Image OCR successful!")
|
||||||
|
print(f" Processing time: {result['processing_time']}s")
|
||||||
|
print(f" Confidence: {result['confidence']}%")
|
||||||
|
print(f" Text: {result['text']}")
|
||||||
|
return True
|
||||||
|
else:
|
||||||
|
print(f"❌ Image OCR failed: {response.status_code} - {response.text}")
|
||||||
|
return False
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
print(f"❌ Error processing image: {str(e)}")
|
||||||
|
return False
|
||||||
|
|
||||||
|
def main():
|
||||||
|
"""Main test function"""
|
||||||
|
print("🧪 OCR API Test Client")
|
||||||
|
print("=" * 50)
|
||||||
|
|
||||||
|
# Test health endpoint
|
||||||
|
if not test_health():
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
print("\n" + "=" * 50)
|
||||||
|
|
||||||
|
# Test PDF OCR if file provided
|
||||||
|
if len(sys.argv) > 1:
|
||||||
|
file_path = sys.argv[1]
|
||||||
|
file_ext = Path(file_path).suffix.lower()
|
||||||
|
|
||||||
|
if file_ext == '.pdf':
|
||||||
|
test_pdf_ocr(file_path)
|
||||||
|
elif file_ext in ['.png', '.jpg', '.jpeg', '.bmp', '.tiff', '.tif']:
|
||||||
|
test_image_ocr(file_path)
|
||||||
|
else:
|
||||||
|
print(f"❌ Unsupported file type: {file_ext}")
|
||||||
|
print("Supported types: .pdf, .png, .jpg, .jpeg, .bmp, .tiff, .tif")
|
||||||
|
else:
|
||||||
|
print("📝 Usage: python test_client.py <file_path>")
|
||||||
|
print(" Example: python test_client.py document.pdf")
|
||||||
|
print(" Example: python test_client.py image.png")
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,96 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
Test script to verify PostgreSQL database creation and setup
|
||||||
|
"""
|
||||||
|
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
|
||||||
|
# Add the app directory to the Python path
|
||||||
|
sys.path.insert(0, os.path.join(os.path.dirname(__file__), 'app'))
|
||||||
|
|
||||||
|
def test_postgres_setup():
|
||||||
|
"""Test PostgreSQL database creation and setup"""
|
||||||
|
print("=== Testing PostgreSQL Database Setup ===")
|
||||||
|
|
||||||
|
# Check if PostgreSQL credentials are available
|
||||||
|
required_vars = ['POSTGRES_HOST', 'POSTGRES_DB', 'POSTGRES_USER', 'POSTGRES_PASSWORD']
|
||||||
|
missing_vars = [var for var in required_vars if not os.getenv(var)]
|
||||||
|
|
||||||
|
if missing_vars:
|
||||||
|
print(f"⚠ PostgreSQL test skipped - missing environment variables: {', '.join(missing_vars)}")
|
||||||
|
print("Set these variables to test PostgreSQL:")
|
||||||
|
print(" POSTGRES_HOST=your-host")
|
||||||
|
print(" POSTGRES_DB=your-database")
|
||||||
|
print(" POSTGRES_USER=your-username")
|
||||||
|
print(" POSTGRES_PASSWORD=your-password")
|
||||||
|
return False
|
||||||
|
|
||||||
|
try:
|
||||||
|
from app.database import db_manager, init_db
|
||||||
|
|
||||||
|
print(f"Database type: {db_manager.db_type}")
|
||||||
|
print(f"PostgreSQL host: {os.getenv('POSTGRES_HOST')}")
|
||||||
|
print(f"PostgreSQL database: {os.getenv('POSTGRES_DB')}")
|
||||||
|
print(f"PostgreSQL user: {os.getenv('POSTGRES_USER')}")
|
||||||
|
|
||||||
|
# Initialize database (this will create the database if it doesn't exist)
|
||||||
|
init_db()
|
||||||
|
print("✓ Database initialized successfully")
|
||||||
|
|
||||||
|
# Test a simple cache operation
|
||||||
|
from app.database import save_to_cache, get_cached_data
|
||||||
|
|
||||||
|
test_url = "https://example.com"
|
||||||
|
test_route = "test"
|
||||||
|
test_data = {"title": "Test Page", "content": "Test content"}
|
||||||
|
|
||||||
|
# Save to cache
|
||||||
|
save_to_cache(test_url, test_route, test_data)
|
||||||
|
print("✓ Data saved to cache")
|
||||||
|
|
||||||
|
# Retrieve from cache
|
||||||
|
cached_data = get_cached_data(test_url, test_route)
|
||||||
|
if cached_data and cached_data == test_data:
|
||||||
|
print("✓ Data retrieved from cache successfully")
|
||||||
|
else:
|
||||||
|
print("✗ Failed to retrieve data from cache")
|
||||||
|
return False
|
||||||
|
|
||||||
|
# Test cache stats
|
||||||
|
from app.services.cache import get_cache_stats
|
||||||
|
stats = get_cache_stats()
|
||||||
|
if stats['status'] == 'success':
|
||||||
|
print("✓ Cache stats retrieved successfully")
|
||||||
|
print(f" Database type: {stats['stats']['database_type']}")
|
||||||
|
print(f" Total entries: {stats['stats']['total_entries']}")
|
||||||
|
else:
|
||||||
|
print("✗ Failed to get cache stats")
|
||||||
|
return False
|
||||||
|
|
||||||
|
print("✓ PostgreSQL database setup test passed!")
|
||||||
|
return True
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
print(f"✗ PostgreSQL setup test failed: {e}")
|
||||||
|
import traceback
|
||||||
|
traceback.print_exc()
|
||||||
|
return False
|
||||||
|
|
||||||
|
def main():
|
||||||
|
"""Run the PostgreSQL setup test"""
|
||||||
|
print("PostgreSQL Database Setup Test")
|
||||||
|
print("=" * 50)
|
||||||
|
|
||||||
|
success = test_postgres_setup()
|
||||||
|
|
||||||
|
print("\n" + "=" * 50)
|
||||||
|
if success:
|
||||||
|
print("🎉 PostgreSQL setup test passed! Database creation and connection working correctly.")
|
||||||
|
return 0
|
||||||
|
else:
|
||||||
|
print("❌ PostgreSQL setup test failed. Please check the configuration and PostgreSQL server.")
|
||||||
|
return 1
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main())
|
||||||
@@ -167,7 +167,7 @@ The container logs all health check activities to both stdout and a log file (`/
|
|||||||
Each log entry includes the host name for easy identification:
|
Each log entry includes the host name for easy identification:
|
||||||
|
|
||||||
```
|
```
|
||||||
2024-01-15 10:30:00 - INFO - Performing health check for server1: http://server1:8000/?url=https%3A//www.google.com&skipCache=true
|
2024-01-15 10:30:00 - INFO - Performing health check for server1: http://server1:8000/?url=https%3A//www.google.com
|
||||||
2024-01-15 10:30:01 - INFO - Health check passed for server1 - API is responding correctly
|
2024-01-15 10:30:01 - INFO - Health check passed for server1 - API is responding correctly
|
||||||
2024-01-15 10:30:02 - WARNING - Health check failed for server2 - Status code: 500
|
2024-01-15 10:30:02 - WARNING - Health check failed for server2 - Status code: 500
|
||||||
```
|
```
|
||||||
@@ -181,6 +181,38 @@ Each log entry includes the host name for easy identification:
|
|||||||
|
|
||||||
## Troubleshooting
|
## Troubleshooting
|
||||||
|
|
||||||
|
### USB Device Mounting Error
|
||||||
|
|
||||||
|
If you encounter an error like:
|
||||||
|
|
||||||
|
```
|
||||||
|
error creating device nodes: mount src=/dev/bus/usb/003/003, dst=/var/lib/docker/overlay2/.../merged/dev/bus/usb/003/003: no such file or directory
|
||||||
|
```
|
||||||
|
|
||||||
|
**Solution**: Remove `privileged: true` from your Docker Compose configuration. The healthcheck container doesn't need privileged access.
|
||||||
|
|
||||||
|
**Correct configuration:**
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
puppeteer-healthcheck:
|
||||||
|
image: bramkel/puppeteer-healthcheck:latest
|
||||||
|
container_name: puppeteer-healthcheck
|
||||||
|
# Remove this line: privileged: true
|
||||||
|
volumes:
|
||||||
|
- /var/run/docker.sock:/var/run/docker.sock:ro
|
||||||
|
environment:
|
||||||
|
- HOSTS=puppeteer-api-1,puppeteer-api-2
|
||||||
|
- TEST_URL=https://www.google.com
|
||||||
|
- MAX_CONSECUTIVE_FAILURES=2
|
||||||
|
- API_KEY=${PUPPETEER_API_KEY}
|
||||||
|
- CHECK_INTERVAL=60
|
||||||
|
- TIMEOUT=15
|
||||||
|
restart: unless-stopped
|
||||||
|
depends_on:
|
||||||
|
- puppeteer-api-1
|
||||||
|
- puppeteer-api-2
|
||||||
|
```
|
||||||
|
|
||||||
### Docker Permission Issues
|
### Docker Permission Issues
|
||||||
|
|
||||||
If you see "Permission denied" errors when accessing the Docker socket:
|
If you see "Permission denied" errors when accessing the Docker socket:
|
||||||
@@ -246,7 +278,7 @@ docker build -t puppeteer-healthcheck .
|
|||||||
|
|
||||||
## Version
|
## Version
|
||||||
|
|
||||||
Current version: 2.0.0
|
Current version: latest
|
||||||
|
|
||||||
### Migration from v1.x
|
### Migration from v1.x
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user