Webhook Callbacks: Asynchronous Service Communication

Introduction

Building scalable, user-friendly applications often means splitting responsibilities across multiple services. Whether you're working with AI-powered features, file processing, or any long-running tasks, you'll eventually face this challenge: How do I keep my main backend in sync with long-running jobs, without constant polling or manual refreshes?

The answer: webhook callbacks. Here's how to design and implement callback system for asynchronous service communication.

The Problem

• Long-running tasks are slow: Processing jobs can take minutes, hours, or even days to complete.
• Users want updates: Your main backend needs to know when a job is ready, so it can update the user interface and database.
• Polling is wasteful: Constantly asking the service "is it done yet?" is inefficient and can overload both services.

The Solution: Webhook Callbacks

Instead of polling, let the processing service notify your main backend as soon as a job is done (or fails). Here's the high-level flow:

1. Main backend submits a job to the processing service, including a callback_url in the request.
2. Processing service handles the job asynchronously.
3. When the job completes (or fails), the processing service sends a POST request to the callback_url with the result.
4. Main backend receives the callback and updates its database and user interface.

Implementation Details

1. Accepting a Callback URL

When the main backend submits a job, it includes a callback_url:

data = {
    'task_parameters': task_data,
    'callback_url': 'https://main-backend.com/api/job-callback'
}
response = requests.post(
    f"{PROCESSING_SERVICE_URL}/api/jobs/create",
    files=files,
    data=data
)

2. Storing the Callback URL

The processing service saves the callback_url as part of the job parameters, so it's available when the job finishes.

3. Sending the Callback

When the job is ready (or if it fails), the processing service sends a POST request to the callback URL:

On Success:

POST https://main-backend.com/api/job-callback
{
  "job_id": "123",
  "result_url": "https://storage.example.com/result.pdf",
  "timestamp": "2024-01-15T10:30:00",
  "status": "completed"
}

On Failure:

POST https://main-backend.com/api/job-callback
{
  "job_id": "123",
  "error": "Processing failed: Invalid input",
  "timestamp": "2024-01-15T10:30:00",
  "status": "failed"
}

This is handled by a simple utility in the processing service:

def notify_job_completion(self, job_id, result_url, callback_url, parameters=None):
    payload = {
        "job_id": job_id,
        "result_url": result_url,
        "timestamp": datetime.now().isoformat(),
        "status": "completed"
    }
    requests.post(callback_url, json=payload, timeout=10)

4. Handling the Callback in the Main Backend

The main backend exposes an endpoint to receive the callback:

@app.route("/api/job-callback", methods=["POST"])
def job_completion_callback():
    callback_data = request.get_json()
    job_id = callback_data.get("job_id")
    status = callback_data.get("status")
    if status == "completed":
        result_url = callback_data.get("result_url")
        # Update your job record
        job = Job.query.filter_by(service_job_id=job_id).first()
        if job:
            job.status = "completed"
            job.result_url = result_url
            db.session.commit()
    return jsonify({"status": "success"})

Why This Approach Rocks

• No polling: The main backend is instantly notified when a job is done.
• Scalable: Works for any number of jobs/users.
• Simple: The payload is minimal—just what's needed.
• Extensible: You can add more fields to the callback as needed.

Best Practices

• Keep callbacks simple: Only send what the receiver needs.
• Handle failures: Always send a callback, even if the job fails.
• Test thoroughly: Use test scripts to simulate callbacks and ensure your backend handles them gracefully.
• Use HTTPS: Always use secure connections for production callbacks.
• Implement retry logic: If a callback fails, retry with exponential backoff.
• Validate callbacks: Verify the source and integrity of incoming callbacks.

Common Use Cases

• File processing: Document conversion, image resizing, video encoding
• AI/ML tasks: Model training, inference, data analysis
• Email campaigns: Bulk email sending with delivery status
• Payment processing: Payment confirmation and failure notifications
• Data synchronization: Database updates, cache invalidation

Conclusion

Webhook callbacks are a powerful pattern for connecting asynchronous services. With just a few lines of code, you can make your backend more responsive, efficient, and user-friendly.

If you're building applications with long-running jobs, or any system that needs real-time updates, consider using callbacks instead of polling. Your users (and your servers) will thank you!