Building a Universal Container Dispatcher in Python

Introduction

When automating heterogeneous tasks, a common pattern emerges where a Python worker needs to launch different containers depending on the task type. Instead of hardcoding each container, we create a universal dispatcher driven by YAML configuration.

YAML Configuration with Runners Section

# config.yaml
runners:
jira-processor:
image: localhost/jira-proc:latest
command: [“python”, “process.py”]
podman_args: [“–pod”, “main-stack”]
timeout: 300
asterisk-handler:
image: localhost/asterisk-ari:latest
command: [“python”, “handle.py”]
timeout: 60

EphemeralContainer Context Manager

The context manager handles the complete container lifecycle — from creation to removal. It generates a unique name for each run, passes data via stdin, and ensures cleanup in the finally block.

class EphemeralContainer:
def init(self, runner_config):
self.config = runner_config
self.name = f"task-{uuid.uuid4().hex[:8]}"

def run(self, input_data):
cmd = [“podman”, “run”, “–rm”, “-i”,
“–name”, self.name]
cmd.extend(self.config.get(‘podman_args’, []))
cmd.append(self.config[‘image’])
cmd.extend(self.config.get(‘command’, []))
result = subprocess.run(cmd,
input=json.dumps(input_data).encode(),
capture_output=True,
timeout=self.config.get(‘timeout’, 300))
return json.loads(result.stdout)

Passing JSON via stdin

Task data is serialized to JSON and passed via the container’s stdin. The container reads stdin, processes the data, and writes the result to stdout. This approach ensures complete isolation and universality — the container doesn’t depend on a specific transport mechanism.

Capturing Results from stdout

The processing result is parsed from the container’s stdout. Stderr is used for logs and debugging. On non-zero return codes, an exception is thrown with stderr contents.

Error Handling with finally Blocks

The finally block in the context manager guarantees container removal even during exceptions. This prevents resource leaks — stuck containers won’t accumulate after failures.

Integration with RQ

The universal RQ task is trivial — it accepts a runner name and data, loads the config, and launches the corresponding container. This allows adding new task types without changing worker code.

Conclusion

A universal container dispatcher is an elegant solution for systems with heterogeneous tasks. YAML configuration, context managers, and stdin/stdout data passing provide flexibility, reliability, and complete execution isolation.