Adaptive video pipeline

LookSuite improves a live camera feed on the user's own machine, frame by frame: lighting and colour, overlays, masks and more. To look smooth in a meeting, every frame has to get through the whole pipeline in about 16 ms, the budget of 60 frames a second.

We started with a CPU pipeline. At that frame budget it could not carry the heavier work, such as masking, even on high-end machines. We stretched it with WebAssembly packages and made it work, but it still left the GPU unused on every machine that had one.

I added a GPU path next to the CPU one and let each machine use the one it can sustain. I began with WebGL and later moved to WebGPU: it has less overhead when it is called from JavaScript and it supports multi-threading, which suits a pipeline that does its heavy per-frame work in background workers. WebGL stays in place as a fallback.

Choosing the path is the tricky part. On Windows it is hard to tell what a machine has, because GPUs, drivers and versions vary so much. So the application does not depend on a library for it. It starts with capability detection and a safe default. Then, while the application is idle, it runs a short, sequential benchmark that tests the GPU against an evaluation step, and it saves the result. The benchmark matters because a GPU being present does not mean it is faster: integrated graphics can lose to the CPU at a given resolution. If the chosen pipeline fails or performs badly, the application falls back to the other one. The user can also pick the pipeline in the application settings, so the automatic choice is never a trap.

• Staying on WebGL only: it worked, but it costs more per call from JavaScript and is not built for multi-threaded work, which is what the worker-based pipeline needs.
• CPU only: it works everywhere, but even with WebAssembly it could not hold the budget on complex frames and left the GPU idle.
• GPU only: it would exclude machines without a capable GPU, and a present GPU is not always the faster option.

Architected a dual video-processing architecture supporting both CPU- and GPU-based pipelines.

Implemented automatic capability detection and runtime evaluation to determine the most appropriate processing pipeline based on factors such as device capabilities, GPU support, and performance benchmarks.

This allowed the application to dynamically select an appropriate processing strategy across different hardware configurations while maintaining real-time video performance.

• The paths must stay identical. CPU and GPU can produce slightly different colors, so values have to be handled with great care. Every new feature or component to render has to be built on each path and matched until they look the same.
• A benchmark takes a little time and a saved result can age, which is why the path can be changed and the user can override it.
• WebGPU is newer than WebGL, so the GPU path rests on a younger platform, and WebGL stays available as a fallback.

The application now uses an idle GPU where it helps, falls back to the CPU where it does not, and lets the user override the choice, all while aiming at 16 ms per frame.