Client-Side Video Processing with WebCodecs: A Practical Tutorial
Client-Side Video Processing with WebCodecs: A Practical Tutorial
Video processing used to be something that only happened on a server. If you wanted to resize a video, trim a clip, or apply a filter, you had to upload the file to a backend, wait for it to process, and then download the result. That approach works, but it comes with problems. Uploads take time. Servers cost money. And if you are building a web app, you are adding a lot of complexity just to handle basic video tasks.
WebCodecs changes that. It is a browser API that lets you decode, process, and encode video directly on the user's device. No server required. No uploads. No waiting. The video never leaves the browser, which is better for privacy and much faster for the user.
In this tutorial, I will show you how to use WebCodecs to build a simple client-side video processing tool. We will load a video file, decode it, apply a filter to each frame, encode it back into a new video, and let the user download the result. I will explain every step in plain English, and I will provide code examples you can copy and adapt.
By the end, you will have a working understanding of how WebCodecs works and how to use it in your own projects.
What Is WebCodecs?
WebCodecs is a JavaScript API that gives you low-level access to the browser's built-in video and audio codecs. Before WebCodecs, the only way to work with video in the browser was through HTML elements like <video> or through WebAssembly libraries like FFmpeg.wasm. Those approaches work, but they are either too limited or too slow for many use cases.
WebCodecs gives you direct access to the codec. You can decode compressed video into raw frames, manipulate those frames, and then encode them back into a compressed format. This is the same kind of access that native applications have, but it runs entirely in the browser.
The API is made up of a few key interfaces:
- VideoDecoder: Takes compressed video chunks and produces raw video frames.
- VideoEncoder: Takes raw video frames and produces compressed video chunks.
- VideoFrame: Represents a single raw frame of video. You can read pixels from it, draw it to a canvas, or create a new frame from a canvas.
- EncodedVideoChunk: Represents a compressed chunk of video data.
- VideoColorSpace: Describes the color space of a video frame.
There are similar interfaces for audio, but we will focus on video for this tutorial.
Why Use WebCodecs?
There are several reasons why you might want to use WebCodecs instead of a server-side solution.
Speed. Processing video on the client is fast. You do not have to wait for an upload or a download. The user sees the result almost immediately.
Privacy. The video never leaves the user's device. This is important for apps that handle sensitive content, like medical records or personal videos.
Cost. You do not need to pay for server time or bandwidth. This is especially important if you have a lot of users or if you are processing large files.
Offline capability. Once the page is loaded, the processing can happen entirely offline. This is great for progressive web apps.
Flexibility. You have full control over the video pipeline. You can apply custom filters, change the resolution, adjust the frame rate, and more.
Of course, there are also limitations. WebCodecs is still relatively new, and browser support is not universal. It works well in Chrome, Edge, and other Chromium-based browsers. Safari has partial support. Firefox is still working on it. You should check the current support before relying on it for a production app.
Another limitation is that WebCodecs does not handle container formats. It works with raw codec data, not with MP4 or WebM files. You will need a separate library to demux the container and extract the encoded video chunks. For this tutorial, we will use a library called mp4box.js to handle the MP4 container.
Prerequisites
Before we start, make sure you have a few things ready.
You need a modern browser that supports WebCodecs. Chrome or Edge is recommended.
You need a basic understanding of JavaScript. You do not need to be an expert, but you should be comfortable with functions, promises, and the DOM.
You need a video file to test with. An MP4 file with H.264 video is a good choice because it is widely supported.
You need a code editor and a way to serve your HTML file. You can use a simple local server like http-server or just open the file directly if your browser allows it. Some browsers restrict WebCodecs when loading from file://, so a local server is safer.
Step 1: Set Up the HTML Page
Let us start with a basic HTML page. Create a new file called index.html and add the following code.
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>WebCodecs Video Processor</title>
<style>
body {
font-family: Arial, sans-serif;
max-width: 800px;
margin: 0 auto;
padding: 20px;
}
video, canvas {
max-width: 100%;
margin-top: 20px;
}
button {
padding: 10px 20px;
font-size: 16px;
margin-top: 10px;
cursor: pointer;
}
</style>
</head>
<body>
<h1>WebCodecs Video Processor</h1>
<p>Select a video file to process.</p>
<input type="file" id="fileInput" accept="video/mp4">
<br>
<button id="processButton" disabled>Process Video</button>
<video id="preview" controls></video>
<canvas id="canvas"></canvas>
<a id="downloadLink" style="display: none;">Download Processed Video</a>
<script src="https://cdn.jsdelivr.net/npm/[email protected]/dist/mp4box.all.min.js"></script>
<script src="app.js"></script>
</body>
</html>
This page has a file input, a button to start processing, a video element for preview, a canvas for displaying frames, and a download link for the output.
We are loading mp4box.js from a CDN. This library will help us parse the MP4 file and extract the encoded video chunks.
Now create a file called app.js in the same folder. This is where we will write our JavaScript code.
Step 2: Load the Video File
The first thing we need to do is let the user select a file and read it into memory. We will use the FileReader API to read the file as an ArrayBuffer.
Add the following code to app.js.
const fileInput = document.getElementById('fileInput');
const processButton = document.getElementById('processButton');
const preview = document.getElementById('preview');
const canvas = document.getElementById('canvas');
const downloadLink = document.getElementById('downloadLink');
let fileBuffer = null;
let fileName = '';
fileInput.addEventListener('change', async (event) => {
const file = event.target.files[0];
if (!file) return;
fileName = file.name;
fileBuffer = await file.arrayBuffer();
preview.src = URL.createObjectURL(file);
processButton.disabled = false;
});
This code listens for a file selection, reads the file into an ArrayBuffer, and sets the video preview to the selected file. The process button becomes enabled.
Step 3: Parse the MP4 File
Now we need to parse the MP4 file to extract the video track and the encoded chunks. We will use mp4box.js for this.
Add the following function to app.js.
function parseMP4(buffer) {
return new Promise((resolve, reject) => {
const mp4boxfile = MP4Box.createFile();
const tracks = [];
const chunks = [];
mp4boxfile.onError = (e) => reject(e);
mp4boxfile.onReady = (info) => {
const videoTrack = info.videoTracks[0];
if (!videoTrack) {
reject(new Error('No video track found'));
return;
}
tracks.push(videoTrack);
mp4boxfile.setExtractionOptions(videoTrack.id, null, { nbSamples: 1000 });
mp4boxfile.start();
};
mp4boxfile.onSamples = (id, user, samples) => {
samples.forEach((sample) => {
chunks.push({
data: sample.data,
timestamp: sample.cts,
duration: sample.duration,
isKey: sample.is_sync,
});
});
};
mp4boxfile.onReady = (info) => {
const videoTrack = info.videoTracks[0];
if (!videoTrack) {
reject(new Error('No video track found'));
return;
}
tracks.push(videoTrack);
mp4boxfile.setExtractionOptions(videoTrack.id, null, { nbSamples: 1000 });
mp4boxfile.start();
};
// Append the buffer and flush
buffer.fileStart = 0;
mp4boxfile.appendBuffer(buffer);
mp4boxfile.flush();
// Wait a moment for samples to be extracted
setTimeout(() => {
resolve({
track: tracks[0],
chunks: chunks,
});
}, 1000);
});
}
This function creates an MP4Box file object, sets up callbacks for when the file is ready and when samples are extracted, and then appends the buffer. The onReady callback gives us information about the video track, including its codec, width, height, and frame rate. The onSamples callback gives us the actual encoded chunks.
We use a setTimeout to wait for the samples to be extracted. In a real app, you would want a more robust way to know when extraction is complete, but this works for a simple tutorial.
Step 4: Configure the Decoder
Now that we have the encoded chunks and the track information, we can set up the VideoDecoder.
Add the following code to app.js.
async function decodeVideo(track, chunks) {
const decoderConfig = {
codec: track.codec,
codedWidth: track.video.width,
codedHeight: track.video.height,
description: track.codec === 'avc1' ? getAvcDescription(track) : undefined,
};
const decoder = new VideoDecoder({
output: (frame) => {
// We will handle frames later
frame.close();
},
error: (e) => console.error('Decoder error:', e),
});
decoder.configure(decoderConfig);
for (const chunk of chunks) {
const encodedChunk = new EncodedVideoChunk({
type: chunk.isKey ? 'key' : 'delta',
timestamp: chunk.timestamp,
duration: chunk.duration,
data: chunk.data,
});
decoder.decode(encodedChunk);
}
await decoder.flush();
decoder.close();
}
This function creates a VideoDecoder with an output callback and an error callback. It configures the decoder with the codec information from the track. Then it loops through the chunks, creates EncodedVideoChunk objects, and feeds them to the decoder. Finally, it flushes the decoder to ensure all frames are processed.
The description field is needed for H.264 (AVC) codec. It contains the SPS and PPS data, which are required for decoding. We need a helper function to extract this from the MP4 track. The mp4box.js library provides this information in the track.avcC property. We can convert it to a Uint8Array.
Add this helper function to app.js.
function getAvcDescription(track) {
if (!track.avcC) return undefined;
const avcC = track.avcC;
const description = new Uint8Array(avcC.length);
for (let i = 0; i < avcC.length; i++) {
description[i] = avcC[i];
}
return description;
}
Now we have a working decoder. But we are not doing anything with the frames yet. Let us change that.
Step 5: Process Frames
We want to apply a filter to each frame. For this tutorial, let us apply a simple grayscale effect. We will draw each frame to a canvas, apply the filter, and then create a new VideoFrame from the canvas.
Modify the decodeVideo function to accept a processing callback.
async function decodeVideo(track, chunks, onFrame) {
const decoderConfig = {
codec: track.codec,
codedWidth: track.video.width,
codedHeight: track.video.height,
description: track.codec === 'avc1' ? getAvcDescription(track) : undefined,
};
const decoder = new VideoDecoder({
output: (frame) => {
onFrame(frame);
frame.close();
},
error: (e) => console.error('Decoder error:', e),
});
decoder.configure(decoderConfig);
for (const chunk of chunks) {
const encodedChunk = new EncodedVideoChunk({
type: chunk.isKey ? 'key' : 'delta',
timestamp: chunk.timestamp,
duration: chunk.duration,
data: chunk.data,
});
decoder.decode(encodedChunk);
}
await decoder.flush();
decoder.close();
}
Now we can pass a function that will be called for each frame. Let us write that function.
const processedFrames = [];
function processFrame(frame) {
// Set canvas size to match the frame
canvas.width = frame.displayWidth;
canvas.height = frame.displayHeight;
const ctx = canvas.getContext('2d');
ctx.drawImage(frame, 0, 0);
// Get the image data
const imageData = ctx.getImageData(0, 0, canvas.width, canvas.height);
const data = imageData.data;
// Apply grayscale filter
for (let i = 0; i < data.length; i += 4) {
const gray = 0.299 * data[i] + 0.587 * data[i + 1] + 0.114 * data[i + 2];
data[i] = gray;
data[i + 1] = gray;
data[i + 2] = gray;
}
// Put the modified data back
ctx.putImageData(imageData, 0, 0);
// Create a new VideoFrame from the canvas
const newFrame = new VideoFrame(canvas, {
timestamp: frame.timestamp,
duration: frame.duration,
});
processedFrames.push(newFrame);
}
This function draws the frame to the canvas, gets the pixel data, converts each pixel to grayscale, and then creates a new VideoFrame from the canvas. It stores the new frames in an array.
We need to call decodeVideo with this callback. Let us update the process button handler.
processButton.addEventListener('click', async () => {
if (!fileBuffer) return;
processButton.disabled = true;
processButton.textContent = 'Processing...';
try {
const { track, chunks } = await parseMP4(fileBuffer);
await decodeVideo(track, chunks, processFrame);
// Now we need to encode the frames
// We will do that in the next step
} catch (error) {
console.error('Processing error:', error);
alert('An error occurred while processing the video.');
} finally {
processButton.disabled = false;
processButton.textContent = 'Process Video';
}
});
At this point, we have decoded the video and processed each frame. The processedFrames array contains all the grayscale frames. Now we need to encode them back into a video.
Step 6: Encode the Frames
Encoding is the reverse of decoding. We will create a VideoEncoder, feed it the processed frames, and collect the encoded chunks. Then we will mux those chunks into a new MP4 file.
First, let us set up the encoder.
async function encodeVideo(frames, width, height, frameRate, codec) {
const encodedChunks = [];
const encoder = new VideoEncoder({
output: (chunk, metadata) => {
encodedChunks.push({ chunk, metadata });
},
error: (e) => console.error('Encoder error:', e),
});
const encoderConfig = {
codec: codec,
width: width,
height: height,
bitrate: 2000000, // 2 Mbps
framerate: frameRate,
};
encoder.configure(encoderConfig);
for (const frame of frames) {
encoder.encode(frame, { keyFrame: false });
frame.close();
}
await encoder.flush();
encoder.close();
return encodedChunks;
}
This function creates a VideoEncoder, configures it with the same codec, width, height, and frame rate as the original video, and then encodes each frame. The output callback collects the encoded chunks.
We need to know the codec, width, height, and frame rate from the original track. We can pass those from the parseMP4 result. Let us update the process button handler to call encodeVideo and then mux the result.
Step 7: Mux the Encoded Chunks into an MP4
Muxing is the process of combining encoded video chunks into a container format like MP4. This is the most complex part of the tutorial, but mp4box.js can help.
We will create a new MP4 file using mp4box.js. We need to create a new track, add the encoded samples, and then generate the file.
function createMP4(encodedChunks, track, width, height, frameRate) {
const mp4boxfile = MP4Box.createFile();
const trackOptions = {
timescale: 1000,
width: width,
height: height,
channel_count: 0,
samples_duration: Math.round(1000 / frameRate),
type: 'video',
codec: track.codec,
};
const newTrack = mp4boxfile.addTrack(trackOptions);
let sampleNumber = 1;
for (const { chunk } of encodedChunks) {
const buffer = new Uint8Array(chunk.byteLength);
chunk.copyTo(buffer);
mp4boxfile.addSample(newTrack.id, buffer, {
duration: Math.round(1000 / frameRate),
dts: sampleNumber * Math.round(1000 / frameRate),
cts: sampleNumber * Math.round(1000 / frameRate),
is_sync: chunk.type === 'key',
});
sampleNumber++;
}
const mp4Buffer = mp4boxfile.save();
return new Blob([mp4Buffer], { type: 'video/mp4' });
}
This function creates a new MP4 file, adds a video track, and then adds each encoded chunk as a sample. Finally, it saves the file and returns a Blob.
Now we can update the process button handler to encode and mux.
processButton.addEventListener('click', async () => {
if (!fileBuffer) return;
processButton.disabled = true;
processButton.textContent = 'Processing...';
try {
const { track, chunks } = await parseMP4(fileBuffer);
// Reset processed frames
processedFrames.length = 0;
await decodeVideo(track, chunks, processFrame);
const width = track.video.width;
const height = track.video.height;
const frameRate = track.video.fps || 30;
const encodedChunks = await encodeVideo(processedFrames, width, height, frameRate, track.codec);
const mp4Blob = createMP4(encodedChunks, track, width, height, frameRate);
// Create download link
const url = URL.createObjectURL(mp4Blob);
downloadLink.href = url;
downloadLink.download = 'processed_' + fileName;
downloadLink.style.display = 'block';
downloadLink.textContent = 'Download Processed Video';
// Optionally preview the processed video
preview.src = url;
} catch (error) {
console.error('Processing error:', error);
alert('An error occurred while processing the video.');
} finally {
processButton.disabled = false;
processButton.textContent = 'Process Video';
}
});
Now when you click the process button, the app will decode the video, apply the grayscale filter, encode the frames, and create a new MP4 file. The download link will appear, and you can save the processed video.
Step 8: Test and Troubleshoot
Now it is time to test your app. Open index.html in a browser that supports WebCodecs, select an MP4 file, and click the process button. You should see the video preview, and after a few seconds, a download link should appear.
If it does not work, here are some things to check.
Make sure your browser supports WebCodecs. You can check by opening the console and typing 'VideoDecoder' in window. If it returns false, your browser does not support it.
Make sure the video file is an MP4 with H.264 video. Other codecs may not work without additional configuration.
Check the console for errors. The decoder and encoder callbacks will log errors if something goes wrong.
If the output video looks wrong, check the timestamps and durations. The encoder and muxer need accurate timing information to produce a smooth video.
If the app is slow, try processing a shorter clip or reducing the resolution. WebCodecs is fast, but it still takes time to process large files.
Advanced Topics
Once you have the basics working, there are many ways to extend this tutorial.
You can add audio support by using AudioDecoder and AudioEncoder. The process is similar to video, but you need to handle audio chunks and samples separately.
You can use Web Workers to run the decoding and encoding in a background thread. This keeps the UI responsive and allows for faster processing.
You can add more filters. The grayscale filter is just one example. You can apply blur, brightness, contrast, or even custom effects using WebGL.
You can support more container formats. mp4box.js handles MP4, but you can use other libraries for WebM, MOV, and other formats.
You can optimize the encoder settings. The bitrate, framerate, and codec parameters can all be adjusted to balance quality and file size.
Common Pitfalls
WebCodecs is powerful, but it has some quirks. Here are a few common pitfalls to avoid.
Forgetting to close frames. VideoFrame objects hold onto memory. If you do not call frame.close(), you will run out of memory quickly. Always close frames when you are done with them.
Ignoring timestamps. Timestamps are critical for video playback. If they are wrong, the video will play at the wrong speed or stutter. Make sure you copy timestamps from the decoded frames to the encoded frames.
Not handling keyframes. Keyframes are important for seeking and for decoding. Make sure you mark keyframes correctly when encoding.
Assuming browser support. WebCodecs is not supported in all browsers. Always check for support and provide a fallback if necessary.
Forgetting about container formats. WebCodecs works with raw codec data, not with MP4 or WebM files. You need a separate library to handle the container.
Final Thoughts
WebCodecs opens up a world of possibilities for client-side video processing. You can build video editors, transcoders, thumbnail generators, and more, all without a server. The API is low-level, which means you have a lot of control, but it also means you have to handle more details yourself.
In this tutorial, we built a simple app that decodes an MP4, applies a grayscale filter, and encodes it back into a new MP4. We covered the core concepts: VideoDecoder, VideoEncoder, VideoFrame, and EncodedVideoChunk. We also touched on muxing with mp4box.js.
With this foundation, you can start exploring more advanced use cases. Try adding audio, using Web Workers, or implementing more complex filters. The more you experiment, the more comfortable you will become with the API.
Client-side video processing is still a relatively new field, but it is growing fast. As browser support improves, we will see more and more apps that take advantage of it. By learning WebCodecs now, you are getting ahead of the curve.
Comments (0)