Transparency with HTML5 Video + Canvas in Manipulator

[If you want to skip directly to looking at the code, the github for the project is here.]

When Matt and I were making Manipulator, the largest technical hurdle we needed to solve was related to the video files themselves: how could we isolate Ty such that he would appear as a video in a static scene? If you aren’t aware of the music video, I’d suggest checking it out so you can see what I mean.


Ty sitting in a bedroom in the first scene.

The simplest solution would have been to simply have Ty’s file format have a transparent background. That way we could do the simple green-screen trick and just plop him in. Unfortunately, HTML5-supported video formats currently do not support transparencies (rumored that they will sometime soon in the future), so we had to think of a new plan. Other non-video file formats weren’t going to be a good idea, either, since types such as GIFs were going to be outrageously huge.

F— It, We’ll Green-Screen it Live

The only real solution that we found was to do a live green screen filter on every video as it played. We knew that other music videos had done this, so we reached out to them. They blew us off, so we figured out how to do it ourselves and, to help all others who might want to do the same thing in the future, we’ve written this guide and put all the code online. Hooray open source!

ty close image still

Screen grab from the second scene movie file before filtering.

The Process

The overall process is quite simple. You’ll want to add an event listener to the movie file to detect when that specific clip is playing so it can run a filtering function on it each frame:

video = document.getElementById('video');
video.addEventListener("play", filterEffect, false);

function filterEffect() {

When a movie file is playing, these steps happen every frame before it renders:

  1. Draw the moving file onto the canvas.
  2. Pull the image data from said canvas.
  3. Go pixel by pixel looking for the green screen. If it’s there, make it that pixel transparent.
  4. Draw the filtered image data back onto the canvas.

The code itself is just a reflection of this 4-step process. A canvas’ 2d drawing context has a couple of functions that make this possible: drawImage() and getImage().

Here is a simplified version of the code we used. Note that getImage() returns a single array which looks like [R1,G1,B1,A1,R2,G2,B2,A2,…,Rn,Gn,Bn,An], where R1, G1, B1, and A1 are the red, green, blue, and alpha of the first pixel, R2 is red of the 2nd pixel, etc.. This function simply checks if the green value is super high and makes that pixel transparent. Also, vidRender and vidRctx are the canvas and context that you want to draw to, respectively.

function renderCanvas(vid) {
     if ( vid.paused || vid.embed ) return false;
     var vidData = vidRctx.getImageData(0,0,vidRender.width, vidRender.height);

     for ( var i=0; i<; i+=4 ) {
         // if green is super high, filter it out.
         if ([i+1] > 230 ) {
    [i+3] = 0;
     }, 20);

Our code has many more filters and some additional code that’s specific to how we used the video file, but it followed the same logic.  Feel free to look at our code on github.

scene 2

Still from scene 2, post filter and with images.

You can see the finished interactive music video here, or find links and more information about it on my portfolio.

3 thoughts on “Transparency with HTML5 Video + Canvas in Manipulator

  1. Robin says:

    Insanely cool! People have been telling me this “can’t be done” :)
    How large is the manipulator video? Will it run on IOS? (I have it up on my iPhone now but it’s been ‘loading’ for quite a while now.
    Thanks!! :)

  2. simon says:

    Hey Robin. Oops late reply! The page is pretty large since the videos themselves are big. Though it should be possible to get either the size or load time way down by doing some magic with streaming (just in case you were considering doing it on your own).

    Theoretically it should run in mobile but we decided to forego it for this project (time and budget restraints), since mobile browsers are more finicky than web. Works best on Chrome, though!

  3. farnaz says:

    that’s great man

Leave a Comment

Your email address will not be published. Required fields are marked *