video

MANIPULATOR

Manipulator is a web-based interactive music video made for Ty Segall’s song, Manipulator, the title track for his 2014 album, Manipulator. Using HTML5 + javascript, the “music video”, in the theme of the song, allows the user/player to manipulate the objects in the scene by clicking on them. Collaboration with Matt Yoka and premiered by the NY Times’s T Magazine.

Website: http://www.ty-segall.com/manipulator/

Select Coverage

Transparency with HTML5 Video + Canvas in Manipulator

[If you want to skip directly to looking at the code, the github for the project is here.]

When Matt and I were making Manipulator, the largest technical hurdle we needed to solve was related to the video files themselves: how could we isolate Ty such that he would appear as a video in a static scene? If you aren’t aware of the music video, I’d suggest checking it out so you can see what I mean.

manipulator03

Ty sitting in a bedroom in the first scene.

The simplest solution would have been to simply have Ty’s file format have a transparent background. That way we could do the simple green-screen trick and just plop him in. Unfortunately, HTML5-supported video formats currently do not support transparencies (rumored that they will sometime soon in the future), so we had to think of a new plan. Other non-video file formats weren’t going to be a good idea, either, since types such as GIFs were going to be outrageously huge.

F— It, We’ll Green-Screen it Live

The only real solution that we found was to do a live green screen filter on every video as it played. We knew that other music videos had done this, so we reached out to them. They blew us off, so we figured out how to do it ourselves and, to help all others who might want to do the same thing in the future, we’ve written this guide and put all the code online. Hooray open source!

ty close image still

Screen grab from the second scene movie file before filtering.

The Process

The overall process is quite simple. You’ll want to add an event listener to the movie file to detect when that specific clip is playing so it can run a filtering function on it each frame:

video = document.getElementById('video');
video.addEventListener("play", filterEffect, false);

function filterEffect() {
     renderCanvas(video);
}

When a movie file is playing, these steps happen every frame before it renders:

  1. Draw the moving file onto the canvas.
  2. Pull the image data from said canvas.
  3. Go pixel by pixel looking for the green screen. If it’s there, make it that pixel transparent.
  4. Draw the filtered image data back onto the canvas.

The code itself is just a reflection of this 4-step process. A canvas’ 2d drawing context has a couple of functions that make this possible: drawImage() and getImage().

Here is a simplified version of the code we used. Note that getImage() returns a single array which looks like [R1,G1,B1,A1,R2,G2,B2,A2,…,Rn,Gn,Bn,An], where R1, G1, B1, and A1 are the red, green, blue, and alpha of the first pixel, R2 is red of the 2nd pixel, etc.. This function simply checks if the green value is super high and makes that pixel transparent. Also, vidRender and vidRctx are the canvas and context that you want to draw to, respectively.

function renderCanvas(vid) {
     if ( vid.paused || vid.embed ) return false;
     vidRctx.drawImage(vid,0,0);
     var vidData = vidRctx.getImageData(0,0,vidRender.width, vidRender.height);

     for ( var i=0; i<vidData.data.length; i+=4 ) {
         // if green is super high, filter it out.
         if ( vidData.data[i+1] > 230 ) {
              vidData.data[i+3] = 0;
         }
     }
     
     vidRctx.putImageData(vidData,0,0);
     
     setTimeout(function(){
          renderCanvas(vid);
     }, 20);
}

Our code has many more filters and some additional code that’s specific to how we used the video file, but it followed the same logic.  Feel free to look at our code on github.

scene 2

Still from scene 2, post filter and with images.

You can see the finished interactive music video here, or find links and more information about it on my portfolio.