javascript

MANIPULATOR

Manipulator is a web-based interactive music video made for Ty Segall’s song, Manipulator, the title track for his 2014 album, Manipulator. Using HTML5 + javascript, the “music video”, in the theme of the song, allows the user/player to manipulate the objects in the scene by clicking on them. Collaboration with Matt Yoka and premiered by the NY Times’s T Magazine.

Website: http://www.ty-segall.com/manipulator/

Select Coverage

No Hashtag, No Reply

No Hashtag, No Reply is a simple project that, using keywords, searches twitter for tweets that contain no hashtags, no mentions, no RT, and no links. It breadcrumbs through users’ thoughts that have been broadcast into the void of the internet. These anonymous thoughts can be anything from useful facts to secret desires.

Website: nohashtagnoreply.simonwiscombe.com

Transparency with HTML5 Video + Canvas in Manipulator

[If you want to skip directly to looking at the code, the github for the project is here.]

When Matt and I were making Manipulator, the largest technical hurdle we needed to solve was related to the video files themselves: how could we isolate Ty such that he would appear as a video in a static scene? If you aren’t aware of the music video, I’d suggest checking it out so you can see what I mean.

manipulator03

Ty sitting in a bedroom in the first scene.

The simplest solution would have been to simply have Ty’s file format have a transparent background. That way we could do the simple green-screen trick and just plop him in. Unfortunately, HTML5-supported video formats currently do not support transparencies (rumored that they will sometime soon in the future), so we had to think of a new plan. Other non-video file formats weren’t going to be a good idea, either, since types such as GIFs were going to be outrageously huge.

F— It, We’ll Green-Screen it Live

The only real solution that we found was to do a live green screen filter on every video as it played. We knew that other music videos had done this, so we reached out to them. They blew us off, so we figured out how to do it ourselves and, to help all others who might want to do the same thing in the future, we’ve written this guide and put all the code online. Hooray open source!

ty close image still

Screen grab from the second scene movie file before filtering.

The Process

The overall process is quite simple. You’ll want to add an event listener to the movie file to detect when that specific clip is playing so it can run a filtering function on it each frame:

video = document.getElementById('video');
video.addEventListener("play", filterEffect, false);

function filterEffect() {
     renderCanvas(video);
}

When a movie file is playing, these steps happen every frame before it renders:

  1. Draw the moving file onto the canvas.
  2. Pull the image data from said canvas.
  3. Go pixel by pixel looking for the green screen. If it’s there, make it that pixel transparent.
  4. Draw the filtered image data back onto the canvas.

The code itself is just a reflection of this 4-step process. A canvas’ 2d drawing context has a couple of functions that make this possible: drawImage() and getImage().

Here is a simplified version of the code we used. Note that getImage() returns a single array which looks like [R1,G1,B1,A1,R2,G2,B2,A2,…,Rn,Gn,Bn,An], where R1, G1, B1, and A1 are the red, green, blue, and alpha of the first pixel, R2 is red of the 2nd pixel, etc.. This function simply checks if the green value is super high and makes that pixel transparent. Also, vidRender and vidRctx are the canvas and context that you want to draw to, respectively.

function renderCanvas(vid) {
     if ( vid.paused || vid.embed ) return false;
     vidRctx.drawImage(vid,0,0);
     var vidData = vidRctx.getImageData(0,0,vidRender.width, vidRender.height);

     for ( var i=0; i<vidData.data.length; i+=4 ) {
         // if green is super high, filter it out.
         if ( vidData.data[i+1] > 230 ) {
              vidData.data[i+3] = 0;
         }
     }
     
     vidRctx.putImageData(vidData,0,0);
     
     setTimeout(function(){
          renderCanvas(vid);
     }, 20);
}

Our code has many more filters and some additional code that’s specific to how we used the video file, but it followed the same logic.  Feel free to look at our code on github.

scene 2

Still from scene 2, post filter and with images.

You can see the finished interactive music video here, or find links and more information about it on my portfolio.

Indesign — Merging All Files

I recently had to combine hundreds of indesign files into a single document, so I made this quick js-based extension. Originally made for CS5; not sure how it works with any others. Feel free to use it.

Download it here or copy-paste from below.

/********* CODE START **********/

// DESCRIPTION: Mass combine a bunch of files (files have one document only) into one.
// Simon Wiscombe
// 6 July 2013
// INSTRUCTIONS
// Put all the files you want to combine into one document in a folder, named in the order
// you want them to appear in the book (e.g. “001.indd”, “002.indd”, etc. or something) so
// they appear correctly when sorted by name in finder/explorer.

#target indesign-7.0
// set the destination document as the active document.
var destination_doc = app.activeDocument;
// select the source folder
var sourceFolder = Folder.selectDialog(“Select a folder with source InDesign files.”);
// if we’re not in the source folder, stop running the script.
if ( !sourceFolder ) {
exit(0);
}
// set file list and run through each file. Sorting them alphanumerically.
var fileList = sourceFolder.getFiles();
fileList.sort();
// run through each file
for ( var i = 0; i < fileList.length; i++ ) {
var source_file = fileList[i];
// making sure it’s an indesign file…
if ( source_file instanceof File && source_file.name.match(/\.indd$/i)) {
app.open(source_file);
var source_doc = app.documents.item(source_file.name);
var sourcePages = source_doc.pages.item(0);
// break the master page items so they can be moved onto the new document.
var masterItems = sourcePages.masterPageItems;
if ( masterItems.length > 0 ) {
for ( var j=0; j<masterItems.length; j++ ) {
masterItems[j].override(sourcePages);
}
}
// removing the applied master (this can mess up some files if not done)
sourcePages.appliedMaster=null;
// duplicating it in the original file (due to errors) and them moving it to
// the destination document.
sourcePages.duplicate(LocationOptions.AFTER, source_doc.pages.item(0));
sourcePages.move(LocationOptions.AFTER, destination_doc.pages.item(-1));
// closes the file that was opened without saving (to avoid memory problems)
app.activeDocument.close(SaveOptions.NO);
}
}

/********* CODE END **********/

 

I’ve been asked to put a version for multiple-document indesign files, so here is a code that SHOULD work.
WARNING: THIS HAS NOT BEEN TESTED. It may not work.

 

/********* CODE START **********/

// DESCRIPTION: Mass combine a bunch of documents (with multiple pages) into one.
// Simon Wiscombe
// 19 Nov 2013
// INSTRUCTIONS
// Put all the files you want to combine into one document in a folder, named in the order
// you want them to appear in the book (e.g. “001.indd”, “002.indd”, etc. or something) so
// they appear correctly when sorted by name in finder/explorer.

#target indesign-7.0

// set the destination document as the active document.
var destination_doc = app.activeDocument;

// select the source folder
var sourceFolder = Folder.selectDialog(“Select a folder with source InDesign files.”);

// if we’re not in the source folder, stop running the script.
if ( !sourceFolder ) {
exit(0);
}

// set file list and run through each file. Sorting them alphanumerically.
var fileList = sourceFolder.getFiles();
fileList.sort();

// run through each file
for ( var i = 0; i < fileList.length; i++ ) {
var source_file = fileList[i];

// making sure it’s an indesign file…
if ( source_file instanceof File && source_file.name.match(/\.indd$/i)) {
app.open(source_file);
var source_doc = app.documents.item(source_file.name);

for ( var count = 0; count < source_doc.pages.length; count++ ) {
var sourcePages = source_doc.pages.item(count);

// break the master page items so they can be moved onto the new document.
var masterItems = sourcePages.masterPageItems;
if ( masterItems.length > 0 ) {
for ( var j=0; j<masterItems.length; j++ ) {
masterItems[j].override(sourcePages);
}
}

// removing the applied master (this can mess up some files if not done)
sourcePages.appliedMaster=null;

// duplicating it in the original file (due to errors) and them moving it to
// the destination document.
sourcePages.duplicate(LocationOptions.AFTER, source_doc.pages.item(0));
sourcePages.move(LocationOptions.AFTER, destination_doc.pages.item(-1));
}

// closes the file that was opened without saving (to avoid memory problems)
app.activeDocument.close(SaveOptions.NO);
}
}

/********* CODE END **********/