The Wilderness Downtown – Behind The Scenes

Project NameThe Wilderness Downtown
Client / AgencyB-Reel
Google Creative Lab
Arcade Fire

Background: coincidences and a bit of luck

A few years ago I was attracted by the great interface of “The Turn” website. Digging into the credits, I found out that Guille Lopez was responsible for the development. Funny enough, Guille was from Barcelona and he studied exactly the same and in the same place as I did.

Encouraged by the parallelism, I decided to write him and send my congratulations for the great work. By that time, I was still based in Berlin and Guille told me that he just moved to London where he started working at B-Reel.

About a year ago, it was me who moved to London and I had the chance to personally meet Guille right before he moved from London to B-Reel’s New York office.

Ten months later, Guille contacted me again and asked me if I was interested in taking part of a very exciting HTML5 music video project in collaboration with Google and B-Reel.

As soon as I got the briefing and we started talking about it, I got very excited. The video was going to be directed by Chris Milk, Aaron Koblin was behind it and Mr.Doob himself (who I casually met just a few days before all this happened) was going to be actively involved.

We were really lucky to have someone like Jaime Sánchez joining the dev team, which was completed by George Michael Bower and Michael Chang from the Google Creative Lab, and Eric Haeton who was in charge of the backend.

My role in the project

As you may already know, The Wilderness Downtown is composed by several pop up windows showcasing different contents during the interactive video-clip experience. Each content pops in and out at a certain time during track’s playback.

We had an amazing team of developers taking care of the production of all window’s contents: bird flocks, canvas 360° street-view, tree explosions, 3D optimisation, Google maps integration and routes calculation, Harmony drawing tool’s adaptation and so on…

B-Reel wanted me to create the mainframe of the application while making sure that everything came nicely together. There were a bunch of feature requests in order to make the concept work: content pre-loading, sharing information in-between windows, sound spectrum support, liquid pop-up window layout, a main controller, tracking …

The sequencer – aka the engine –

Luckily enough, Mr.Doob created this magic little Sequencer JS object that provides sequencing control of what we call Effects. Effects can be added to the sequence as follows:

seqeuncer.add ( effect, start_time, duration );

Effects are objects with show / hide functions which are used to update sequencer’s state at a given time.

So the Sequencer object provides the core functionality of the engine but this needed to be extended to fully support the content / pop up window system we were aiming for.

One of the biggest constraints was obviously the fact of using pop-up windows since these need to be launched by a user click in order to avoid the default popup-blocker behaviour. This actually means that we need such a work-flow:

1) Create all content objects and start loading them
2) When loaded, tell the user to proceed by clicking a launch button
3) Create all popup windows at once and “hide” non-used windows
4) Define content targets (tell every content on which window should go)
5) Start audio+sequencer playback
6) Append contents to target windows as soon as they need to be in the sequence

So we needed to create two different kinds of Effects for the sequencer: Popup Window Effects and Content Effects.
Popup Window Effects can be understood as cue-points that define the size and position of a popup window at a given time.
Content Effects are mostly Canvas experiments that are appended into a popup window’s body at “show” (start_time) and removed at “hide” (end_time = start_time + duration). Content Effects may need to preload images or video and need to notify when they are successfully preloaded.

The complexity of making adjustments

We had initially a 8-window setup, and only during the first act of the video, we had already generated about 30 window cue-points. One cue-point looks like something like this:

sequencer.add(new PopupEffect(popup3, {x: 10, y: 1024, width: 100, height: 60}, false), 132.36700439453125, 24.432998657226562);

Notice that the latest two numbers refer to “start_time” and “duration”.
Besides the window cue-points, we need to assign every content to a specific window as follows:

addContentToSequence(1, popup2, 31.117, 28.383);

So, only during the first act of the video, we had already about 50 lines of code with floating point numbers defining times and durations.
The full clip is defined by 4 acts, so we could expect about 200 lines of code like the ones above.

I suddenly realised how painful would it be when Chris and Ben, creative director at B-Reel, would like to make content and time adjustments to the sequence, as well as change content / window assignments, pushing a part of the sequence a few seconds forwards, resizing windows, and so on.
Doing such changes manually could become a mess and would keep me too busy right at the end of the project, which could be quite inconvenient.

We like tools to automate boring processes

I thought it would be useful for Ben and Chris to be able to do changes in the sequence by themselves, so I decided to build an interface for generating our sequencer code.

Since the project was about Chrome and HTML5, I thought it would make sense to use the same technologies, so the sequencer tool was created with CSS/HTML + Javascript and a little bit of PHP to dynamically generate JS files on the server. To minimise production time, the tool was uniquely targeting Chrome / Webkit since was for internal use only.

The soundtrack timeline

The tool works time-based around a main timeline reproducing the same “We Used To Wait” soundtrack as a reference. On besides, the original animatic sketch from Chris Milk is added into it, so we have a constant visual reference of the current act / sequence.

It is possible to navigate through the timeline either by dragging the progress marker or by entering a time code manually. Hitting the spacebar key will toggle play/pause.

The cue-point editor: controlling windows

I have created a light JS window manager to control windows. The “fake” windows can be resized and moved around which updates / creates automatically a window cue-point.

The cue-points are stored and sorted by time values into a history drop-down which allows to directly jump to the particular cue-point.

We have a tween option that allows the windows to be animated when a cue-point is reached (We didn’t use this feature at the end although is still implemented and ready to be used).

If the sequence is played, it is possible to see on-the-fly the window animatic sequence as it will behave later within the “real experience”.

Copy & Paste
Ben wanted to be able to copy & paste cue-points since the “hidden” status of a window happens to be always the same width, height, x and y settings. This saved Ben some time :)

Content cue-points: assigning contents to windows

We need to distinguish in between two types of content. In one side we have pure video contents, and in the other hand the interactive / JS Canvas experiments.

The tool has a built-in video library that can be browsed and which displays video information such as size in pixels and duration. When assigning a video content into a window, we can automatically let the system calculate the duration of the content based on the original video duration.

If we are assigning other contents rather than video, we’ll only be accessing the “standard” parameters: width, height and duration.

All content objects support dynamic size (width and height). When selecting a target window for a content, the width and height are automatically adjusted to fit window’s size, so everything makes sense.

Dynamic canvas

We had a minimum target resolution of 1024 x 768 but loads of users have big screens with higher resolutions.

The tool displays a cyan 1024 x 768 canvas where to position and resize the windows. Later on, the real browser application is dynamically adjusting the positions of the windows and contents according to user’s resolution.

I need a break, please save my progress

There is a save functionality built-in, so the sequence can be finished another time.

Finally, let me preview it and see how it looks like

Current settings can be saved as Javascript sequencer code and be previewed in a version of the real application within a click.

Considerations and what could have been much better

The sequencer tool was a very useful toy for playing around and generating the final animatic but the interface and UX could have been much better.

You have to take into account that I couldn’t spend too much time on it since I needed to be working on all the other stuff; but would have been great to have a graphic drag-and-drop timeline interface to visualise and move around the cue-points.

Another cool feature would have been previewing content directly built in the fake windows of the sequencing tool.

Some problems faced during the progress and things worth sharing

The contents were generated before being appended to the DOM. Some contents needed to fetch data from Google and preload some images but I faced a problem with the videos since media elements (video and audio) won’t load if they aren’t appended to the DOM; so we had to rely on cache for the videos.

It was only possible to preload a video by appending it into a DIV located outside the main window or invisible and remove it again when the video reached the “ready” state (no need to actually preload the whole video).

Safari is very, very picky
Safari is very strict on which audio and video codecs should be in the web. It would be a nice move if Apple would add support for Vorbis/Theora but they clearly want H264 and AAC to be the standard HTML5 codecs.

I recommend to any Safari user in Mac to install this great Quicktime component that will allow your Safari as well as your Quicktime player to support OGG and OGV files:

We obviously needed to have OGG / OGV with MP3 and MP4 fallbacks, but the most ridiculous episode was when Safari wasn’t working with MP3 files encoded with LAME. We had issues with different versions of Safari and at the end we had to encode the MP3 file using iTunes. Surprised?

A lovely python script to generate sound spectrum values at a perfect framerate
We needed some audio spectrum support and you can’t do this in real-time using Javascript. Well, Firefox is able to do this (at least the 4 beta), but not Safari nor Chrome. Mr.Doob pointed me at a great python script that lets you generate an array of spectrum values at a perfect framerate, given a WAVE file:

import math
import struct
import wave
import sys

w =[1], 'rb');
# We assume 44.1k @ 16-bit, can test with getframerate() and getsampwidth().
sum = 0;
value = 0;
delta = 0;
amps = [ ];

for i in xrange(0, w.getnframes()):
	# Assume stereo, mix the channels.
	data = struct.unpack('<hh', w.readframes(1))
	sum += (data[0]*data[0] + data[1]*data[1]) / 2
	# 44100 / 60 = 735 (60 fps)
	if (i != 0 and (i % 735) == 0):
		value = int(math.sqrt(sum / 735.0) / 10)
		amps.append(value) # absolute value no delta
		delta = value
		sum = 0

print amps

More information in his post here.

The web trend loop

When I started working in Barcelona back in 1999 there was a hype thanks to DoubleYou. It was the Netscape vs Explorer times and the hype was called DHTML (Dynamic HTML). DHTML isn’t far of the technology used for “The Wilderness Downtown”: HTML + Javascript.

Flash was by then stuck at version 4 and AS1 and it was becoming popular thanks to people like Joshua Davis and Gmunk.

I remember one of my first projects. We created a website for the spanish pavilion of an art fair that I barely remember. We used DHTML and we recreated a Mac OS 9 OS using Javascript and with a JS window manager, quite a coincidence :)
I remember one of the “plus” features being an animated (DIV tween) desktop background. It was 2000 and it was running in the two major browsers: Netscape and Explorer. The site won’t work in any of the modern browsers today.

I have been working with AS2 and AS3 for many years now and I really wanted to work in a HTML / Javascript interface again, and this project was as challenging as ironically bringing me in a way back to where I started. Of course many things have changed: AJAX, Canvas, Video, Audio, minimised cross-browser pain and WebGL around the corner; but Javascript feels quite the same as it did back then.

Oh, and don’t get me wrong. I am not trying to participate in any “Flash VS HTML5″ bullshit. I just enjoy using different technologies and keep learning things out of challenging projects. In fact, I think I should step out of the web soon and invest a few months in other types and formats of work.

I would like to know more about the graphics / content part

If you would like to know more about the 3D / Graphics, optimisation and the Google API, you should read this great post by Mr. Doob.

There is also some more information about the project here.

Have your say:

Johnb205 says:

What’s up Jackson, if you are a new internet user then you have to visit daily this web site and read the updated content at at this place. ebfcfekeekef

Seo Articles says:

With havin so much written content do you ever run into any
problems of plagorism or copyright violation? My
blog has a lot of exclusive content I’ve either authored myself or outsourced but it
appears a lot of it is popping it up all over the internet without my authorization.
Do you know any methods to help prevent content from
being ripped off? I’d truly appreciate it.

Wedding rings of the hcg weight loss diet has our throw choose comparable to dieter
those a lot Health calorie-restricted diet regime. You will then be prescribed a daily very low dose of HCG (oral or injection) and begins your low-calorie, yet sensible diet (typically 800-1200 calories a day).
The diet incorporates the HCG hormone that improves and re-sets the metabolic rate for your system therefore
an individual doesn’t wind up having intensive desire to have food.

Mathias says:

With online search on the internet. Instalment Plan many of the
unions’ members, who tend to trade in their vehicles frequently, choose its 72-month loans because the longer terms makes the monthly payments promptly which will develop your credit score. In a future post I will reconcile my asking about personal loanss.

We’ll be liveblogging a few, but if grandma entered the singles bar and take his Farmacia On Line with him. So if you are using the erectile dysfunction is used to improve other items.

[...] Weekly Link Roundup 11/15 – 11/19 Posted on Tue, November 23, 2010 by Rick Mason Comments Animatic for Chris Milk’s Wilderness Downtown Remember the HTML 5 project Chris Milk did in coordination with Google for Arcade Fire a few months ago? Well here’s the original animatic showing the vision for the piece. Or for a more technical look at the guts of this project see this detailed post. [...]

[...] Prats Molner finally has a write-up on the behind the scenes here. This entry was posted in interactive. Bookmark the permalink. ← Visual Trend Report [...]

Welcome to the web archive of Eduard Prats Molner, a developer exploring interactive media, user experience and visualization.