Robert Harder » PhD

Eye Track Visualization with Scalable Vector Graphics

Robert Harder — Mon, 30 Mar 2009 15:32:12 +0000

A friend of mine Patrick Jungkunz (German Navy) is doing some research that involves gathering eye track data as participants look for hostile forces in a computer-generated urban battle scene, and I thought that being able to play back the data would be helpful. It was time to dust off the old XML/XSL skills and brush up on Scalable Vector Graphics (SVG), an XML-based graphics format that also supports animated elements (actually, I first thought I’d use SMIL, a multimedia format that seemed perfect for this, but I found better SVG support in my tool chain).

You can download the XML, XSL, and SVG files below or an MPEG-4 video capture of Safari rendering the SVG animation. If you want to try the SVG file itself, you’ll need a fairly capable browser. At the time of this writing, only Safari 4 and Opera 9 had sufficient SVG support.

Click to view SVG file in your browser

Click to view MPEG-4 screen capture of SVG animation

Click to download source material (zipped)

Python-Pickled Raw Data

My friend processed the raw data collected by the commercial eye tracker for his own analysis and stored the data in a Python pickle (*.pkl) file, a sort of internal serialization of Python objects to disk. My first step was to convert the data to XML, and I feared I would have to invent some sort of ad hoctagging system for notating eye track data, fixations (lingering on a position), and where the participant clicked when detecting a bad guy. Fortunately I found the gnosis.xml.pickle package which let me convert each standard pickle file to an XML pickle file with two lines, where fin and fout are two files (input and output) opened with standard Python file(..) commands:

x = pickle.load(fin)
gnosis.xml.pickle.dump(x,fout)

Since the original Python object that was serialized was an array of arrays (of arrays…) of data, the XML-pickled file also had these characteristics:

...

You can begin to imagine the format of the data, with scene identifications (0003), time stamps, and x/y coordinates of where a click took place. Here is the definition provided by my friend. It requires a little reading between the lines.

sceneL ::= [ scene ]
scene ::= [sceneName, startTime, endTime, actions,
  gazes,fixations]
actions ::= [action]
gazes ::= [gaze]
gaze ::= [frame number, gmt seconds, gmt milliseconds,
  saccade flag, gaze pixel location X, gaze pixel
  location Y, velocity, acceleration, jerk]
fixations = [fixation]
fixation = [x,y,start,end]

Using XSL to Transform XML to SVG

Let me just take a moment to say that although XML and XSL were supposed to bring us some kind of unity with data with organs making reverential sounds in the background, I’ve never actually had XML live up to its promise the way it has with this project.

Before processing individual eye track points, I needed to collect some overall data about the scene (like duration) and establish some parameters to use over and over again (like colors). The xsl:variable tag proved invaluable here, and I was able to do basic math on the time stamps provided in the pickle data. Incidentally, since this isn’t an XSL tutorial, I’m going to skip some of the basics like the header line of an XSL file and talk instead about the more interesting parts. Feel free to refer to the actual files (available for download from the link at the beginning of the post) for all the gory details.

Here are some of the xsl:variable tags used at the beginning of the document:

...

You can see that XPath is sufficient to reference the Python pickled arrays by position (XPath is one-indexed, not zero-indexed). The START_MILLIS and END_MILLIS data is boldfaced in the data extract above. Note that although Python thought one of the data values was numeric and one was a string, it did not adversely affect us since XSL coerced the string into a number when it calculated the DURATION_MILLIS variable.

Root Node

I establish the first xsl:template node that matches the pickled XML document root (/).

...

You’ve got to love the xsl:attribute tag that lets you dynamically add attributes to nodes so easily. OK, it could be easier if it had a select attribute, but what are you going to do? Come to think of it, why doesn’t it have a select attribute? If you’re assigning an attribute value, it’s not like you’re going to stick a whole XML node in there!

Eye Tracks

Let’s jump to where we want to start showing individual eye tracks, represented by faint blue circles in the final image.

    Eye Tracks - Persistent

Nice and clean, we just call xsl:apply-templates and pass in the array of items representing the tracks. The Python Gnosis XML pickling is a little funny in that you end up with what seems like extra item tags, but it’s quite consistent and easy to figure out.

Further down we have the entire eye track template. Note how we match the same XPath statement that we’re passing.

We initially make the circles invisible by setting the display property to none. The timestamp associated with each track determines when we make the circle visible by setting the displayproperty to block. This is a CSS property, so none and block may be familiar to you if you’ve used CSS before. We use the SVG tag s:set to change a value at a point in time.

Similar to s:set is s:animate which makes changes to a value over time as opposed to instantly. The circle will fade in opacity from 100% (1) to 0% (0) over a period of 0.2 seconds. This has the effect that a circle appears brightly and then fades into the background. Because the circle’s opacity is set to 0.1 in the line 0.1“>, that is the value to which the circle will return at the end of the animation. The animation should probably go from 1 to 0.1 instead of 1 to 0 but the difference is hardly noticeable.

To help the eye follow the appearing circles, I also make a single circle that moves from point to point, and the SVG engine will interpolate the positions. See the source code for that one.

Check out the XSL in the source download above for complete details. Also there are more examples here: http://www.movesinstitute.org/~rharder/eye/.

Initial Thoughts: 3D via the web

Robert Harder — Tue, 20 Jan 2009 22:09:02 +0000

Thanks to inspiration at the 2008 I/ITSEC event, I found what I think will be a good dissertation topic for me. I’m interested in the 3D worlds, especially with how we keep reinventing the wheel. It occurred to me that we’re sort of repeating some previous computer history: First we had BBSes; then we had Compuserve, America On Line, and others; then we the web. Similarly we had 3D games that interconnected (like BBSes), now we have massive 3D social worlds (like America On Line), but where are we going with it. Are we doomed to publish our 3D content only via Second Life?

Below is the text of what I wrote up about my idea as well as the source PDF.

PDF: 3DViaWebOverview.pdf

3D via the Web and the New WWW: World Within World
Robert Harder, Naval Postgraduate School, rharder@nps.edu
January 20, 2009

Abstract

From sparsely connected bulletin board systems to monolithic online services like America On Line, we broke the mold when we embraced the world wide web as the medium where anyone could publish content as they saw ﬁt. 3D worlds also moved from connected ﬁrst person shooter games to monolithic 3D social worlds. Now the architecture is almost entirely in place for anyone to publish 3D content as rich or as lean as the imagination allows.

From Bulletin Boards to the World Wide Web

When we were ﬁrst learning to interact with people on remote computers there were Bulletin Board Systems (BBS) where users who were close geographically (or could aﬀord long distance phone calls) dialed in and exchanged email, ideas, and ﬁles. Many of these BBSes were successfully linked to each other by Fidonet and other services to exchange data.

Eventually companies poured money into systems to build major cross-country and international worlds such as Compuserve, Prodigy, and America On Line. These successful online communities were isolated from each other, but they provided users with impressive and appreciated content. The content was provided by 1) the service itself, 2) customers paying for space, and 3) the community of users.

When the World Wide Web ﬁrst came on the scene, enabled by the Hyper- Text Transfer Protocol (HTTP) and HyperText Markup Language (HTML), there was not enough content available to entice people to switch from their on- line provider to this new medium, but then individuals and companies realized that they could publish content and provide a virtual presence without having to go through the stove piped providers. Anyone with a web server could have complete control over what visitors saw when they visited a site.

Since those early web days, new capabilities, such as dynamic HTML, script- ing, and new realizations, such as scripts both making their own connections and modifying pages dynamically (commonly referred to as Web 2.0), new classes of applications have arisen and user expectations have grown. Users want rich, interactive content. They expect to see who is online and what their friends are doing. Social networks and collaboration tools connect users synchronously and let them coordinate on shared data. They also want immersive 3D worlds.

From 3D Games to World Within World

When users ﬁrst learned to interact in 3D worlds, gaming companies connected their customers to online communities, usually in the form of “ﬁrst person shooter” games. Although Game A from one company could not talk to Game B from another, the experience was new and exciting and led to a boom in the gaming industry. As with the BBSes of old, participants could interact with others of similar interests—namely, the game they were playing. While few companies embraced talking across games as with the Fidonet of yesteryear, the United States military forced its vendors to provide simulations that could talk in a common language to avoid the dreaded “lock-in.” (These simulations use two protocols known as Distributed Interactive Simulation (DIS) and High Level Architecture (HLA) to try to ensure Simulation A from one company can talk to Simulation B from another).

Soon companies began investing signiﬁcant resources into becoming the next Compuserve or America On Line in the social, 3D world. Now individuals and companies can exchange real money for virtual money, purchase “land” in a provider’s virtual world, and build it up within the conﬁnes of that company’s architecture. Again these provide users with impressive and appreciated con- tent, and the content is again provided by 1) the service itself, 2) customers paying for space, and 3) the community of users.

These 3D world providers oﬀer what might be considered an irreplaceable side eﬀect of moving to a 3D world that models reality: they are the brokers who decide that this piece of land “belongs” to this customer and that land to another. If we miss the sleight of hand, we might think that we are stuck with this model.

There is already another model for taking control of our own content and publishing as rich or as lean an experience as we can imagine. It is the World Wide Web, and the new WWW is World Within World.

World Within World Real Estate

Instead of tying oneself to a single “world” where everyone is vying for a piece of it, one can publish a world that links to others as the current web does. The architecture is right under our noses and is almost entirely ready to execute.

From a hyperlink in either kind of WWW a user could connect to an immersive 3D world just as easily as loading a traditional web page. Depending on the sophistication of the 3D site, users might click on doors (one of several apt metaphors for the new hyperlink) to move to a new room or they might move from one room or ﬁeld to another as content is dynamically paged in, just as successful Web 2.0 sites do now. A door might lead to another area of the same site or to another site altogether—a World Within World.

The original architects of the web gave us HTTP and HTML, and now there is X3D, which is the World Wide Web Consortium’s (W3C) standard for 3D modeling and a good start for 3D markup. The social networks learned the algorithms for connecting friends with common interests. The online forums learned how to report who is live and online in real time. The Web 2.0 crowd learned how to interact dynamically with content and report back to a host server. HTML 5 even proposes server-side event generation to more easily enable live data feeds—a boon for companies wishing their 3D presence to include a reﬂection of other users in the system.

Your Presence in a World Within World

In the simplest case, a user’s avatar (their presence embodied in the 3D world) might not see any other humans except those provided by the hosting site, possibly driven by cognitive models to resemble real human behavior. A social networking site however might reﬂect back to the user others who are in a room of interest. And who says avatars have to resemble uglier versions of ourselves? If an avatar is also just another bit of markup, the artistic among us can build designer bodies that can go with us from world to world. In some worlds a user’s avatar may not be humanoid at all. A ﬂight simulator could instead become a World Within World browser with your favorite airplane being your avatar. A fantasy world might have dragon avatars as easily as knights.

Content providers have complete control over what their world looks like and how a user interacts with it. Will you provide training to employees or recruit new people? Will you show oﬀ 3D models of products in your retail store? Will you let users try on clothes on their designer avatar bodies? Will you be a portal to other worlds, oﬀering users a 3D Main Street experience?

What Are We Missing?

Many of the required components for the new worlds already exist: web servers, HTTP, server- and client-side scripting, 3D markup languages. Absolutely critical will be the presence of avatar-aware, 3D applications which render 3D content and provide motion for the avatars and interaction with the environment.

There are plenty of questions to answer on how best to represent avatars, how to communicate interactions with the server, how to redistribute data about fellow users, and more. How should the speciﬁcations support both a ballerina’s graceful movements and a more generic description of movement from Point A to Point B? How can these worlds provide new behaviors for avatars (trying on virtual clothes, learning a virtual dance, holding an object its hands)? Do avatars follow a user from site to site (this may be a chance for OpenID to shine)? How much can Asynchronous JavaScript and XML (AJAX) via the Transmission Control Protocol (TCP) sustain synchronizing avatars in the social 3D worlds?

These questions and more will someday be answered, but in the meantime, we are close to realizing a 3D web where anyone can publish content and where worlds eﬀortlessly hyperlink to other worlds. A ubiquitous, immersive, virtual 3D world is in fact nearly a reality.