Eye Track Visualization with Scalable Vector Graphics
A friend of mine Patrick Jungkunz (German Navy) is doing some research that involves gathering eye track data as participants look for hostile forces in a computer-generated urban battle scene, and I thought that being able to play back the data would be helpful. It was time to dust off the old XML/XSL skills and brush up on Scalable Vector Graphics (SVG), an XML-based graphics format that also supports animated elements (actually, I first thought I’d use SMIL, a multimedia format that seemed perfect for this, but I found better SVG support in my tool chain).
You can download the XML, XSL, and SVG files below or an MPEG-4 video capture of Safari rendering the SVG animation. If you want to try the SVG file itself, you’ll need a fairly capable browser. At the time of this writing, only Safari 4 and Opera 9 had sufficient SVG support.
Python-Pickled Raw Data
My friend processed the raw data collected by the commercial eye tracker for his own analysis and stored the data in a Python pickle (*.pkl) file, a sort of internal serialization of Python objects to disk. My first step was to convert the data to XML, and I feared I would have to invent some sort of ad hoctagging system for notating eye track data, fixations (lingering on a position), and where the participant clicked when detecting a bad guy. Fortunately I found the gnosis.xml.pickle package which let me convert each standard pickle file to an XML pickle file with two lines, where fin and fout are two files (input and output) opened with standard Python file(..) commands:
x = pickle.load(fin) gnosis.xml.pickle.dump(x,fout)
Since the original Python object that was serialized was an array of arrays (of arrays…) of data, the XML-pickled file also had these characteristics:
You can begin to imagine the format of the data, with scene identifications (0003), time stamps, and x/y coordinates of where a click took place. Here is the definition provided by my friend. It requires a little reading between the lines.
sceneL ::= [ scene ] scene ::= [sceneName, startTime, endTime, actions, gazes,fixations] actions ::= [action] gazes ::= [gaze] gaze ::= [frame number, gmt seconds, gmt milliseconds, saccade flag, gaze pixel location X, gaze pixel location Y, velocity, acceleration, jerk] fixations = [fixation] fixation = [x,y,start,end]
Using XSL to Transform XML to SVG
Let me just take a moment to say that although XML and XSL were supposed to bring us some kind of unity with data with organs making reverential sounds in the background, I’ve never actually had XML live up to its promise the way it has with this project.
Before processing individual eye track points, I needed to collect some overall data about the scene (like duration) and establish some parameters to use over and over again (like colors). The xsl:variable tag proved invaluable here, and I was able to do basic math on the time stamps provided in the pickle data. Incidentally, since this isn’t an XSL tutorial, I’m going to skip some of the basics like the header line of an XSL file and talk instead about the more interesting parts. Feel free to refer to the actual files (available for download from the link at the beginning of the post) for all the gory details.
Here are some of the xsl:variable tags used at the beginning of the document:
You can see that XPath is sufficient to reference the Python pickled arrays by position (XPath is one-indexed, not zero-indexed). The START_MILLIS and END_MILLIS data is boldfaced in the data extract above. Note that although Python thought one of the data values was numeric and one was a string, it did not adversely affect us since XSL coerced the string into a number when it calculated the DURATION_MILLIS variable.
I establish the first xsl:template node that matches the pickled XML document root (/).
You’ve got to love the xsl:attribute tag that lets you dynamically add attributes to nodes so easily. OK, it could be easier if it had a select attribute, but what are you going to do? Come to think of it, why doesn’t it have a select attribute? If you’re assigning an attribute value, it’s not like you’re going to stick a whole XML node in there!
Let’s jump to where we want to start showing individual eye tracks, represented by faint blue circles in the final image.
Eye Tracks - Persistent
Nice and clean, we just call xsl:apply-templates and pass in the array of items representing the tracks. The Python Gnosis XML pickling is a little funny in that you end up with what seems like extra item tags, but it’s quite consistent and easy to figure out.
Further down we have the entire eye track template. Note how we match the same XPath statement that we’re passing.
We initially make the circles invisible by setting the display property to none. The timestamp associated with each track determines when we make the circle visible by setting the displayproperty to block. This is a CSS property, so none and block may be familiar to you if you’ve used CSS before. We use the SVG tag s:set to change a value at a point in time.
Similar to s:set is s:animate which makes changes to a value over time as opposed to instantly. The circle will fade in opacity from 100% (1) to 0% (0) over a period of 0.2 seconds. This has the effect that a circle appears brightly and then fades into the background. Because the circle’s opacity is set to 0.1 in the line
To help the eye follow the appearing circles, I also make a single circle that moves from point to point, and the SVG engine will interpolate the positions. See the source code for that one.
Check out the XSL in the source download above for complete details. Also there are more examples here: http://www.movesinstitute.org/~rharder/eye/.