Small steps. Big impact.

THE FITBIT BLOG

You talkin’ to me?

| Leave a comment

In our last blog post, we showed that you could read President Obama’s and Mr. Romney’s heart rate right off of their faces during their first presidential debate.  We got a lot of questions asking us to explain the technology behind it a little more, and we’re eager to show other ways it can be used.  It turns out that this works with all sorts of video, including our favorite Hollywood movies.  To demonstrate this technology to you a little further, today we’ll be looking at Robert De Niro during his seminal “You talkin’ to me?” scene in Taxi Driver (1976).

First, how does this work?  The basic idea is that the color of your face fluctuates slightly as blood perfusion under the skin changes from your heart’s pumping. If you measure the color change, you can measure heart rate. It sounds simple, but it’s not. One thing that makes this difficult is user motion: if the person being measured moves, it becomes difficult to disentangle cardiac-induced color changes from changes in lighting conditions on the face, rotations of the face, etc. But in some cases, we’re actually able to filter out some of this motion. Pretty cool, right?

Now let’s walk through the scene as a way to show how this technique works.  Fig. 1 (below) depicts the motion of Mr. De Niro during the performance. Big spikes indicate lots of motion and flat, low regions indicate periods of relative stillness. We see that the sequence where Mr. De Niro says,You make the move,” is the most still period. It’s about 15 seconds prior to “You talkin’ to me?”, but it’s a good starting point to try this.

Figure 1. Mr. De Niro undergoes significant motion in the “You talkin’ to me?” scene. The most still period is when he says “You make the move”, which precedes “You talkin’ to me?” by about 15 seconds

There’s still motion during this sequence, so to get rid of it, we’ll do a simple feature tracking on Mr. De Niro’s face, as shown in the movie below.

Now taking that patch of skin and analyzing its color changes, we can derive the cardiac signal for Mr. De Niro and get his heart rate.  Fig. 2 shows a nice sequence (below). In this chart, you can see the color intensity swing up and down in a visible pattern that corresponds to specific heart beats. The rest is just arithmetic to figure out the rate at which the heart is beating. Mr. De Niro’s heart rate for this scene: 95 bpm!  The average heart rate for adults falls more around 72 bpm.  Since Mr. De Niro was in his early 30′s at the time of the movie and seems to be in good shape, his heart rate was probably slightly elevated, which isn’t suprising since it was a pretty exciting moment in the movie where he pulls out a gun with a crazy mechanical contraption.  Or maybe Mr. De Niro did jumping jacks right before the scene?

Figure 2. Mr. De Niro's cardiac signal during the "You make the move" sequence. The overlaid hearts denote heart beats.

Now that we’ve cut our teeth on the easy part, let’s see if we can find Mr. De Niro’s heart rate during the epic, zeitgeist-defining moment of truth.  This movie shows the “You talkin’ to me?”  sequence with the feature tracker going to filter out motion.

Notice that there’s significant motion throughout, but there is one sweet spot when things are still.  Fig. 3 below shows the corresponding cardiac signal during that interval with beats overlaid.  The answer: 79 bpm!

Figure 3. Mr. De Niro's cardiac signal during the "You talkin' to me?" scene.

Figure 3. Mr. De Niro's cardiac signal during the "You talkin' to me?" scene. The overlaid hearts denote heart beats.

Did Robert De Niro know that he would change popular culture permanently with a single line?  …Ultimately, we don’t know.  What we do know is that his heart rate was a little elevated, so he was at least probably excited about his work.  Check out the scene in its entirety below.  Warning: there is some profanity in this clip!

There’s a lot that could be done with this technology in the long run, from tracking the heart rates of our nation’s leaders to tracking your friends’ or your own heart rate during a video chat.  It could be really interesting to see how heart rate changes when you’re interrupted by your boss or coworkers at the office.  Of course, some care still needs to be taken with video compression, optical and digital filtering, motion, and the like – all of which interferes with the cardiac signal. Nevertheless, it’s still a lot of fun to think about.

Finally, some caveats: this should not be taken as a rigorous scientific work, but more as a fun show-and-tell prototyping project.  There may be confounding factors in this analysis that have not been investigated in detail.  If you can think of them or have any additional thoughts or comments, we’d enjoy hearing them all.

No comments yet

Leave a Reply

Note: You can use basic XHTML in your comments. Your email address will never be published.

Subscribe to this comment feed via RSS