Small steps. Big impact.

THE FITBIT BLOG

You talkin’ to me?

| 4 Comments

In our last blog post, we showed that you could read President Obama’s and Mr. Romney’s heart rate right off of their faces during their first presidential debate.  We got a lot of questions asking us to explain the technology behind it a little more, and we’re eager to show other ways it can be used.  It turns out that this works with all sorts of video, including our favorite Hollywood movies.  To demonstrate this technology to you a little further, today we’ll be looking at Robert De Niro during his seminal “You talkin’ to me?” scene in Taxi Driver (1976).

First, how does this work?  The basic idea is that the color of your face fluctuates slightly as blood perfusion under the skin changes from your heart’s pumping. If you measure the color change, you can measure heart rate. It sounds simple, but it’s not. One thing that makes this difficult is user motion: if the person being measured moves, it becomes difficult to disentangle cardiac-induced color changes from changes in lighting conditions on the face, rotations of the face, etc. But in some cases, we’re actually able to filter out some of this motion. Pretty cool, right?

Now let’s walk through the scene as a way to show how this technique works.  Fig. 1 (below) depicts the motion of Mr. De Niro during the performance. Big spikes indicate lots of motion and flat, low regions indicate periods of relative stillness. We see that the sequence where Mr. De Niro says,You make the move,” is the most still period. It’s about 15 seconds prior to “You talkin’ to me?”, but it’s a good starting point to try this.

Figure 1. Mr. De Niro undergoes significant motion in the “You talkin’ to me?” scene. The most still period is when he says “You make the move”, which precedes “You talkin’ to me?” by about 15 seconds

There’s still motion during this sequence, so to get rid of it, we’ll do a simple feature tracking on Mr. De Niro’s face, as shown in the movie below.

Now taking that patch of skin and analyzing its color changes, we can derive the cardiac signal for Mr. De Niro and get his heart rate.  Fig. 2 shows a nice sequence (below). In this chart, you can see the color intensity swing up and down in a visible pattern that corresponds to specific heart beats. The rest is just arithmetic to figure out the rate at which the heart is beating. Mr. De Niro’s heart rate for this scene: 95 bpm!  The average heart rate for adults falls more around 72 bpm.  Since Mr. De Niro was in his early 30’s at the time of the movie and seems to be in good shape, his heart rate was probably slightly elevated, which isn’t suprising since it was a pretty exciting moment in the movie where he pulls out a gun with a crazy mechanical contraption.  Or maybe Mr. De Niro did jumping jacks right before the scene?

Figure 2. Mr. De Niro's cardiac signal during the "You make the move" sequence. The overlaid hearts denote heart beats.

Now that we’ve cut our teeth on the easy part, let’s see if we can find Mr. De Niro’s heart rate during the epic, zeitgeist-defining moment of truth.  This movie shows the “You talkin’ to me?”  sequence with the feature tracker going to filter out motion.

Notice that there’s significant motion throughout, but there is one sweet spot when things are still.  Fig. 3 below shows the corresponding cardiac signal during that interval with beats overlaid.  The answer: 79 bpm!

Figure 3. Mr. De Niro's cardiac signal during the "You talkin' to me?" scene.

Figure 3. Mr. De Niro's cardiac signal during the "You talkin' to me?" scene. The overlaid hearts denote heart beats.

Did Robert De Niro know that he would change popular culture permanently with a single line?  …Ultimately, we don’t know.  What we do know is that his heart rate was a little elevated, so he was at least probably excited about his work.  Check out the scene in its entirety below.  Warning: there is some profanity in this clip!

There’s a lot that could be done with this technology in the long run, from tracking the heart rates of our nation’s leaders to tracking your friends’ or your own heart rate during a video chat.  It could be really interesting to see how heart rate changes when you’re interrupted by your boss or coworkers at the office.  Of course, some care still needs to be taken with video compression, optical and digital filtering, motion, and the like – all of which interferes with the cardiac signal. Nevertheless, it’s still a lot of fun to think about.

Finally, some caveats: this should not be taken as a rigorous scientific work, but more as a fun show-and-tell prototyping project.  There may be confounding factors in this analysis that have not been investigated in detail.  If you can think of them or have any additional thoughts or comments, we’d enjoy hearing them all.

4 Responses leave one →
  1. John Brinkman permalink
    October 24, 2012

    What a fascinating submission. I’m surprised that this has no comments on it. In fact, based on the overall amount of comments, this blog feels dead.

    Why is that Fitbit Inc? The Fitbit Ultra I have is phenomenal and I am in love with it. I’d be surprised if business wasn’t going well. So my conclusion seems that the blog is hard to find and isn’t well advertised.

    Either way I thought I’d share my thoughts. Been loving the product but now looking forward to future blog posts as well. :)

  2. November 23, 2012

    I have on my iPhone the cardiio app that tracks hearth rate exactly in the way you described.
    It would be very cool to have that feature integrated with Fitbit app or to have cardiio push data to my fitbit logs. Is that possible?

  3. December 27, 2012

    A VERY interesting post, especially in the wake of reading Crowley & Lodge’s Younger Next Year, which seems to indicate that everything is heart rate. Now, when can we expect a Fitbit wrist or chest strap that collects heart rate data and bluetooth’s it to the Fitbit?

  4. Linda Olson permalink
    July 30, 2013

    I Love my fitbit…love it…track all my steps and golfing up and down mountains. Still a shitty golfer…but I have fun. Anyway, we went out for our anniversary dinner at Saltys…and I wore a dress…no pockets. So I wanted to wear my fitbit so I hung it on to my underware…old lady style… The next morning I could not find my fitbit…looked everywhere…tore bed apart, looked under bed, couches, floors, etc. NO FITBIT. We went to Eastern WA with friends to golf…and came home 4 days latter. I am still looking for my fitbit because I love it. So, am washing clothes, and decided the bathroom off the kitchen needs to be cleaned. Take my supplies in there…look in the toilet…it needs to be cleaned..and I see something in there…I look again…and then again…and guess what…it is my FITBIT!!! I remove it, wipe it down, and press signal…and it says i did 25 steps before I went to bed!
    It still works…haven’t received a weekly fitbit note…but haven’t used it for a week or so..This baby was in the toilet for 5…yes 5 days and it is still working. Can I also tell you I had 2 martinis with dinner….And als0…I think I have sold at least 7 other fitbits to friends…and will keep selling…..This is better than a Timex…for those of you who remember….

Leave a Reply

Note: You can use basic XHTML in your comments. Your email address will never be published.

Subscribe to this comment feed via RSS