This is Peter Rubin, flying solo this time

for a very special episode of Game Life.

I'm here on Microsoft's campus, we've got an exclusive look

at something you might have heard a little something about,

but we're the only ones with the details,

so let's take a look.

(upbeat music)

Why do I look like a clay man right now?

Here to that question, please introduce yourself, sir.

Hey, I'm Scott Evans.

I'm the Group Project Manager for Kinect at Microsoft.

Excellent.

So, Scott, this is what?

This is just a 3D depth map

of the new Kinect sensor, correct?

Yeah, you're seeing what the sensor itself sees.

You're seeing yourself represented here in three dimensions.

The 3D vision itself, which is what made Kinect famous,

we made high definition, and high fidelity.

If you step a little closer even,

you can see the buttons in your shirt.

You can see the buttons, you can see--

This is little minute changes in depth

that we can see with the sensor.

Now, Kinect is more than just a 3D vision camera.

It also has a 2D vision camera,

and you can see that we've got a 1080P high-definition,

color feed across that whole field of view.

But one of the challenges with the color feed

is it's very susceptible to changes in room lighting.

We've introduced a new technology we call Active IR,

and the magic here is really in how it filters out

room lighting.

A good way to show that is we can make,

we can lower the lights in this room.

We can close the blinds, and you'll see that,

that color image becomes unusable, but this active IR image

is completely unchanged.

I can't see any differences from when it was light

in the room, and when it was totally dark in the room.

Another fun way to show this technology at work,

we'll use this little flashlight here,

and you can simulate a lamp, for example, on the side.

We'll go back to color and you can imagine

if the lamp was sitting next to you on the end table,

completely washing out only half of your face.

If we go back to active IR, we can see that light

is completely removed from the scene.

Let's bring the lights back up.

All of this, I should just point out,

this is just all what the Kinect sensor sees and hears,

before it gets worked into an experience for you

at home with the Xbox.

Exactly.

Now, the first capability is skeletal tracking.

Okay.

This was one of the cortex that we introduced

with Kinect for 360.

You see we've taken skeletal tracking,

and leveraged that 3D vision,

and made it high definition as well.

We got more joints, more expressive movement.

You can see the lean in your spine.

You can shrug your shoulders.

All that movement is being represented

here in your skeleton.

We even have enough fidelity to show your hand joints.

The tips of your fingers, your thumbs.

It pinches.

A gun.

It knows if I'm pointing a gun at the screen.

Yup.

What if I want to go menace to society,

and go from the side?

Got it, got it, boop!

Now, on top of skeletal tracking, we have enough fidelity

to even look at how your joints are rotating.

I'm going to turn on rotation.

This is a representation we call box man.

No more cheating in dance central.

It's going to know exactly what you're doing.

Even your head, if you rotate your head

and make little circles there,

we can see that represented as well,

so we can see exactly where you're looking in the room.

My yoga game is going to be on point.

We can combine a physics model

to start to understand muscles and forces that are exerted

through the body.

We're going to turn that on here.

We've got a representation here of you,

and what you're seeing is, as the skeleton is moving

from green to red, it's showing where the force

is being applied.

As your shifting your weight,

and you're seeing the weight shift on one foot.

If you jump up in the air, while you're in mid air,

you'll be all green because there's no force,

and as you land you see your feet turn red.

We can even see force applied through movement.

If you can throw some punches here at the sensor,

we see these white circles.

If you put you back into it, however,

we can see a much bigger circle.

Yeah, but how about a fireball?

(yells)

Ahhh!

It's on now.

Street fighter, we're coming.

On top of muscle and force, from a fitness perspective,

we can also start to estimate your heart rate.

We're going to bring that up.

We use the sensor feed.

We can look for patches of your skin and look at

micro-fluctuations in the blood underneath your skin,

and estimate your heart rate from that.

Finally, we start to look at the people in the room

and start to estimate moods and how they are feeling,

and how they are engaging with the system

through what we call our expressions platform.

We'll bring that up.

Here you can see that we zoomed in your face,

and we can show if your neutral or smiling.

We can see if your looking at the system

verses looking away, and maybe you're reading a newspaper,

or playing with your smartphones.

Now, did you actually use me as a reference

for this expression guy, because that's uncanny.

It is very close resemblance.

That's our generic representation of what's going on.

I'm going to jump in the scene with you.

I'm going to grab a controller.

Now we got two people.

You're controller one.

I'm controller one, you're controller two.

We can change controllers.

You can see that, it knows now that we swapped controllers.

And it signed you in.

It signed me in automatically using,

because I've used this Xbox before.

You didn't have to do the whole thing

where you wave and then it sees you,

and you have to sign in and do things.

It just recognized me, and signed me in.

We can even switch positions here,

and you can see that not only did the controllers follow us,

but we swapped spaces on the screen.

You can imagine a multiplayer game now,

it doesn't matter where I'm sitting in a couch,

I can jump in without any user interface.

Without any sign in, and the screen can split

relative to where I'm sitting in the room.

We've got orientation, we've got muscle force,

we've got heart rate, we've got expression engagement,

we've got dark, but what if we tie this all together,

and do five people in the dark?

We can do six, but we'll do five here to put them in.

Okay, alright we'll slow it down for you, then.

I'll bring up skeletal tracking again,

let's bring some friends in.

Yeah let's cut the lights.

Alright, totally dark.

Totally dark.

Alright guys, come on in.

I'll switch over to Active IR

so we can see in the dark here.

Bring in more people in here.

So, now look at that.

Let's jump.

One, two, three.

Alright.

Simon says touch your head.

Crane technique.

You have to do the harlem shake.

Nah, I'm just playing.

Alright, we got it.

Wired.