See the YouTube version

Creating necessary assets, Media Player, Media Texture, and a Material

• From the content browser

• ➕ create a Media Player asset

  

• In the prompt that follows, check Video output MediaTexture asset

  

• Name the Media Player asset, and notice, 2 assets are created, the Media Player and an associated Media Texture

• ➕ create a material

Now that all our assets are created, lets open them up.

Setup the Media Player Asset

• open the Media Player asset

• click the tiny folder icon beside the empty Url input text box, and select Video-> My Camera Feed

  

• notice, now the camera feed is being projected through the Media Player viewport

Setup the Material, and Chroma Key node

• open the material

• setup the material graph with something similar to

  

• the important bits in the material are the Chroma_Key_Alpha node, and the Texture Sample node

  • the Texture Sample node should reference the Media Texture asset created earlier

  • wire up and parameterize most of the inputs into the Chroma_Key_Alpha node

  • throw in a one minus following the Chroma_Key_Alpha node

  • perhaps, Linear Interpolate / Lerp following the one minus with some debug color, into the Base Color material output pin, this will allow you to more easily see where the green screen is / and isn’t being masked fully

• perhaps, enable the material ‘realtime’ setting to more easily get feedback

  

• now, mess around with the settings until you get the red value just right
  

  

• and now, alas connect the opacity from the one minus and the red should just disappear
  

if you experience an overly contrasted image, where the subject appears extremely bright

Do not fret, too much. It may simply be that the default material exposure is setup with Auto Exposure causing overly exposed image (if the resulting output image is mostly white to begin with).

Now to resolve the issue,

• Click Lit

• From the Exposure section

  • disable Auto

  • and set the EV100 property to -1

And some result

May look a little better and normally exposed.

Reconnecting the feed when launching

Now, this is great and you can use this material in the editor and you should see your subject, masked from the background to where ever you apply the texture.

But, even while in the editor every time you launch PIE, you must set the Media Player’s feed Url again, and again to reconnect the feed.

Querying for capture devices in Native C++

In order to achieve this in C++ we must use the MediaAssets library.

Include MediaAssets Framework as a Dependency

In your MyGame.Build.cs, add an entry of MediaAssets to PrivateDependencyModuleNames.

Setup a UI that lists connected devices

The important bit here is pretty straight forward. The actual UI presentation I will leave up to you.

Here, i demonstrate how I was able to populate my game setting with connected camera devices.

TArray<FMediaCaptureDevice> Devices;
UMediaBlueprintFunctionLibrary::EnumerateVideoCaptureDevices(Devices);
for (const auto& Device : Devices)
{
  Setting->AddDynamicOption(Device.Url, Device.DisplayName);
}

When the setting changes, the value of this setting (or Device.Url) is stored in SharedSettings. It may be you want to store the value in LocalSettings instead since this value doesn’t need to be replicated across user’s devices.

Querying for capture devices in Blueprints

To create a UI in Blueprints, you may use the EnumerateVideoCaptureDevices blueprint node.

The incomplete example below just demonstrates querying the API, but does not setup UI. I will leave that to you to create.

Using Media Player OpenUrl and Play APIs

Now, when I launch my game and setup an actor that wants to use the Material that uses the live camera feed, I must do a couple things.

There are many ways in which you can achieve this, and I will capture the important bits.

• setup an actor blueprint

• add a variable named MediaPlayer of type MediaPlayer

  • set the variable to reference the MediaPlayer asset created earlier

• we must instruct the MediaPlayer to OpenUrl with the capture device URL that we stored earlier

• and finally, instruct theMediaPlayer to Play

Caution

• If your camera is offline, then OpenUrl may fail

• If your feed is already being used by another program, then instructing theMediaPlayer to Play may fail

If your camera is setup correctly, this should be enough to activate the feed, and your material should now be updating with camera’s feed

References

• https://dev.epicgames.com/documentation/unreal-engine/playing-live-video-captures-in-unreal-engine

• https://dev.epicgames.com/documentation/unreal-engine/BlueprintAPI/Media/Capture/EnumerateVideoCaptureDevices