Matthias

Do you want to make your own games with lots of cutscenes like in Command & Conquer, Wing Commander, or the newer Roundabout? In this introduction to the Unity Video Player you’ll learn how to do that, and how to implement videos in a variety of ways of your own into your game. Unity comes with its own Video Player component, which means adding videos and creating games around it is not at all complicated.

You will learn how to:

Start and stop full-screen cutscenes.
Display videos on surfaces inside the game-world.
Add videos from YouTube and other video-sites.
Create 360°-videos for your Virtual Reality app.

Getting Started

You will need the newest version of Unity, and the starter-files which you can download here. This tutorial will build on them. In that download you can also find the finished project!

You will also need to know the basics of Unity, and have already created something with it.

Load the starter-project, Land of Full Motion Video Starter, and take a look at the folder-structure inside the Assets folder.

Here is what each folder contains:

Fonts: The font used for the in-game texts.
Materials: The materials used in the world, including grass and trees.
Models: The 3d-models used in this game.
Prefabs: The used objects as prefabs.
Scenes: The various scenes used in this tutorial.
Scripts: Several pre-made scripts, some of which you will extend.
Shader: A special occluded 3d-text shader.
Sounds: The sound for pressing a button.
Standard Assets: The first-person controller used in this game.
Textures: The grass and metal textures used in the game.
Videos: The two example videos provided with this project.

The Land of Full Motion Video

Now open up the scene 01FullScreenCutscene. It should look something like this:

Welcome to the magical Land of Full Motion Video, where 3d-environments intersect with real-life video-recordings in completely plausible ways.

Right now you can see a small environment you can explore in first-person, and a console with buttons on it. Try it out! Walk around using the wasd-keys and try to press the buttons by pointing at them and left-clicking.

Nothing will happen though. That’s what you’ll fix next. :)

First you will need to open the VideoController.cs file, which barely has anything in it! Of course nothing is playing.

Your First Cutscene

Start by adding these two methods to the VideoController class:

//1
public void PlayFullScreenOfflineVideo()
{
    StartFullScreenVideo(videoFilePath);
}

private void StartFullScreenVideo(string path)
{
    //2
    if (fullScreenVideoPlayer)
    {
        Destroy(fullScreenVideoPlayer);
    }
    
    //3
    fullScreenVideoPlayer =
        Camera.main.gameObject.AddComponent();

    //4
    firstPersonInteractor.SetImmobile(true);

    //5
    fullScreenVideoPlayer.playOnAwake = false;
    fullScreenVideoPlayer.renderMode =
        UnityEngine.Video.VideoRenderMode.CameraNearPlane;
    fullScreenVideoPlayer.targetCameraAlpha = 1F;
    fullScreenVideoPlayer.url = path;
    fullScreenVideoPlayer.frame = 0;
    fullScreenVideoPlayer.isLooping = false;

    //6
    fullScreenVideoPlayer.Play();
}

Here is what happens in the code of PlayFullScreenOfflineVideo():

PlayFullScreenOfflineVideo executes when you press the button. It then calls StartFullScreenVideo() with a saved filepath as parameter.
The method checks if a Video Player component already exists, and if so destroys it. As a new Video Player instantiates every button press you should be sure to removed old ones.
The main camera receives the Video Player component. In this part a full-screen-video will show, and you’ll need the main camera to do that.
The player-controller is set to not move. This is important so that you cannot move around during a playing cutscene and change the environment. This is a simple game, where nothing except the player actually moves. In a more complex one you’d have to be sure to pause and/or deactivate everything that could affect a change on the gameworld during a fullscreen-video.
Here the various settings are being set. playOnAwake = false; makes sure the video will not start immediately. The renderMode is set to cameraNearPlane for fullscreen-videos. The isLooping variable sets if the video should loop endlessly. frame sets the start-frame of the video. With a larger number it would start at that point (although the number should be within the actual length of the videoclip). targetCameraAlpha sets how “transparent” the fullScreen-video is going to be. 1F means it will not fully visible, 0.5F would be 50% visible.
The video begins playing.

Enabling User Interaction

But you are not done yet, as these functions will not call themselves.

Open the WorldUIElement.cs file. It is assigned to the buttons you see in the level, and can be set to be a certain type of button (in this case ButtonTypes.playFullScreenOfflineVideo). In order to call the VideoController class and start the video add these lines to the start of Interact().

if (buttonType == ButtonTypes.playFullScreenOfflineVideo)
{
    videoController.PlayFullScreenOfflineVideo();
}

The entire function should look like this:

public void Interact()
{
    if (!isActivated)
    {
        if (buttonType == ButtonTypes.playFullScreenOfflineVideo)
        {
            videoController.PlayFullScreenOfflineVideo();
        }

        isActivated = true;
        displayMesh.transform.localPosition = buttonPressedPosition;
        audioSource.PlayOneShot(buttonSound);
        thisRenderer.material = materialNormal;
        StartCoroutine(DeactivateButton());
    }
}

The important part is that now that PlayFullScreenOfflineVideo() is called, and a video will play when you press the Offline Video button on the console! The rest of the code handles the button animating, and does not need to be modified by you.

Try it out and press the Offline Video button! The result should look something like this for about six seconds:

Exiting the Cutscene

You cannot quit the cutscene yet. To do that you’ll need this code inside Update() of the VideoController class:

void Update()
{
    //1
    if (fullScreenVideoPlayer && fullScreenVideoPlayer.isPlaying)
    {
        //2
        if (Input.GetKey(KeyCode.Escape))
        {
            //3
            fullScreenVideoPlayer.Stop();
            firstPersonInteractor.SetImmobile(false);
        }
    }
}

Here is what the code does:

This checks if a full-screen video is currently playing.
This code executes when you press the escape-key.
This will stop the video and make your character be able to move again.

Once the video has finished press the escape-key to quit the video-view.

Well done, you created your very own first full-screen cutscene! Now you can get some actors together and create your own Command & Conquer-esque masterpiece. ;]

Note: When using a local file you need to adapt its filepath to the current save-location. The editor-filepath will no longer work when deployed. Luckily you can solve this by prefixing Application.dataPath to the filepath. This has already been happening in Start():

private string videoFilePath = "/RW/Videos/ForestFlyover.webm";
void Start()
{
    videoFilePath = Application.dataPath + videoFilePath;
}

Adding the Video Player Component Manually

After watching the video pause the gameplay and take a look at the main camera in the inspector. This is what you should see:

Here are all the variables that you set via code. It is also possible to manually add the Video Player component by selecting the needed GameObject, and then selecting Component ▸ Video ▸ Video Player. Afterwards only a call to the Video Player and Play() is necessary to start a video.

Note: There are some limitations on what kind of files you can use. Depending on the system you use Unity’s file import will not work for certain formats. More information and details can are available here. For example a wmv-file will only work on windows, and no other platform. Unity can import the webm-format on the other hand without issue on any platoform.

The next thing to keep in mind is that not every video file (even if the import works) will run on every mobile device, as only certain codecs run on Android devices or iPhones. More details on that are on this page.

Be sure to chose a format and codec that works on the platforms you intend to deploy too!

Playing Online Videos

Your astute eyes probably noticed the second, still not functional button labelled Online Video.

Right now the Video Player uses a locally saved file, which has been hard-coded. Alternatively you can also use a hyperlink and use an online source! To make it work add this method to the VideoController class:

public void PlayFullScreenOnlineVideo()
{
    StartFullScreenVideo(videoURL);
}

What this does is call StartFullScreenVideo, which you used before, but with the video-url as parameter.

You will also need to adapt Interact() in the WorldUIElement class with this code:

if (buttonType == ButtonTypes.playFullScreenOnlineVideo)
{
    videoController.PlayFullScreenOnlineVideo();
}

When you press the Online Video button inside the game this will call the VideoController, and PlayFullScreenOnlineVideo(), which you just added.

The video will play the same, try it out!

Setting Your own Online Source

Right now an online video has already been selected for you to use:

private string videoURL = "https://www.videvo.net/videvo_files/converted/2016_01/preview/Forest_15_3b_Videvo.mov47209.webm";

You can find this variable and swap out the link for your own favorite YouTube-video, and that video (assuming you have a working internet-connection) will play once you hit the Online Video button in the gameworld.

This offers lots of possibilities in regards to game-design, where you could possibly create a game which has a new video every time, or you could host your tutorial-video on the web, and then change the video without having to update the game itself.

Creating a Fancy & Interactive In-Game Video Screen

Open the scene 02VideoPlayer. it is very similar to the first one, only that this one has a an actual video-screen inside the world, and new buttons.

Like before the buttons don’t do anything right now. In order to have a video displayed on the big screen first add this variable to the VideoController class:

public VideoPlayer displayVideoPlayer;

This is where the script will access the Video Player. Next assign the DisplayMesh of the VideoDisplayScreen object to this variable. If you look the the DisplayMesh in the inspector you’ll see that it already has a Video Player component attached to it.

Next add this method to the VideoController:

public void PlayInWorldVideo()
{
    //1
    if (!displayVideoIsPaused) {
        //2
        displayVideoPlayer.playOnAwake = false;
        displayVideoPlayer.renderMode =
            UnityEngine.Video.VideoRenderMode.MaterialOverride;
        displayVideoPlayer.url = videoFilePath;
        displayVideoPlayer.frame = 0;
        displayVideoPlayer.isLooping = true;
    }

    //3
    displayVideoPlayer.Play();
    //4
    displayVideoIsPaused = false;
}

As before you will need this check in Interact() in the WorldUIElement class:

if (buttonType == ButtonTypes.playInWorldVideo)
{
    videoController.PlayInWorldVideo();
}

This is what happens when the code runs:

It checks if the video is currently already running and paused, or not running at all. If the video has not started yet the Video Player settings are set again, as in the previous example.
The most important setting is displayVideoPlayer.renderMode = UnityEngine.Video.VideoRenderMode.MaterialOverride;, which makes the video play on the object it’s on, and not in full-screen.
The video starts.
The displayVideoIsPaused variable is set to false. This will be more important for using the pause-feature.

Try it out! When you press the Play button on the console the video will now run on the screen.

Adding Pause and Stop Buttons

Next up is the Pause Button, which will pause the video, if it is playing. Add this method to the script:

public void PauseInWorldVideo()
{
    //1
    if (displayVideoIsPaused)
    {
        //2
        displayVideoPlayer.Play();
        displayVideoIsPaused = false;
    }
    else
    {
        //3
        displayVideoPlayer.Pause();
        displayVideoIsPaused = true;
    }
}

The script checks if the video is currently paused.
If yes, the video continues, and the displayVideoIsPaused variable is set to false. This affects the usage of the Play and Pause buttons.
If no, the video is paused, and the displayVideoIsPaused variable is set to true.

Next add this method to stop video-playback:

public void StopInWorldVideo()
{
    //1
    displayVideoPlayer.Stop();
    //2
    displayVideoIsPaused = false;
}

Adapt the Interact method of the WorldUIElement class with these checks to make it all work again:

if (buttonType == ButtonTypes.pauseInWorldVideo)
{
    videoController.PauseInWorldVideo();
}
else if (buttonType == ButtonTypes.stopInWorldVideo)
{
    videoController.StopInWorldVideo();
}

The Interact method should have all five checks now. You can also turn these into a single if-else block, instead of multiple ones:

if (buttonType == ButtonTypes.playFullScreenOnlineVideo)
{
    videoController.PlayFullScreenOnlineVideo();
}
else if (buttonType == ButtonTypes.playFullScreenOfflineVideo)
{
    videoController.PlayFullScreenOfflineVideo();
}
else if (buttonType == ButtonTypes.playInWorldVideo)
{
    videoController.PlayInWorldVideo();
}
else if (buttonType == ButtonTypes.pauseInWorldVideo)
{
    videoController.PauseInWorldVideo();
}
else if (buttonType == ButtonTypes.stopInWorldVideo)
{
    videoController.StopInWorldVideo();
}

Here is what happens in the new functions:

The video stops. This is different from pausing as it cannot continue from its current position.
The displayVideoIsPaused variable is set to false.

Now you have a simple and working in-game video player! You can start, pause, and stop the playback of videos. But you can go one step further.

Adding a Timer

See the text-display over the video-screen?

This is where you can add a timer-display. Start by adding this variable to the scrip, which will hold the 3d text mesh:

public TextMesh timeDisplay;

And assign the TimeDisplay object from the hierarchy onto it.

Then add DisplayTime:

private void DisplayTime()
{
    //1
    string minutes = Mathf.Floor ((int)displayVideoPlayer.time / 60).ToString ("00");
    string seconds = ((int)displayVideoPlayer.time % 60).ToString ("00");
    string lengthMinutes = Mathf.Floor ((int)displayVideoPlayer.clip.length / 60).
        ToString ("00");
    string lengthSeconds = ((int)displayVideoPlayer.clip.length % 60).
        ToString ("00");
    //2
    timeDisplay.text = minutes + ":" + seconds + " / " + lengthMinutes + ":" +
        lengthSeconds;
}

This is what each part does:

This calculates the elapsed time of the video into minutes and seconds.
This assign the minute and second values to the text in a minutes:seconds format.

Also add this code to Update():

//1
if (displayVideoPlayer && timeDisplay)
{
    DisplayTime();
}

This makes sure DisplayTime() only executes when both an in-game video display object and a time display exist.

Try it out! Your video-screen will now display the elapsed time.

Importing Videos with Transparency into Unity

You can also use videos with a built-in alpha channel! A lot of stock-video sites like Videvo offer these. Sadly only certain video-types and codecs are supported. On Mac you use .mov files, and on both platforms .webm. The full details can be read on the page for Video Transparency in the Unity Reference Manual.

Note: It’s possible to transform videos with transparency that aren’t .mov or .webm files, but it can be a bit tricky, but here is the highly abbreviated version:

First you need the tool FFmpeg. Navigate to the folder where the file ffmpeg is stored (ffmpeg ▸ bin on Windows, on Mac you just get the file without any others) in you command-line-terminal. Also make sure the file you want to transform is in there too. Then run this command: ./ffmpeg -i ORIGINAL_FILE_NAME.mov -c:v libvpx -pix_fmt yuva420p -auto-alt-ref 0 NEW_FILE_NAME.webm.

This should create a new file with the necessary encoding.

Luckily there is already a video with transparency available in your project-files, and it’s called ColorHeartsTrasparent.

Correctly Tweaking Everything so that Transparency Actually Works

In addition to the Video Player you also need a custom Render Texture on the object that will be made transparent. Take the TransparentVideo material and assign it to the DisplayMesh object.

Then set the videoFilePath in the VideoController from

private string videoFilePath = "/RW/Videos/ForestFlyover.webm";

private string videoFilePath = "/RW/Videos/ColorHeartsTransparent.webm";

so that the correct (transparent) video will play, and not the previous Forest Flyover.

This is how the Video Player should look like in the inspector:

And if you try it out it should look like this! :)

Note: There are also some videos with a green background that could be potentially be chroma-keyed out, like the wonderful, free-to-use Van Damme Action Video Set.

That way you could replace the green parts with transparency. There are some advanced shaders out there on the web, but they are a bit tricky to use. If you find a good solution post it in the comments below!

Displaying a 360° Virtual Reality Video

Another cool thing you can do is implement 360°-videos in your games. A 360°-video is a video created with a special camera (often a series of cameras attached to each other to record an entire sphere of action). You can then watch it by turning your VR-headset (or even your cellphone!) to see all different parts of it. You possibly have seen these already on YouTube and Facebook, which already integrate 360°-videos. If you haven’t yet check out the VR category on Youtube!

The Inverted Sphere

These videos play like the in-game videos you saw before, but on an inverted sphere instead of a flat plane. You can imagine it like a regular ball that you can already create in Unity by clicking Game Object ▸ 3D Object ▸ Sphere, but with the outside facing inwards. If you now add a Video Player and a 360°-video it will get displayed on it, and if your point of view is inside it will be as if you are inside it!

Luckily there is already an inverted sphere present in the last scene, 03SphericalVideo. Open it and check it out!

If you press play you can see the sphere from the inside, without a video playing. You cannot move outside the sphere, something that is important to not break immersion in this scenario.

If you look the sphere from inside the editor you can see that the surface is inside, i.e. you can see the sphere from being within it, but not the outside. The same thing happens when you glitch into a wall in a game and you cannot see the wall you are in from the backside.

Adding the Virtual Reality Video

In order to show a video on the sphere create a new script and name it EnvironmentalVideo.cs. Add it to the inverted sphere, and add this code to it:

public class EnvironmentalVideo : MonoBehaviour
{
    //1
    public GameObject displayMesh;
    //2
    public string videoFilePath = "/RW/Videos/HundraKnektarsMarsch.webm";

    void Start()
    {
        //3
        videoFilePath = Application.dataPath + videoFilePath;

        //4
        var videoPlayer = displayMesh.AddComponent();
        videoPlayer.url = videoFilePath;

        videoPlayer.isLooping = true;
        videoPlayer.renderMode = UnityEngine.Video.VideoRenderMode.MaterialOverride;
        videoPlayer.targetMaterialRenderer = GetComponent();
        videoPlayer.frame = 0;

        //5
        videoPlayer.Play();
    }
}

This is the 3d-mesh the video will play on.
Here is the local filepath for the 360° video in relation to the assets
Filepath-adaptation with Application.dataPath so that it also works in a deployed build
Setting the Video Player preferences. In this case you want the video to loop.
The video starts.

Then do not forget to assign the sphere-mesh to the displayMesh variable.

Note: Sadly implementing VR-videos directly from YouTube does not work, which is why you need to use a local file here. If you can find a good 360° file you can still implement it. The stock video site Videvo and Wikipedia have a few good ones you can experiment with!

Try it out! If you press play you should be inside the video-sphere.

Note: You can create you own Virtual-Reality game or app by looking into the Google Cardboard SDK for Unity, which comes with its own premade first-person VR-Controller! If you add it to this scene and deploy it to your phone (or even better if you have your own inexpensive Google Cardboard VR Headset) you can point and look in any direction.

Note: It is technically possible to sync events in your game to the framerate of a video, but this can have unforeseen repercussions. Synchronized events that work with regular videos might possibly not work in 360°-videos.

Enjoyment in Virtual Reality especially is tied to a smooth framerate, otherwise lags and skips will destroy the presence of the player, and cause VR-sickness. In these cases make sure beforehand that synchronization would work.

Where to Go From Here

You can download the final project here, or at the top of this tutorial.

And you learned how to implement videos into Unity, nice work!

There are multiple ways to now use these features in a game. You could go the established way and use it for cutscenes and other videos, like credits. Or you could implement in-game video-players, which display what you recorded before. If you implement video-recording you can show the last ten seconds in an instant-replay, which is perfect for local multiplayer games like Worms.

-Matthias

Video Sources

Flying Over Forest 3 (Royalty Free Licence, Videvo)
Hundra knektars marsch på Forum Vulgaris (Creative Commons Attribution-Share Alike 4.0 International Licence, Jan Ainali)

Matthias Zarzecki

iOS Development

Author Archives: Matthias

Kingfisher in Swiftui

How to join an Array by Language in Swift

Backen.de Voiceover

How to make a Skeleton Loading View in SwiftUI

The Matthias iOS Development Show Episode 2

The Matthias iOS Development Show

My pics are in a Boardgame Magazine

Cat Bot is no more

Auto-Formatting C# in VSCode for Unity 3D

How to use the Video Player in Unity