As this project marks the end of my academic journey toward earning a Master’s degree at Kingston University, my goal is to create something that showcases my years of experience in software engineering within the gaming industry. With that in mind, I’ve chosen to develop a digital pet experience that leverages the latest technologies.
Since the project is primarily aimed at Windows users (and potentially Mac users), one of the key innovations introduced is face recognition through the computer or laptop’s webcam.
The first challenge encountered was the limited timeframe, as the project spans only three months over the summer. Attempting to build the AI required for face detection and recognition within this period proved to be highly ambitious. Additionally, a significant obstacle was the lack of sufficient data needed to train the model effectively.
To address these limitations, a pre-made library called OpenCV for Unity—developed by Enox Software and available on the Unity Asset Store—was chosen for face detection (link). This library also proved highly effective in facilitating face recognition.
One of the biggest challenges in implementing face recognition was the initial behavior of the model: it successfully recognized the user at first, but it was later observed that it also identified different people as the same user. This issue stemmed from the model being trained only on "positive" examples, images of the user, without any "negative" examples to distinguish other faces. In order to reliably determine whether the person in front of the camera was indeed the user, the model needed exposure to a variety of different faces.
To source images of strangers, AI assistance,specifically ChatGPT, was used to recommend publicly available face databases. Based on the suggestions provided, A portion of the CelebA dataset (link) was selected for use in the project.
Since the process of training the model using the textures took several seconds, it was moved to an asynchronous method using Coroutines. To provide feedback to the user during this loading phase, a loading bar was added to display the current progress.
public IEnumerator TrainFromTexturesCoroutine(List<Texture2D> faceImages)
{
loadingBar.gameObject.SetActive(true);
loadingBar.value = 0;
float progress = 0f;
// STEP 1: Load model
if (!trainer.LoadModel())
{
Debug.LogError("Model not found");
yield break;
}
progress = 0.3f;
loadingBar.value = progress;
yield return null;
// STEP 2: Load face cascade
string cascadePath = Utils.getFilePath(LBP_CASCADE_FILENAME);
faceCascade = new CascadeClassifier(cascadePath);
if (faceCascade == null || faceCascade.empty())
{
Debug.LogError("Failed to load cascade classifier from " + cascadePath);
yield break;
}
progress = 0.6f;
loadingBar.value = progress;
yield return null;
// STEP 3: Open webcam
capture = new VideoCapture(0);
if (!capture.isOpened())
{
Debug.LogError("Cannot open webcam");
yield break;
}
progress = 1f;
loadingBar.value = progress;
frameMat = new Mat();
isCameraRunning = true;
yield return new WaitForSeconds(1);
loadingBar.gameObject.SetActive(false);
}