Unity and OpenCV – Part three: Passing detection data to Unity
We’ll now integrate OpenCV face detection into Unity. In this part, the camera stream and pixel processing will be done within OpenCV, and we will only send the location and size of the detected faces to Unity. This approach is used for applications which don’t need to overlay any visuals onto the camera stream, but only require the OpenCV data as a form of input.
Let’s start on the C++ side. First, add these files to the dependencies (as shown in the previous part):
opencv_core310.lib
opencv_highgui310.lib
opencv_objdetect310.lib
opencv_videoio310.lib
opencv_imgproc310.lib
Here’s the full Source.cpp which is used to track faces and send their location to Unity. I will not cover the actual OpenCV code – it’s mostly just sample code, and the scope of this tutorial is purely to introduce you to a way of making OpenCV and Unity communicate in an optimized way.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 |
#include "opencv2/objdetect.hpp" #include "opencv2/highgui.hpp" #include "opencv2/imgproc.hpp" #include <iostream> #include <stdio.h> using namespace std; using namespace cv; // Declare structure to be used to pass data from C++ to Mono. struct Circle { Circle(int x, int y, int radius) : X(x), Y(y), Radius(radius) {} int X, Y, Radius; }; CascadeClassifier _faceCascade; String _windowName = "Unity OpenCV Interop Sample"; VideoCapture _capture; int _scale = 1; extern "C" int __declspec(dllexport) __stdcall Init(int& outCameraWidth, int& outCameraHeight) { // Load LBP face cascade. if (!_faceCascade.load("lbpcascade_frontalface.xml")) return -1; // Open the stream. _capture.open(0); if (!_capture.isOpened()) return -2; outCameraWidth = _capture.get(CAP_PROP_FRAME_WIDTH); outCameraHeight = _capture.get(CAP_PROP_FRAME_HEIGHT); return 0; } extern "C" void __declspec(dllexport) __stdcall Close() { _capture.release(); } extern "C" void __declspec(dllexport) __stdcall SetScale(int scale) { _scale = scale; } extern "C" void __declspec(dllexport) __stdcall Detect(Circle* outFaces, int maxOutFacesCount, int& outDetectedFacesCount) { Mat frame; _capture >> frame; if (frame.empty()) return; std::vector<Rect> faces; // Convert the frame to grayscale for cascade detection. Mat grayscaleFrame; cvtColor(frame, grayscaleFrame, COLOR_BGR2GRAY); Mat resizedGray; // Scale down for better performance. resize(grayscaleFrame, resizedGray, Size(frame.cols / _scale, frame.rows / _scale)); equalizeHist(resizedGray, resizedGray); // Detect faces. _faceCascade.detectMultiScale(resizedGray, faces); // Draw faces. for (size_t i = 0; i < faces.size(); i++) { Point center(_scale * (faces[i].x + faces[i].width / 2), _scale * (faces[i].y + faces[i].height / 2)); ellipse(frame, center, Size(_scale * faces[i].width / 2, _scale * faces[i].height / 2), 0, 0, 360, Scalar(0, 0, 255), 4, 8, 0); // Send to application. outFaces[i] = Circle(faces[i].x, faces[i].y, faces[i].width / 2); outDetectedFacesCount++; if (outDetectedFacesCount == maxOutFacesCount) break; } // Display debug output. imshow(_windowName, frame); } |
We obviously start with a couple of imports and namespace using statements. Then, we declare a struct: this structure will be used to pass data directly from the unmanaged C++ code into the managed Unity scripts. This will be covered in more detail once we get to the Unity side of things. The structure is made to suit the application’s needs – you are free to change this as required.
1 2 3 4 5 |
struct Circle { Circle(int x, int y, int radius) : X(x), Y(y), Radius(radius) {} int X, Y, Radius; }; |
Next up, we have all the methods which can be called from within Unity. Because we are using C++, we need to explicitly tell the compiler how to expose these methods. Normally, the C++ compiler will mangle the method names when packaging them into a .dll. Therefore, we instruct it to use the classic “C” style of signatures, which leaves the method names just as you wrote them. You will always have to use this syntax when exposing C++ methods to a managed application.
1 |
extern "C" void __declspec(dllexport) __stdcall Detect(Circle* outFaces, int maxOutFacesCount, int& outDetectedFacesCount) |
Important to note here are the parameters Circle* outFaces and int& outDetectedFacesCount. The first one is a pointer to a Circle struct, indicating here that we are sending an array of Circles to Detect(). The latter indicates that outDetectedFacesCount is sent by reference.
Compile the project as x64 Release, and copy the resulting .dll to the Assets/Plugins folder in your Unity project. You will also need to copy the OpenCV .dlls to that same folder, as our own .dll depends on those. The OpenCV .dll’s were compiled in the first part, and can be found in \OpenCV 3.1\bin\Release.
It can be a bit tricky to know exactly which .dll’s you need – copying just the ones declared in the #include statements isn’t enough, as these in turn are dependent on other .dll’s. You can use Dependency Walker on our .dll to figure out exactly which .dll’s are required, or if you’re feeling a bit lazy, you can just copy all of the OpenCV .dll’s. If Unity tells you our .dll can’t be loaded even though it’s in the Plugins folder, it’s because dependencies are missing.
A final thing you will need to copy is the cascade classifier .xml. In this sample, I’m using the lbp frontal face cascade – lbp cascades are the significantly faster than haar cascades, though slightly less accurate. You will need to copy it from your OpenCV directory into the working directory of your Unity application – when you’re within the editor, this is the root project directory.
With all the files in place, we can get to the Unity scripts. Create a new script called OpenCVFaceDetection, and copy this underneath the generated class:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
// Define the functions which can be called from the .dll. internal static class OpenCVInterop { [DllImport("UnityOpenCVSample")] internal static extern int Init(ref int outCameraWidth, ref int outCameraHeight); [DllImport("UnityOpenCVSample")] internal static extern int Close(); [DllImport("UnityOpenCVSample")] internal static extern int SetScale(int downscale); [DllImport("UnityOpenCVSample")] internal unsafe static extern void Detect(CvCircle* outFaces, int maxOutFacesCount, ref int outDetectedFacesCount); } |
The static OpenCVInterop class exposes to C# all the C++ methods we just marked as dllexport. Note that the method signatures have to match. The DllImport attribute takes the file name of your dll.
Underneath that, add this structure declaration. It needs to have the same exact fields as the one declared in C++, in the same order, and it must be marked to have a sequential layout. This way we’ll be able to read the struct data coming from the unmanaged environment.
1 2 3 4 5 6 |
// Define the structure to be sequential and with the correct byte size (3 ints = 4 bytes * 3 = 12 bytes) [StructLayout(LayoutKind.Sequential, Size = 12)] public struct CvCircle { public int X, Y, Radius; } |
This is the class itself:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 |
public class OpenCVFaceDetection : MonoBehaviour { public static List<Vector2> NormalizedFacePositions { get; private set; } public static Vector2 CameraResolution; /// <summary> /// Downscale factor to speed up detection. /// </summary> private const int DetectionDownScale = 1; private bool _ready; private int _maxFaceDetectCount = 5; private CvCircle[] _faces; void Start() { int camWidth = 0, camHeight = 0; int result = OpenCVInterop.Init(ref camWidth, ref camHeight); if(result < 0) { if(result == -1) { Debug.LogWarningFormat("[{0}] Failed to find cascades definition.", GetType()); } else if(result == -2) { Debug.LogWarningFormat("[{0}] Failed to open camera stream.", GetType()); } return; } CameraResolution = new Vector2(camWidth, camHeight); _faces = new CvCircle[_maxFaceDetectCount]; NormalizedFacePositions = new List<Vector2>(); OpenCVInterop.SetScale(DetectionDownScale); _ready = true; } void OnApplicationQuit() { if(_ready) { OpenCVInterop.Close(); } } void Update() { if (!_ready) return; int detectedFaceCount = 0; unsafe { fixed(CvCircle* outFaces = _faces) { OpenCVInterop.Detect(outFaces, _maxFaceDetectCount, ref detectedFaceCount); } } NormalizedFacePositions.Clear(); for(int i = 0; i < detectedFaceCount; i++) { NormalizedFacePositions.Add(new Vector2((_faces[i].X * DetectionDownScale) / CameraResolution.x, 1f - ((_faces[i].Y * DetectionDownScale) / CameraResolution.y))); } } } |
The important bit happens in Update(): in an unsafe block, we call OpenCVInterop.Detect(), and pass the fixed pointer of an array of CvCircle. This means that the C++ OpenCV code will write the detected faces directly into this struct array we defined in C#, without the need for performance heavy copies from unmanaged space into managed space. This is a good trick to know for any C++ interop you may have to do in the future.
Because we don’t know how many faces will be detected, we create the array at a predefined size, and ask our C++ code to tell us how many faces were actually detected using a by ref integer. We also pass the array size to C++ to prevent buffer overflows.
In case you are not familiar with the two above keywords, unsafe simply allows you to use pointers in C#, and fixed tells the compiler that the given variable has to stay at its assigned position in memory, and is not allowed to be moved around by the garbage collector – otherwise the C++ code could inadvertently be writing to a different bit of memory entirely, corrupting the application.
This same procedure can be used to pass an array of pixels between OpenCV and Unity without having to copy it, allowing you to display video footage from OpenCV within Unity, or passing a WebcamTexture stream to OpenCV for processing. That is beyond the scope of this part, however.
[2020 edit] To use unsafe
, in recent Unity versions (starting from at least 2018) you can now tick Allow unsafe code in the player preferences under “Configuration”, and you can ignore the below info.
[Original post] In order to be able to use unsafe, we need to add a file called “mcs.rsp” to the root asset folder, and add the line “-unsafe” to it. (in versions before 5.5 you may need to use either smcs.rsp for .NET 2.0 subset, or gmcs.rsp for the full .NET 2.0). This file is an instruction to the compiler to allow unsafe code.
While this will let Unity compile your scripts, Visual Studio will still complain when you try to debug with an unsafe block – normally you add a flag in the project properties, but Visual Studio Tools for Unity blocks access to those. To be able to debug, you will have edit the .csproj (root project folder) manually, and set the two <AllowUnsafeBlocks>false</AllowUnsafeBlocks> lines to true. You will have to do this after every script change since the .csproj is recreated by Unity after every compile, so it’ll be useful to comment out the unsafe lines when you’re working on something else in the project.
In this sample, I’m sending the face viewport positions to a list, to be consumed by other scripts such as this one, which simply moves an object at that position:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
using UnityEngine; public class PositionAtFaceScreenSpace : MonoBehaviour { private float _camDistance; void Start() { _camDistance = Vector3.Distance(Camera.main.transform.position, transform.position); } void Update() { if (OpenCVFaceDetection.NormalizedFacePositions.Count == 0) return; transform.position = Camera.main.ViewportToWorldPoint(new Vector3(OpenCVFaceDetection.NormalizedFacePositions[0].x, OpenCVFaceDetection.NormalizedFacePositions[0].y, _camDistance)); } } |
That wraps up this part – hopefully I’ve taught you enough to set you on the path. Good luck!
[2020 Edit] Community member Iseta has set up a full Github repo with everything from this tutorial, which should help you get set up even faster!
Hi
Firstly, thanks for this detailed article.
I got an error in the PositionAtFaceScreenSpace script at the line
“if (OpenCVFaceDetection.NormalizedFacePositions.Count == 0)”
Error msg : Object reference not set to an instance of an object
I have placed both the scripts “PositionAtFaceScreenSpace ” and “OpenCVFaceDetection” as components of same gameobject. Any suggestions as to what is causing this error or how to solve it would be great.
Thanks again
Hi Kunal,
It seems that your NormalizedFacePositions list is not initialized before you try to access it. In OpenCVFaceDetection, NormalizedFacePositions is initialized in Start(), and it is first accessed in the Update of PositionAtFaceScreenSpace. Check where and when you are calling OpenCVFaceDetection.NormalizedFacePositions and make sure you’re not calling it before it is initialized (for instance, in another script’s Start). You can check the newly added Github repo with the full sample project.
Hi Thomas,
Yes you are right. The first error which is thrown is at line
“int result = OpenCVInterop.Init(ref camWidth, ref camHeight);” in Start() of OpenCVFaceDetection script.
Error msg : “DllNotFoundException: UnityOpenCVSample”
I checked the DLLs in plugins->x64 folder and didn’t find any DLL named “UnityOpenCVSample”. I also have iseta’s git project and it also didn’t have any such DLL. There was however a DLL named “OpenCVUnity.dll” [but using it also kept giving same error “DllNotFoundException: OpenCVUnity”].
What am i missing here? Any suggestions?
Thanks
In case you still have not found, You have to put your own .dll which you generated after building the .cpp Project.
Hi
Could you update this a bit to show how exactly I would be able to perform a function of opencv in Unity. Like, maybe add an empty gameobject and then add the scripts to the gameobject?
I assumed this would be obvious? The two Unity scripts I have provided run the code in Update(), which means they must be components on a gameobject for them to be called.
Hi Thomas Wonderful tutorial I must say though. Very precise about integrating openCV with unity as their are no errors. Unfortunately, because of not being very good at unity, I am still not able to run the file. I tried putting both the scripts on an empty gameobject, as well as on the main camera, but didn’t get any results. If would be great of you to help me out if you can. Also, can normal openCV code be used for creating apps using this method? And can I build these for android. Again, thanks a lot for providing such… Read more »
It’s hard to debug the problem remotely like this. Can I suggest you take a look through the forum thread and see if you find something there that helps you? https://forum.unity.com/threads/tutorial-using-c-opencv-within-unity.459434/
Hi! Got everything to work. Previously I had not completely understood all the steps (being a newbie) and had followed some steps incorrectly. I just want to say your posts were wonderful. There is just one more question that I have. You asked to compile the release version but I wasn’t able to do it as the release folder in the opencv directory doesn’t contain all the lib files like the debug folder does. I was able to get everything to work by doing all the processes with debug mode so works fine anyway, but would you have any idea… Read more »
Glad you got everything working after all! Did you actually build the release version of OpenCV as well? Make sure you follow step 9 of part 1 in its entirety.
Hi Thomas
So for some reason the release folder of openCv had no files. I decided to build everything with the debug files only. It worked, although following the same process for android build (exported .so from android studio) didn’t exactly work. Maybe it’s because of the unsafe code, I am not sure.
I have a video stream from an ESP32 CAM with the output to a URL. Can you provide an example of how I get the URL to a gameobject and use the data in Unity?
Hi Ron,
Sadly, I can’t help you with that. Maybe someone on the Unity forums can, though.
Hi Thomas,
Awesome tutorial, thank you so much!
Have you ever tried using Dlib in Unity? I’m struggling using it; I’ve been able to make it work directly in C++, but as soon as I export the DLL, Unity gets stuck on start. I was wondering if it is necessary to generate a DLL for Dlib to make it work in Unity, as when I use it in C++ I use a .lib instead of a .dll.
Hi Sergio,
Glad you enjoyed the tutorial! Unfortunately I’m not familiar with Dlib, so I’m afraid I can’t help you there.
Hi,
Thank you. Your demonstration helped me a lot. I had a question though.
In the cpp file for the dll you use “imshow(_windowName, frame);” at line 81. I don’t see any call to waitKey() after that. Usually, in OpenCV we call the waitKey() function to display the image. Why here we don’t need it anymore? Is there an exception if it is being used in a DLL? Thank you again for this post.
Hi Mukit,
In this case, we’re invoking the OpenCV function to query the camera from Unity, not from within the OpenCV script itself. What you are referring to is usually nested in a loop – notice that in this script, there is no loop. It’s Unity’s Update() that is asking OpenCV to query the camera each frame.
Hi,
Thanks for the tutorial! Everything works perfectly in the unity editor, but when I build the project, it crashes with a runtime error when loading the scene including the .dll import. Do you have any idea how to troubleshoot this?
Thanks!
Hi Kugelfisch, did you copy the .dll’s of OpenCV and your OpenCV project next to the executable of your Unity project?
Thank you for your reply! I actually figured it out eventually. I used some classifiers similar to what you did for the face detection, but the .xml files were not included in the Unity build, I had to manually copy them over!
Hi,
I have the following error :
[OpenCVFaceDetection] Failed to open camera stream.
UnityEngine.Debug:LogWarningFormat(String, Object[])
OpenCVFaceDetection:Start() (at Assets/Scripts/OpenCVFaceDetection.cs:56)
I have got no camera on my system currently .I believe that is the reason for this error. However from unity forum I got to know a bit about how this is to be run with webcam. Can you tell me if my understanding is right? And if so , I want to send the frames from the camera present in Unity(in future Hololens). Have you got any inputs for me regarding how I could do this?
Thanks for the tutorials!
Hi ! this solution can be used in a WebGL build ?? or just local
I never work with WebGL, so I wouldn’t know. I would be surprised if it works, though.
Hi Thomas. Thank you for the tutorial. I’ve taken the steps in the Part one (Install) and Part two (Project setup). When I deployed the code in Part three (Passing detection data to Unity) in visual studio (c++) the following massage popped up on my screen: “c:/project2/x64/debug/project2.dll is not a valid Win32 application” (project2 is the name of the code) and the code stopped working.
Could anyone give me help me dealing with the error?
Thanks