As a creative developer, sooner rather than later you will require computer vision or image processing in your projects. It allows user interaction through cameras, for instance, but it can also be used to process your rendered frames directly should the need arise. OpenCV is a world class computer vision library, available free of charge – though if you found this tutorial, you are probably already aware of this. Whenever you Google Unity and OpenCV though, you are immediately greeted with a (relatively expensive) paid asset, which I personally find… well, let’s not go into what I think about asking money for the work others provided for free.
Needless to say, in this set of tutorials I will show how to integrate C++ OpenCV with Unity, as the native version will always be the fastest and the most memory efficient. Don’t let the C++ bit scare you off – OpenCV is set up to be very high level, so don’t go thinking you require a C# (or even Java) wrapper. If you’ve experimented with the most prominent C# wrapper, EmguCV, and Unity before, you will know that the performance leaves something to be desired.
Before we get any work done in Unity though, we need to set up OpenCV itself. If you already have all the .dll’s and .lib’s ready to go, skip on ahead to the next part. If you don’t, and especially if you’ve never set up OpenCV before, read on – I remember getting started to be quite tricky and not very well documented, so I’d like to make it as straight forward as possible for my readers. I’m on Windows, and I don’t have any other OS’es laying around, so that’s all the help I can offer.
The C++ libraries for OpenCV aren’t shipped as direct downloads, but rather need to be built by you. This allows users to more easily manage the vast number of included libraries (and significantly decreases file size), though it makes getting started with the basics a bit more daunting. I’ll walk you through it – you’ll be ready to go in about 15 minutes.
- Download the current release from opencv.org. I will be using OpenCV 3.1 in these tutorials, and I suggest you do the same (at least until this tutorial grows horribly outdated). [2019 edit: OpenCV 4 still works just fine with this tutorial]
- Run the .exe and extract the files to a temporary location – you won’t need these anymore soon enough. I will be extracing them to G:\Temp\opencv unpack.
- Download and install CMake. CMake is a tool for building packages, widely used in the open source C++ world, and it’s required to build OpenCV.
- Run CMake. At the top, click “Browse Source”, and navigate to the directory where you unpacked OpenCV in step 2. Under that directory, select /opencv/sources as the source folder for CMake. In my case, that would be G:\Temp\opencv unpack\opencv\sources.
- Select a new temporary location as build directory for CMake. I chose G:\Temp\OpenCVInstall.
- With the two paths set up, go ahead and click “Configure” at the bottom. Once CMake is done, you will see a large number of red items appear in the center area of the application. These are all the build options I mentioned earlier – right now it will suffice to simply stick to the defaults. Remember that this is where you’ll have to look if certain features are not included, like OpenNI support for the Kinect, for instance.
- Click “Generate”, select the Visual Studio version you’re using (most likely Visual Studio 14 2015 Win64) and let CMake do its thing. If you’re using MonoDevelop, then maybe now is the time to evaluate certain life choices.
- Now navigate to the build directory you set up in step 5, and open OpenCV.sln. If you’re using Visual Studio 2015, chances are that you will not be able to open the solution. This is because Visual Studio 2015 no longer includes C++ support by default. Go to File > New > Project > C++, and you will be prompted to install the required components.
- Make sure the INSTALL project is set as the active project in the Solution Explorer, then Build (F6) both the Debug and Release x64 version of OpenCV. You should give Visual Studio some time to initialize the projects and scan all #included files before doing so – wait for the “Ready” status at the bottom of the window.
The builds will take a couple of minutes. Now would be the ideal time to take a breather – stretch your legs and your eyes, grab a coffee, or have a chat at the water cooler. Once the builds are done, we can finish setting everything up.
- Create a new directory, which will serve as the permanent home for your OpenCV files. I chose G:\OpenCV\OpenCV 3.1
- In the build directory from step 5, you will now have a bin and a lib folder. Copy both to this new folder.
- In the original unpack folder, navigate to opencv/build (or opencv/install), and copy the include folder to your final OpenCV directory. You should now have G:\OpenCV\OpenCV 3.1\bin, G:\OpenCV\OpenCV 3.1\lib and G:\OpenCV\OpenCV 3.1\include, or at least your equivalent of the same.
- You can go ahead and delete all the temporary files now, as we have everything we need.
- Now it’s time to add the OpenCV directory to Window’s Path system variable – this just serves to tell the OS where to look for dll’s, so you don’t have to copy the dll’s to every OpenCV project you make. Go to the Control panel > System > Advanced system settings and click Environment Variables….
- Under “System variables”, add a new entry called OPENCV_DIR, and assign the path defined in step 10.
- Finally, in the User variables section above the system variables, find the PATH entry, and click edit. At the end, add %OPENCV_DIR%\bin.
That’s it for now! In the next part, we’ll set up an OpenCV project that will be usable from within Unity.