Tag Archives: Image Processing

Image Processing : Acquisition with OpenCV Part II

This tutorial will explain on how to grab frames from IP Camera and Kinect or Xtion Pro Live.

Read IP Camera

Prepare a router and an IP camera.  See how to configure your IP camera to connect to your already-connected internet-router via wireless and access that IP camera via wireless public IP here. 

To access IP camera using OpenCV we’ll need to compile OpenCV with ffmpeg codec support because we will stream mjpg type data. The code similar with opening webcam but with different address. Remember to delete ?mute in <ipcameraaddress>/video.mjpg?mute because OpenCV only accept video streaming.

int main(int argc, char** argv)

{

VideoCapture ipcam;
Mat frame;
bool status;

// the address format is 
// http://<user>:<password>@<ipaddress>:<port>/video.mjpg
// example with my ip camera connected to internet via EVDO-router
// http://admin:admin@10.241.126.66:3333/video.mjpg
// 
// Find your ip camera address : 
// 1. Open your ip camera on chrome, 
// 2. Right click on the streaming video – select inspect element. 
// 3. There your  address revealed 
// 4. You need to add your username and password to access ip camera
// 5. The format is just like the given format above

ipcam.open(“http://admin:admin@10.241.126.66:3333/video.mjpg“); 
if (ipcam.isOpened()) 
{
   int keyCode = 1;
   // we can choose the size of captured frame
   // set frame width to 640
   ipcam.set(CV_CAP_PROP_FRAME_WIDTH, 640); 
   // set frame height to 480 
   ipcam.set(CV_CAP_PROP_FRAME_HEIGHT, 480); 
   // stop ipcam capture if keyboard ‘space’ pressed
   while (keyCode != 32)
   {
      // grab frame from streaming video
      ipcam >> frame;                        
      if (frame.empty() == false) 
      {
         imshow(“Preview IP-Cam”, frame);     // show frame
         keyCode = cvWaitKey(1);
      }
   }
}
// don’t forget to release after it is not used
frame.release();
ipcam.release();
return 0;
}

Read Kinect / Asus XTion Pro Live

I assume you already have OpenNI, PrimeSense NITE installed in your system and read about configuring Visual Studio 2010 with OpenCV, OpenNI, PrimeSense, and CUDA here.  I didn’t use OpenCV to stream Asus XTion Pro Live RGB-D, but I will converted that stream to OpenCV Mat so I can manipulate further with that input. You can find these code from OpenNI samples but they will use OpenGL to view the stream. You can use OpenNI with OpenCV if you compile OpenCV with WITH_OPENNI flag checked.

These are the header that will be used:

#include <Windows.h>
#include <XnCppWrapper.h>
#include <XnPropNames.h>
#include <opencv2\opencv.hpp>

using namespace cv;
using namespace xn;

These are variables that will be used:

enum StreamMode             // streaming mode
{
  DUAL_SENSOR,
  OVERLAY_SENSOR
};

Context context;            // create context
XnStatus statusRet;         // status for an operation
XnMapOutputMode outputMode; // setting output mode

float* depthHist;           // histogram depth
XnDepthPixel depthZRes;     // depth-Z
DepthGenerator depthMap;    // depth generator
DepthMetaData depthData;    // depth metadata

XnRGB24Pixel* texMap;       // texture map
unsigned int texMapX = 0;   // X-size texture map
unsigned int texMapY = 0;   // Y-size texture map
ImageGenerator colorMap;    // color generator
ImageMetaData colorData;    // color metadata

int streamFPS;     // streaming FPS 
Size streamSize;   // size streaming

// OpenCV Mat that contain depth and color pixel matrix
Mat depthFrame, colorFrame, overlayFrame; 

Function to draw overlay RGB-D with accumulative histogram

void DrawOverlay(DepthMetaData &data)
{
  const XnDepthPixel* depthBuff = data.Data();
  texMapX = (unsigned short)data.FullXRes();
  texMapY = (unsigned short)data.FullYRes();
  depthZRes = data.ZRes();
  depthHist = (float*)malloc(depthZRes * sizeof(float));
  // create accumulative histogram from depth  
  unsigned int nValue = 0;
  unsigned int numPoints = 0;
  memset(depthHist, 0, depthZRes * sizeof(float));
  for (XnUInt y = 0; y < data.YRes(); ++y)
  {
    for (XnUInt x = 0; x < data.XRes(); ++x)
    {
      nValue = *depthBuff;
      if (nValue != 0)
      {
++depthHist[nValue];
++numPoints;
      }
++depthBuff;
    }
  }
  for (int nIndex=1; nIndex < depthZRes; nIndex++)
    depthHist[nIndex] += depthHist[nIndex-1];
  if (numPoints)
  {
    for (int nIndex=1; nIndex < depthZRes; nIndex++)
      depthHist[nIndex] = (unsigned int)(256 * (1.0f – (depthHist[nIndex] / numPoints)));
  }
  // create texture map  
  memset(texMap, 0, texMapX * texMapY * sizeof(XnRGB24Pixel));
  const XnDepthPixel* pDepthRow = data.Data();
  XnRGB24Pixel* pTexRow = texMap + data.YOffset() * texMapX;
  for (XnUInt y = 0; y < data.YRes(); ++y)
  {
    const XnDepthPixel* pDepth = pDepthRow;
    XnRGB24Pixel* pTex = pTexRow + data.XOffset();
    for (XnUInt x = 0; x < data.XRes(); ++x, ++pDepth, ++pTex)
    {
      if (*pDepth != 0)
      {
int nHistValue = depthHist[*pDepth];
pTex->nRed = nHistValue;
pTex->nGreen = nHistValue;
pTex->nBlue = 0;
      }
    }
    pDepthRow += data.XRes();
    pTexRow += texMapX;
  }
  delete(depthHist);
}

Function to capture stream RGB, Depth sensor and Overlay RGB-D:

XnStatus CaptureStream(int mode)
{
  statusRet = context.WaitAnyUpdateAll();
  if (statusRet == XN_STATUS_OK)
  {
    switch (mode)
    {
    // Read dual RGB-Depth Sensor
    case 0:  
// Depth
depthMap.GetMetaData(depthData);
const XnDepthPixel* depthBuff = depthMap.GetDepthMap();
depthFrame.create(depthData.FullYRes(), depthData.FullXRes(), CV_16UC1);
memcpy(depthFrame.data, depthBuff, depthData.FullYRes() * 
depthData.FullXRes() * sizeof(XnDepthPixel));
// RGB
colorMap.GetMetaData(colorData);
const XnRGB24Pixel* colorBuff = colorMap.GetRGB24ImageMap();
colorFrame.create(colorData.FullYRes(), colorData.FullXRes(), CV_8UC3);
memcpy(colorFrame.data, colorBuff, colorData.FullYRes() *
colorData.FullXRes() * sizeof(XnRGB24Pixel));
cvtColor(colorFrame, colorFrame, CV_RGB2BGR);
break;
    // Overlay RGB-Depth Sensor
    case 1:
depthMap.GetMetaData(depthData);
colorMap.GetMetaData(colorData);
DrawOverlay(depthData);
overlayFrame.create(texMapY, texMapX, CV_8UC3);
memcpy(overlayFrame.data, texMap, texMapY * texMapX * sizeof(XnRGB24Pixel));
cvtColor(overlayFrame, overlayFrame, CV_RGB2BGR);
break;
    }
  }
  return statusRet;
}

Main program: 

Just press “1” or “2” to change streaming mode dual RGB&Depth – Overlay

Press “space” to exit program

int main(int argc, char** argv) 
{
  outputMode.nFPS = 30;    // set fps to 30
  outputMode.nXRes = 640;  // set frame width
  outputMode.nYRes = 480;  // set frame height

  int cIdxGen = DUAL_SENSOR;     // default view dual RGB-Depth
  int pIdxGen = cIdxGen;
  statusRet = context.Init();    // initialize openni
  context.SetGlobalMirror(true); // mirror output stream
  if (statusRet == XN_STATUS_OK) 
  {
    statusRet = depthMap.Create(context); // create depth map
    statusRet = colorMap.Create(context); // create color map
    // set output stream from each sensor to our settings
    statusRet = colorMap.SetMapOutputMode(outputMode); 
    statusRet = depthMap.SetMapOutputMode(outputMode);
    // begin to stream
    statusRet = context.StartGeneratingAll();
    if (statusRet == XN_STATUS_OK)
    {
      texMap = new XnRGB24Pixel[outputMode.nXRes*outputMode.nYRes];
      while (true)
      {
        if (CaptureStream(cIdxGen) == XN_STATUS_OK)
{
         switch (cIdxGen)
         {
         case 0:     
           imshow(“Depth Sensor”, depthFrame);
           imshow(“RGB Sensor”, colorFrame);
           break;
         case 1:      
           imshow(“Overlay Sensor”, overlayFrame);
           break;
         }
         char key = cvWaitKey(10);
         if (key == 49)                // keyboard “1” pressed
           cIdxGen = DUAL_SENSOR;
         else if (key == 50)           // keyboard “2” pressed
           cIdxGen = OVERLAY_SENSOR;
         else if (key == 32)           // keyboard “space” pressed
           break;  
         if (pIdxGen != cIdxGen)
           pIdxGen = cIdxGen;
          }
        }
      }
    }
    else
      MessageBox(NULL, L“Failed to initialize OpenNI!”, L“Error!”, 0);
  }
  // release all unused created memory
  depthMap.Release(); colorMap.Release(); userMap.Release();
  depthFrame.release(); colorFrame.release();
  overlayFrame.release(); userFrame.release();
  context.Shutdown();

  return 0;
}

You can do the same to get streaming from Infra Red sensor

Image Processing : Acquisition with OpenCV Part I

I assume that you already configure your project in Visual Studio C++ with OpenCV from the previous tutorial here.

Before manipulating digital image or video, we need to read the input which obtained from optical device. OpenCV make this acquisition easier. But first we need to make an OpenFileDialog class to browse the file we want.

Create a File Browser in C++

First create OpenFileDialog.h and OpenFileDialog.cpp in your Visual Studio project.

open-file-browser

Write this code in OpenFileDialog.h

#pragma once
#include <Windows.h>
#include <Commdlg.h>
#include <tchar.h>
class OpenFileDialog
{
public:
TCHAR* DefaultExtension; ///< Default extension
TCHAR* FileName; ///< File name
TCHAR* Filter; ///< Filter file type
TCHAR* Title; ///< GUI title
TCHAR* InitialDir; ///< Initial directory
int FilterIndex; ///< Filter index
int Flags; ///< Flag, true if file is open
HWND Owner; ///< Window handle
/** Open file */
OpenFileDialog(void);
/** Show GUI */
bool ShowDialog();
};

Write this code in OpenFileDialog.cpp

#include “OpenFileDialog.h”
// Open file
OpenFileDialog::OpenFileDialog(void)
{
this->DefaultExtension = 0;
this->FileName = new TCHAR[MAX_PATH];
this->Filter = 0;
this->FilterIndex = 0;
this->Flags = OFN_PATHMUSTEXIST | OFN_FILEMUSTEXIST;
this->InitialDir = 0;
this->Owner = 0;
this->Title = 0;
}
// Show GUI
bool OpenFileDialog::ShowDialog()
{
OPENFILENAME ofn ;
ZeroMemory(&ofn, sizeof(ofn));
ofn.lStructSize = sizeof(ofn);
ofn.hwndOwner = this->Owner;
ofn.lpstrDefExt = this->DefaultExtension;
ofn.lpstrFile = this->FileName;
ofn.lpstrFile[0] = ‘\0’;
ofn.nMaxFile = MAX_PATH;
ofn.lpstrFilter = this->Filter;
ofn.nFilterIndex = this->FilterIndex;
ofn.lpstrInitialDir = this->InitialDir;
ofn.lpstrTitle = this->Title;
ofn.Flags = this->Flags;
GetOpenFileName(&ofn);
if (_tcslen(this->FileName) == 0) return false;
return true;
}

Now in your main program write this code, for example in your MainProgram.cpp this are the headers that will be used:

#include <Windows.h>
#include <string>
#include <opencv\cv.h>
#include <opencv\highgui.h>
#include “OpenFileDialog.h”
using namespace std;
using namespace cv;

We will wrap up that class on simple Open() function that return path to any file we choose.

/** Open File */
string Open(string paths)
{
string path = “”;
TCHAR* initDir = new TCHAR[paths.size() + 1];
initDir[paths.size()] = 0;
copy(paths.begin(), paths.end(), initDir);
OpenFileDialog* openFileDialogs = new OpenFileDialog();
openFileDialogs->FilterIndex = 1;
openFileDialogs->InitialDir = initDir;
openFileDialogs->Title = _T(“Open image file”);
if (openFileDialogs->ShowDialog())
{
wstring arr_w(openFileDialogs->FileName);
string arr_s(arr_w.begin(), arr_w.end());
path = arr_s;
}
return path;
}

Read Image

OpenCV can read many types of image such as .jpg, .bmp, .png, etc. This code shows how you open file browser and read an image with OpenCV.

int main(int argc, char** argv)
{
Mat image; // create matrix container for image
string path = Open(“”); // get path image
if (path != “”) // check if path is empty
{
image = imread(path, 1); // read image from path
imshow(“Preview Image”, image); // show image to the screen
cvWaitKey(0); // wait any key pressed to continue
}
image.release(); // do not forget to release what you have created
return 0;
}

Read Video
OpenCV can also read file video format. This code shows how to open file video and play it using OpenCV.

int main(int argc, char** argv)
{
VideoCapture video; // memory container for video
Mat frame; // matrix container for frame
int nFrame; // number of frame
int frameRate; // frame rate
double fps; // frame per second
bool status; // status if capture is vaild
string path = Open(“”); // open file video
if (path != “”) // check if path is empty
{
video = VideoCapture(path); // load video from path to memory
if (video.isOpened()) // check if video is valid
{
fps = video.get(CV_CAP_PROP_FPS); // get frame per second from the video
nFrame = (int)video.get(CV_CAP_PROP_FRAME_COUNT); // get number of frame
frameRate = 1000 / fps; // set frame rate
status = video.set(CV_CAP_PROP_POS_FRAMES, 1); // go to frame 1
while (status == true) // check if status is true
{
video >> frame; // begin grab frame by frame along with the loop iteration
if (frame.empty() == false) // check if grab frame is succeed
{
imshow(“Preview Video”, frame); // show frame video on the screen
if (cvWaitKey(frameRate) == 32) // check if user press ‘esc’
status == false; // end the loop
}
else
status == false; // end the loop
}
}
}
// release after memory is not used
frame.release();
video.release();
return 0;
}

Read Webcam 

This code shows how to read webcam.

// read webcam
int main(int argc, char** argv)
{
VideoCapture webcam; // memory container for webcam stream
Mat frame; // matrix container for frame
bool status; // status looping
// capture webcam index 0
// if we have 2 webcam, then the index will be 0 or 1
webcam = VideoCapture(0);
if (webcam.isOpened()) // check if webcam is valid
{
int keyCode = 1; // key to end the loop
// set frame width from webcam stream
webcam.set(CV_CAP_PROP_FRAME_WIDTH, 640);
// set frame height from webcam stream
webcam.set(CV_CAP_PROP_FRAME_HEIGHT, 480);
// begin looping until user press ‘esc’
while (keyCode != 32)
{
webcam >> frame; // grab frame from webcam stream
if (frame.empty() == false) // check if frame is valid
{
imshow(“Preview Webcam”, frame); // show frame on the screen
keyCode = cvWaitKey(1); // give a miliseconds gap between frame
}
}
}
// release every memory
frame.release();
webcam.release();
return 0;
}
 

Done! There was some basic acquisition in OpenCV.

Image Processing : Configure Visual Studio with OpenCV

I assume you have compiled OpenCV 2.4.10 library with GPU support from the last tutorial or install OpenCV 2.4.10 in your machine. We will use Visual Studio 2010 C++ or any Visual Studio C++ you have. Before starting, you need to configure your Visual Studio project to know where you put OpenCV or additional library such as CUDA, OpenNI, and PrimeSense NITE.

  • CUDA is required for OpenCV GPU module (you can ignore it if you do not have NVidia graphics card in your machine).
  • OpenNI and PrimeSense NITE are required for ASUS XTion Pro Live and Kinect (you can ignore it if you do not have ASUS XTion Pro Live or Kinect).

So CUDA, OpenNI, and PrimeSense NITE act as additional requirement and can be ignored if you don’t need them. Here is the configuration:

Create Empty Project in Visual Studio

  • Open Visual Studio 2010.
  • Create a C++ project and select Win32 console application.
  • Give a name to the project (without space or characters) then click OK.

vs-solution

  • Click Next — select Console Application — check Empty project — click Finish.

empty-win32-console

  • Now you are done creating empty project in Visual C++.

Create New File in Visual Studio

  • Go to Solution Explorer — Source Files — Right click — Add — New Item.solution-new-file
  • Select C++ File (.cpp) file then give it a name — click Add

make-cpp-file

  • Now you get a .cpp file in your project. The Source Files is a folder where you put all your .cpp / .c / .cu files and the Header Files is a folder where you put all your .h / .hpp / .cuh files

Configure Visual Studio with OpenCV

  • Go to properties of your project by right click in the project name and choose Properties.

properties-of-project

  • In VC++ Directories — Include Directories, put the path of your OpenCV include directory. Do the same thing for Debug and Release mode. Here is the preview of my setting [You can ignore the PrimeSense NITE, CUDA and OpenNI path if you don’t need it] :

include-setting-openni

  • In VC++ Directories — Library Directories, put the path of your OpenCV library directory. Do the same thing for Debug and Release mode. Here is the preview of my setting [Again you can ignore the PrimeSense NITE, CUDA and OpenNI path if you don’t need it] :

library-setting-openni

  • In Linker — Input — Additional Dependencies, put all the library names from OpenCV. If you are in Debug mode then copy these OpenCV libraries [Note that this is for OpenCV 2.4.10 so if you have different version of OpenCV, just look in the path of your OpenCV library directory]:
opencv_calib3d2410d.lib
opencv_contrib2410d.lib
opencv_core2410d.lib
opencv_features2d2410d.lib
opencv_flann2410d.lib
opencv_gpu2410d.lib
opencv_highgui2410d.lib
opencv_imgproc2410d.lib
opencv_legacy2410d.lib
opencv_ml2410d.lib
opencv_nonfree2410d.lib
opencv_objdetect2410d.lib
opencv_photo2410d.lib
opencv_stitching2410d.lib
opencv_ts2410d.lib
opencv_video2410d.lib
opencv_videostab2410d.lib

If you are using CUDA and OpenNI, you need to copy these CUDA and OpenNI libraries as well [Again this is optional]:

cudart.lib
openNI.lib

If you are in Release mode then copy these OpenCV libraries [Note that this is for OpenCV 2.4.10 so if you have different version of OpenCV, just look in the path of your OpenCV library directory]:

opencv_calib3d2410.lib
opencv_contrib2410.lib
opencv_core2410.lib
opencv_features2d2410.lib
opencv_flann2410.lib
opencv_gpu2410.lib
opencv_highgui2410.lib
opencv_imgproc2410.lib
opencv_legacy2410.lib
opencv_ml2410.lib
opencv_nonfree2410.lib
opencv_objdetect2410.lib
opencv_photo2410.lib
opencv_stitching2410.lib
opencv_ts2410.lib
opencv_video2410.lib
opencv_videostab2410.lib

Again if you use CUDA and OpenNI, you need to copy these CUDA and OpenNI libraries [Again this is optional]:

cudart.lib
openNI.lib
  • Last step, add OpenCV binary (bin folder) by Right click in My Computer — Properties — Advanced System Setting — Environment Variables — Path — add your path to OpenCV bin folder
  • Or the simple way is just copy all the .dll inside OpenCV bin folder to your Output project (where your .exe generated).
  • Again if you use CUDA and OpenNI, you need to add their bin folder to the environment path too.

path-bin-opencv

Done! Now you are good to go to the coding step!

Image Processing : Build OpenCV with GPU Support

Leave a reply

There are lots of library for computer vision but I choose to use OpenCV. You can get OpenCV anywhere just google it. You can choose easy install (the .exe one) or build from source. I prefer .exe one but there is a problem, it is not support GPU. So we need to compile OpenCV 2.4.10 myself. You must have NVidia driver that support CUDA and Visual Studio 2010 or later. Fortunately, my notebook has GeForce GT540M 2GB.

Here is the step:

  • Download and install OpenCV 2.4.10 source from OpenCV website. If you already download the .exe then extract it, look at the installation folder there are folder called source. You can use that to build OpenCV just like I did.
  • Download and install CMake, I use CMake 3.1.0.
  • Download and install NVidia CUDA Toolkit 6.5 from NVidia website.
  • Download and install OpenGL, OPENNI and TBB.
  • Download Python from Python website.

Now your preparation is completed, the rest are the installation steps.

How to build OpenCV with GPU support?

First you need to configure CMake. Here is the steps :

  • Open CMake and set CMake source to OpenCV source location.
  • Set CMake build to empty folder where you build the source.
  • Check the grouped and advanced checkbox. The setting just like this preview.

header-cmake

  • Click configure, it will redirect you to select compiler. I choose Visual Studio 2010 (32bit).
  • Select specify native compiler then click next.
  • Browse for cl.exe in Visual Studio 10.0/VC/bin/cl.exe, fill in for C and C++ compiler, click finish. You will see many red lines there but don’t be affraid you’ve done nothing wrong.

You just done configuring CMake. Now you need to change some of these configuration.

  • Leave the Ungrouped Entries, BUILD, CLAMDBLAS, CLAMDFFT, CMAKE, CPACK as it is.
  • Expand the CUDA group, there are some entry that already checked by default but you need to unchecked CUDA_ATTACH_VS_BUILD_RULE_TO_CUDA_FILE.

cuda-body

  • Leave the ENABLE, GHOSTSCRIPT, GIGEAPI, INSTALL, JAVA, OPENCV as it is but make sure ENABLE_SOLUTION_FOLDERS checked.
  • Expand the WITH group, there are some entry that already checked by default. You need to check WITH_CUBLAS, WITH_CUDA, WITH_OPENGL, WITH_OPENNI, WITH_TBB.

body-with

  • Click configure to update the configuration.
  • Expand the OPENGL, OPENNI, PYTHON and TBB group. You need to fix the location of NOT-FOUND folder such as include and libs folder to their right path where you install them.

body-other

  • After every folder path / location correct, click configure to update the configuration.
  • Now you should not see any red lines left like this preview. If you still seeing red lines just check it and fix the wrong path until it is right.

body-cmake

Build OpenCV

  • Click generate and wait the process to finish.
  • Go to the build folder you’ve created before, look for OpenCV.sln.
  • Open it on Visual Studio and set to VS2010 to Debug mode.
  • This solution contain many projects just select project with name ALL_BUILD then click build. It will build all the project for you.
  • Just click NO if pops up from Visual Studio appeared.
  • Take a coffee because it will take long time to finish and hopefully without any errors.
  • After all build done without error, select project named INSTALL then build it.
  • That will generate install folder in build folder that contain necessary include, bin and lib folder for your OpenCV project.
  • Do the same for Release mode.

Now, you finish all the hard work and you have compiled OpenCV with GPU support.