With the newly acquired robot eyes, we were left with the question of what distinguishes the objects from each other?

Shape and size: Using these characteristics and constructing custom object (similar to facial) recognition using opencv and haartraining the different objects could be “seen” by the rover. This ended up being a balance of accuracy and speed, of which neither was good enough to use in the application.

Position: Using the rover’s position, both location and orientation, the specific object could be determined from its coordinates. The numbers coming back from the rover, through the beacon, ended up being too cryptic and not reliable enough to give specific positioning information, especially for moving a precise onscreen target. There was also no way to get vertical movement information.

Color: Pairing a unique color with the object to be identified could allow color detection to run the brain of the rover. This ended up being the most streamlined and accurate approach to identifying object in the interactive. Using a vibrant red, green and blue gave enough differentiation between the objects to bring up specific prompts for each target. Running the color detection on the video stream and then creating a box around the identified color gave a specific position in the video frame. The target could then be positioned in the center of the box, and exceptions could be made for outlying false positives. Color scraps could be disregarded and focus could be on heavy concentrations of the specific colors. Below you can see the blue box (hidden when not in debug mode) surrounding the green colored image.

Lighting greatly impacted the color detection on the objects. Above is a picture of two of the colored boxes blacked out by the background window light. The objects needed to be lit from a reliable, consistent source of light. This light would have to be pointed on the object or what was done in this case from within the object. The targets were lit with colored light to attempt to nullify lighting differences due to sunny or cloudy days (the interactive being in a heavily windowed area). Below are images against the window areas. First, the transparent gels of the three different colors (the red needed some help and has a light behind it). Second is a stronger light competing with the self lit color cone.

Update (prior to posting):

After using the previous version of color detection where one color was identified, it was determined that this was a fickle method given external lighting conditions. The color detection was updated with the following:

For aesthetic reasons, the object was to be internally lit. When in the final space, the bulb was not colorful enough and was whitewashed in the rover’s view. We tried to use the colored gels and other manipulations but due to the fact that the rover was looking for one specific color we were not able to tune it in enough to be accurate. Take two.

Now we were looking for two objects. First a strong white light source, which is what we had and were trying to fight the first time, and then look below the light for a color. The first method was based on the blob detection outlined here by Tomek Augustyn (Og2t). This takes the video object, draws it to a bitmap object and then creates another bitmap object using the threshold method finding the brightest spots in the video. After singling out the brightest spots, we take some measurements of the individual rectangles found with getColorBoundsRect(). If they meet the minimum and maximum requirements for size, of which in this case the minimum height is a ratio of the width, then we look for the next piece of information.

Below the bright spot is a colored ring that can be averaged out and determined to be in a specific range. Using the technique outlined here by Justin Windle we have averaged out the ring color and can ask if the RGB values are in the range that we are looking for:

if((colorin >> 16 & 0xFF) < 0x5f && (colorin >> 8 & 0xFF) > 0x84 && (colorin & 0xFF) < 0xc6){do something}

If the color is in the correct range then we are displaying the found target icon over the bright area. This gives a bit more flexibility and accuracy in finding the correct targets.

Update (3/29/10):

You are witnessing the evolution of the color detection idea as it plays out in the physical space. The previous method, which relied on a strong light source, did not play out in a brightly-lit, white-walled space because the light source was buried in the sea of bright white wall behind it. I was able to scavenge the threshold method that was used previously and base the next method around that.

For each of the three colors we were looking for (red, green, blue), I ran the video through a couple of thresholds to filter out the precise color we were looking for. Red for instance uses these lines to filter out the red color from the background images:

aBitmapDataObject.draw(theVideoObject);
var pt:Point = new Point(0,0);
var rect:Rectangle = new Rectangle(0,0,aBitmapDataObject.width,aBitmapDataObject.height);
var threshold1:uint = 0x00004040; //green and blue values
private var color1:uint = 0xff000000; //turn this black for next threshold because it is greater than >
private var maskColor1:uint = 0x0000ffff; //mask on green and blue
private var threshold2:uint = 0x00600000; //red value
private var color2:uint = 0xffff0000; //change this to a indicating color (bright red) to mark
private var maskColor2:uint = 0x00ff0000; //mask on red value
//turn anything with green and blue pixel values greater than threshold1 black to filter out strong blue/green color
aSecondBitmapDataObject.threshold(aBitmapDataObject, rect, pt, ">", threshold1, color1, maskColor1, true);
//turn anything with red pixel values greater than threshold2 indicator color2 to find strong red
aSecondBitMapDataObject.threshold(aSecondBitMapDataObject, rect, pt, ">", threshold2, color2, maskColor2, true);
boundRectangle = aSecondBitMapDataObject.getColorBoundsRect(0xffffff, 0xff0000,true);

Here is the blue:

aBitmapDataObject.draw(theVideoObject);
var pt:Point = new Point(0,0);
var rect:Rectangle = new Rectangle(0,0,aBitmapDataObject.width,aBitmapDataObject.height);
var threshold1:uint = 0x00000099; //blue values
private var color1:uint = 0xffffffff; //turn this white for next threshold because it is less than <
private var maskColor1:uint = 0x000000ff; //mask on blue
private var threshold2:uint = 0x00440000; //red value
private var color2:uint = 0xff0000ff; //change this to a indicating color (bright blue) to mark
private var maskColor2:uint = 0x00ff0000; //mask on red value
//turn anything with blue pixel values less than threshold1 white to filter out weak blue
aSecondBitmapDataObject.threshold(aBitmapDataObject, rect, pt, "<", threshold1, color1, maskColor1, true);
//turn anything with red pixel values less than threshold2 indicator color2 because the blue does not contain red
aSecondBitMapDataObject.threshold(aSecondBitMapDataObject, rect, pt, "<", threshold2, color2, maskColor2, true);
boundRectangle = aSecondBitMapDataObject.getColorBoundsRect(0xffffff, 0xff0000,true);

After we have the boundRectangle we can use the technique previously mentioned from Tomek Augustyn (Og2t), to check for height and width sizes or ratios. And the color averaging method to weed out false positives by Justin Windle.

Download Flex Builder 4 Project – Simple color detection

Download Flex Builder 4 Project – Complex color detection