Update
Ok, you can now grab the MotionTracker source code (AS2 & AS3). Version 2 eventually will include the other methods for detecting and tracking motion which I mentioned. For now I have just included code for the technique used in the demo.
Download: AS3 Webcam Motion TrackingFor those of you without access to a webcam (and as an example of a practical use for this class) here is a short video demonstrating the program I wrote for the installation piece.
End of update
Webcam required to view the demo (obviously…)
I’m currently working on putting up a show, part of which will be a live generative piece, constructed from the movement of visitors in the gallery space. So of course I needed a method for not only detecting but tracking motion. I’ve seen some of Grant Skinner and Mario Klingemann’s incredible experiments in this field, the most accurate of which used a coloured object in the camera’s field of vision (such as a ball or glove) to track movement, however I wanted the live response of the artwork to be a surprise to the viewer, and so asking them to hold a coloured object would have detracted from the overall impact.
If you ever want to be amazed by the human mind, try recreating the tasks that it performs, particularly those which we take for granted; you will quickly realise the incredible amount of computation which must be happening just to, for example, detect movement and track an object, or to discern more than one object and their relative motions, velocities and distances. This isn’t an essay on Neuroscience though so I’ll save that for another time…
Anyway; I researched the concept of motion tracking, and made some notes on my own ideas. A few possible approaches emerged.
One of the most attractive ideas was to divide the screen into a grid, and average the colours within each segment at regular intervals. The difference in brightness within each segment (from one calculation to the next) would determine whether or not motion had occurred in this area of the screen. Using this data, clusters of segments with a high rate of change could be used to assume an object in motion. This approach is quite accurate, however it would require a lot of getPixel and setPixel, as well as converting these results into Hue, Saturation, Brightness objects; all of which would be quite processor heavy. A decent computer could handle this fairly easily, however I had to also consider the processing load of the actual generative work running over the top.
The method I used in the end was much lighter, and providing the parameters could be adjusted to suit the environment, it would yield enough accuracy to provide a quite responsive tracking point for the artwork to follow.
Here’s how it works
- After the camera object is set up, it is passed to a MotionTracker class I wrote, which extends the Point object. The MotionTracker class contains two BitmapData objects; one representing the current data from the webcam, the other is used to store the previous frame.
- When the update method is called, the new frame is drawn on top of the previous using the difference filter.
- The result then has a ColorMatrixFilter applied to it, increasing the contrast of the image and dropping the darker areas (with less movement) further into the background.
- A blur filter is then applied in order to further reduce noise and to form blobs from areas where more movement is occurring.
- The threshold method is then called in the resulting BitmapData, effectively causing all near black pixels to be ignored and all light pixels (where movement has been detected) to be set to a certain colour.
So that takes care of the motion detection, but what about the tracking?
- First we call the getColorBoundsRect method on the BitmapData object, which gives us a Rectangle object representing the area occupied by pixels of the colour we used when applying the threshold filter.
- If the rectangle has an area larger than a specified constant, then sufficient movement is occurring and it can be tracked (choosing to ignore small areas of movement helps to eradicate noise caused by background and other interferences).
- We then find the centre point of this rectangle, and then move the x and y position of the MotionTracker instance to these coordinates, applying some gentle easing to give more continuity to the movement.
I will post some source code once the MotionTracker class has been refined, and once I have build a nicer API so that, amongst other things, you can create your own UI for tweaking parameters at runtime; a useful feature if you have requirements such as mine where getting the most accuracy from a particular environment is important.
Download: AS3 Webcam Motion Tracking
You can also download the AS3 Motion Tracking source code from the soulwire code repository.
Thanks.
Your code will help me with a couple of demos, I did with AS2 some time ago, and that I’d like to upgrade to AS3.
Cheers
When I try to publish this from Flash source file, nothing happens. It doesn’t ask to allow access to the camera. If I just run the swf file – it works fine (and does ask for access to camera). I’m using Flash CS4 10.0.2. Can anyone think why it wouldn’t work?
Also, there is no sign of AS2 demo in the zip??
Please help. Thanks.
How much would you charge if using your code for some bisnuess project?
It is free to use for commercial projects, but please give credit if appropriate and let me know what you do with it as I’m interested in how you use it!
The problem here was to do with local security settings of Flash. I moved the folder to the desktop and it worked. However, I can’t change the settings because every time I go to the Flash Player settings (in any browser), it crashes with the “…. If it continues to run your computer might become unresponsive”, and that’s a movie set up by Adobe!! But that’s a different story.
This motion tracking class is excellent. Thanks indeed!
AS3 Motion Tracking was the best project on the net. What about the “v2″, will you update it? It would be so great if you do the “v2″, i wait so long time for it. Thanks any way, you’re my mentor!
Thanks for your help. The project is still on the way. I will show you some demo when my work is done.
i ma in search of same concept but ma plateform is vb.net , i need to detect the motion of the finger …plzz help me …
Hey,
I found your idea really good!
I tried it myself here:
http://vargaandras.com/bme/diploma/webcam/
One addition I made was calculating the center of mass instead of the geometrical center of the rectangle containing the white pixels. This gave me much better results, of course it adds some cpu time but I think its worth it, you could give it a try!
I noticed your method of tracking is primative. Are you attempting refining your detection by using the SIFT algorithm or identifying and tracking specific items instead of detecting motion. It appears that the core functionality hopes only one object is moving and only one needs to be tracked.
Implementing SIFT, or a varient, would allow multiple users to be tracked even as the pass by eachother.
Hey Mark. Yep, you’re right this is very primitive. It was coded for speed and suited my needs for the project, but is by no means a decent piece of CV. You should check out what Eugene has been doing with pattern recognition and CV in flash though.
In terms of this experiment though – I think some auto thresholding and blod detection / tracking would improve it and allow the tracking of multiple objects (or features of one person). I haven’t touched this for a couple of years though :)
I am impressed with the speed that you able to get. I’ve been working with this ( http://sotipro.com/camFun.php ). The Kaleidoscope and toonify are basic in themselves. What i’m working with is the Pinscape app. I believe most of my lag comes from the 12,533 images that i’m maintaining. My implementation of the motion tracking variant is actually more primitive than yours because at this time I haven’t attempted any filters nor do I care about the region that is moving per se. I have also included a delay in still areas to get a tracer effect.
Have you had any experience implementing a Kalman filter?
I’m still somewhat new the the nuances of Flash. Do you know which of the following would be more efficient?
Import 1 image and duplicate it X number of times (ie: 12,533) as seperate Bitmaps objects.
Import 1 image and draw it X number of times on a single Bitmap object.
Use the sprite object and draw my object within ActionScript.
Hey, i used it in one of my project. Great Work. Thanks.
Wow what a website! Very nice projects.
I was searching for such thing quite some time, will use it for my school project! Thnks!
How would I alter the code to control the computer’s cursor?
Thanks.
You can’t control the user’s cursor! But you could call Mouse.hide() then move a Sprite which looks like a cursor using the tracker x / y coordinates.