Color Detection & Tracking with ESP32 CAM & OpenCV

Overview

This project is all about Color Detection & Tracking with ESP32 CAM Module & OpenCV. Thus, we will be detecting any specific colors during live video streaming. Colour detection is necessary to recognize objects, it is also used as a tool in various image editing and drawing apps. This method is completely different from other ESP32-CAM Color Detection methods as we are not writing the color detection code for Microcontroller. Rather we will use our laptops for Python Code. This method makes the processing faster.

Later we will also share the values from our python code to ESP32-CAM using a Client-Server connection. The method of color detection used here is HSV or Hue Saturation Value conversion.

Here we have used the ESP32-CAM module, which is a small camera module with the ESP32-S chip. Besides the OV2640 camera and several GPIOs to connect peripherals, it also features a microSD card slot that can be useful to store images taken with the camera.

Initially we will write the C Code for ESP32 CAM & then install Python and the required Python libraries. Later we will go through the Python Programming for using OpenCV. This is an essential tutorial as you will be able to use any sort of Image processing or Machine Learning on the live video without having to write it on Arduino IDE. But before getting started you can check our previous project about Face Recognition System & also Object Identification System.

Bill of Materials

The following is the list of Bill of Materials for building an ESP32 CAM Based Color Detection System. You can purchase all these components from Amazon.

S.N.	Components	Quantity	Purchase Links
1	ESP32-CAM Board AI-Thinker	1	Amazon \| AliExpress
2	FTDI Module	1	Amazon \| AliExpress
3	Micro-USB Cable	1	Amazon \| AliExpress
4	Jumper Wires	10	Amazon \| AliExpress

ESP32 CAM Module

The ESP32 Based Camera Module developed by AI-Thinker. The controller is based on a 32-bit CPU & has a combined Wi-Fi + Bluetooth/BLE Chip. It has a built-in 520 KB SRAM with an external 4M PSRAM. Its GPIO Pins have support like UART, SPI, I2C, PWM, ADC, and DAC.

The module combines with the OV2640 Camera Module which has the highest Camera Resolution up to 1600 × 1200. The camera connects to the ESP32 CAM Board using a 24 pins gold plated connector. The board supports an SD Card of up to 4GB. The SD Card stores capture images.

To learn in detail about the ESP32 Camera Module you can refer to our previous Getting Started Tutorial.

ESP32-CAM FTDI Connection

The board doesn’t have a programmer chip. So In order to program this board, you can use any type of USB-to-TTL Module. There are so many FTDI Module available based on CP2102 or CP2104 Chip or any other chip.

Make a following connection between FTDI Module and ESP32 CAM module.

ESP32-CAM	FTDI Programmer
GND	GND
5V	VCC
U0R	TX
U0T	RX
GPIO0	GND

Connect the 5V & GND Pin of ESP32 to 5V & GND of FTDI Module. Similarly, connect the Rx to UOT and Tx to UOR Pin. And the most important thing, you need to short the IO0 and GND Pin together. This is to put the device in programming mode. Once programming is done you can remove it.

Project PCB Gerber File & PCB Ordering Online

If you don’t want to assemble the circuit on a breadboard and you want PCB for the project, then here is the PCB for you. The PCB Board for ESP32 CAM Board is designed using EasyEDA online Circuit Schematics & PCB designing tool. The PCB looks something like below.

The Gerber File for the PCB is given below. You can simply download the Gerber File and order the PCB from ALLPCB at 1$ only.

Download Gerber File: ESP32-CAM Multipurpose PCB

You can use this Gerber file to order high quality PCB for this project. To do that visit the ALLPCB official website by clicking here: https://www.allpcb.com/.

You can now upload the Gerber File by choosing the Quote Now option. From these options, you can choose the Material Type, Dimensions, Quantity, Thickness, Solder Mask Color and other required parameters.

After filling all details, select your country and shipping method. Finally you can place the order.

You can assemble the components on the PCB Board.

Installing ESP32CAM Library

Here we will not use the general ESP webserver example rather another streaming process. Therefore we need to add another ESPCAM library. The esp32cam library provides an object oriented API to use OV2640 camera on ESP32 microcontroller. It is a wrapper of esp32-camera library.

Go to the following Github Link and download the zip library as in the image

Once downloaded add this zip library to Arduino Libray Folder. To do so follow the following steps:
Open Arduino -> Sketch -> Include Library -> Add .ZIP Library… -> Navigate to downloaded zip file -> add

Source Code/Program for ESP32 CAM Module

Here is a source code for Color Detection & Tracking with ESP32 CAM Module. Copy the code and paste it in the Arduino IDE.

#include <WebServer.h>

#include <WiFi.h>

#include <esp32cam.h>

const char* WIFI_SSID = "ssid";

const char* WIFI_PASS = "password";

WebServer server(80);

static auto loRes = esp32cam::Resolution::find(320, 240);

static auto midRes = esp32cam::Resolution::find(350, 530);

static auto hiRes = esp32cam::Resolution::find(800, 600);

void serveJpg()

{

auto frame = esp32cam::capture();

if (frame == nullptr) {

Serial.println("CAPTURE FAIL");

server.send(503, "", "");

return;

}

Serial.printf("CAPTURE OK %dx%d %db\n", frame->getWidth(), frame->getHeight(),

static_cast<int>(frame->size()));

server.setContentLength(frame->size());

server.send(200, "image/jpeg");

WiFiClient client = server.client();

frame->writeTo(client);

}

Void handleJpgLo()

{

if (!esp32cam::Camera.changeResolution(loRes)) {

Serial.println("SET-LO-RES FAIL");

}

serveJpg();

}

Void handleJpgHi()

{

if (!esp32cam::Camera.changeResolution(hiRes)) {

Serial.println("SET-HI-RES FAIL");

}

serveJpg();

}

Void handleJpgMid()

{

if (!esp32cam::Camera.changeResolution(midRes)) {

Serial.println("SET-MID-RES FAIL");

}

serveJpg();

}

Void setup(){

Serial.begin(115200);

Serial.println();

{

using namespace esp32cam;

Config cfg;

cfg.setPins(pins::AiThinker);

cfg.setResolution(hiRes);

cfg.setBufferCount(2);

cfg.setJpeg(80);

bool ok = Camera.begin(cfg);

Serial.println(ok ? "CAMERA OK" : "CAMERA FAIL");

}

WiFi.persistent(false);

WiFi.mode(WIFI_STA);

WiFi.begin(WIFI_SSID, WIFI_PASS);

while (WiFi.status() != WL_CONNECTED) {

delay(500);

}

Serial.print("http://");

Serial.println(WiFi.localIP());

Serial.println(" /cam-lo.jpg");

Serial.println(" /cam-hi.jpg");

Serial.println(" /cam-mid.jpg");

server.on("/cam-lo.jpg", handleJpgLo);

server.on("/cam-hi.jpg", handleJpgHi);

server.on("/cam-mid.jpg", handleJpgMid);

server.begin();

}

Void loop()

{

server.handleClient();

}

Before Uploading the code you have to make a small change to the code. Change the SSID and password variable and in accordance with your WiFi network.

Now compile and upload it to the ESP32 CAM Board. But during uploading, you have to follow few steps every time.

Make sure the IO0 pin is shorted with the ground when you have pressed the upload button.
If you see the dots and dashes while uploading press the reset button immediately
Once the code is uploaded, remove the I01 pin shorting with Ground and press the reset button once again.
If the output is the Serial monitor is still not there then press the reset button again.

Now you can see a similar output as in the image below.

Congratulations, half of the work is done now.

Python Library Installation

For the live stream of video to be visible on our computer we need to write a Python script that will enable us to retrieve the frames of the video. The first step is to install Python. Go to python.org and download Python.

Once download, install Python.
Then Go to the command prompt and install NumPy and OpenCV libraries.

type: pip install numpy and press enter. After the installation is done.
type: pip install opencv-python and press enter, close the command prompt.

Now open Idle code editor or any other python code editor.

Python Code + Color Detection using ESP32 CAM

Now lets learn in detail about the method for Color Detection & Tracking with ESP32 CAM using the OpenCV. Our main objective here is to detect color, thus we will be using HSV conversion using the OpenCV library.

HSV (hue saturation value), also known as HSB (hue, saturation, brightness) is a conversion that is available in the OpenCV library. It is very widely used to detect specific colors. But a limitation to this is that there are some constant values related to detection, which vary from different objects and places.

We convert the RGB image to an HSV image and then we set a lower bound of all three values i.e; hue, saturation, and brightness, and also an upper bound, thus whatever comes in that range is what we desire to see.

For further in-depth understanding, you can see the OpenCV documentation.

Let’s see the ESP32 CAM Python Color Detection code. Create a new file and then copy and paste the code below.

import cv2

import urllib.request

import numpy as np

def nothing(x):

pass

#change the IP address below according to the

#IP shown in the Serial monitor of Arduino code

url='http://192.168.1.61/cam-lo.jpg'

cv2.namedWindow("live transmission", cv2.WINDOW_AUTOSIZE)

cv2.namedWindow("Tracking")

cv2.createTrackbar("LH", "Tracking", 0, 255, nothing)

cv2.createTrackbar("LS", "Tracking", 0, 255, nothing)

cv2.createTrackbar("LV", "Tracking", 0, 255, nothing)

cv2.createTrackbar("UH", "Tracking", 255, 255, nothing)

cv2.createTrackbar("US", "Tracking", 255, 255, nothing)

cv2.createTrackbar("UV", "Tracking", 255, 255, nothing)

while True:

img_resp=urllib.request.urlopen(url)

imgnp=np.array(bytearray(img_resp.read()),dtype=np.uint8)

frame=cv2.imdecode(imgnp,-1)

hsv = cv2.cvtColor(frame, cv2.COLOR_BGR2HSV)

l_h = cv2.getTrackbarPos("LH", "Tracking")

l_s = cv2.getTrackbarPos("LS", "Tracking")

l_v = cv2.getTrackbarPos("LV", "Tracking")

u_h = cv2.getTrackbarPos("UH", "Tracking")

u_s = cv2.getTrackbarPos("US", "Tracking")

u_v = cv2.getTrackbarPos("UV", "Tracking")

l_b = np.array([l_h, l_s, l_v])

u_b = np.array([u_h, u_s, u_v])

mask = cv2.inRange(hsv, l_b, u_b)

res = cv2.bitwise_and(frame, frame, mask=mask)

cv2.imshow("live transmission", frame)

cv2.imshow("mask", mask)

cv2.imshow("res", res)

key=cv2.waitKey(5)

if key==ord('q'):

break

cv2.destroyAllWindows()

Now before you run this code make sure you replace the IP address that you got in the Arduino Serial monitor with the one which we are having in the URL variable in the above code.

Now Run the code by pressing the F5 button. After running the code you can see three screens that appear in front of you.

Now as in the above frame, we wish to track the blue color. Therefore, we adjust the LH, LS, LV, UH, US and UV values from the tracking bars in the Tracking window by sliding them. Adjust these bars values such that at the Mask window, white color appears only at blue colored places as in the images below.

As you can see above that, although I am in the dark with the blue-colored object. Still, it can detect it.

Congratulations you have detected the blue color.
Similarly, by adjusting the Tracking bars you can detect any specific colors.

Now our next step is to Draw a contour around the colored image and specify the average center coordinates of the detected color. To do so save the LH, LS, LV, UH, US, and UV values in the Trackbar window from the previous program.

Now create a new file and copy and paste the code below.

import cv2

import urllib.request

import numpy as np

def nothing(x):

pass

url='http://192.168.1.61/cam-lo.jpg'

##'''cam.bmp / cam-lo.jpg /cam-hi.jpg / cam.mjpeg '''

cv2.namedWindow("live transmission", cv2.WINDOW_AUTOSIZE)

l_h, l_s, l_v = 92, 57, 50

u_h, u_s, u_v = 142, 153, 178

while True:

img_resp=urllib.request.urlopen(url)

imgnp=np.array(bytearray(img_resp.read()),dtype=np.uint8)

frame=cv2.imdecode(imgnp,-1)

#_, frame = cap.read()

hsv = cv2.cvtColor(frame, cv2.COLOR_BGR2HSV)

l_b = np.array([l_h, l_s, l_v])

u_b = np.array([u_h, u_s, u_v])

mask = cv2.inRange(hsv, l_b, u_b)

cnts, _ = cv2.findContours(mask,cv2.RETR_TREE,cv2.CHAIN_APPROX_SIMPLE)

for c in cnts:

area=cv2.contourArea(c)

if area>2000:

cv2.drawContours(frame,[c],-1,(255,0,0),3)

M=cv2.moments(c)

cx=int(M["m10"]/M["m00"])

cy=int(M["m01"]/M["m00"])

cv2.circle(frame,(cx,cy),7,(255,255,255),-1)

cv2.putText(frame,"blue",(cx-20, cy-20),cv2.FONT_HERSHEY_SIMPLEX, 1, (0,0,255),2)

res = cv2.bitwise_and(frame, frame, mask=mask)

cv2.imshow("live transmission", frame)

cv2.imshow("mask", mask)

cv2.imshow("res", res)

key=cv2.waitKey(5)

if key==ord('q'):

break

cv2.destroyAllWindows()

Now make sure you have changed the l_h, l_s, l_v, u_h, u_s & u_v variables with the values of LH, LS, LV, UH, US, and UV from the previous code. And run the program. Similar output must be visible.

Congratulations you have detected the blue color. Similarly, you can detect any specific color or colors by adjusting HSV values and creating as many contours.

This is how you can use the ESP32 CAM along with OpenCV for Color Detection or Tracking. This project can be implemented at an advanced level along with the application of Machine Learning & AI.

Video Tutorial & Guide

Color Detection & Tracking with ESP32 Camera & OpenCV

Watch this video on YouTube.

View 13 Comments

13 Comments

Witek on August 26, 2022 9:19 AM

Unfortunately ide compiling reports errors: error: ‘esp_camera_sensor_get_info’ was not declared in this scope camera_sensor_info_t* info = esp_camera_sensor_get_info(&sensor->id);

Axel on September 8, 2022 1:57 AM

Here it is the answer just update and install the arduino-esp32 2.0 version.

https://github.com/yoursunny/esp32cam/issues/26

Timothy Malche on November 27, 2022 6:39 AM

I am getting following error

Traceback (most recent call last):
File “C:\hostedtoolcache\windows\Python\3.8.10\x64\Lib\site-packages\PyInstaller\hooks\rthooks\pyi_rth_multiprocessing.py”, line 17, in
File “PyInstaller\loader\pyimod03_importers.py”, line 540, in exec_module
File “multiprocessing__init__.py”, line 16, in
File “PyInstaller\loader\pyimod03_importers.py”, line 540, in exec_module
File “multiprocessing\context.py”, line 6, in
File “PyInstaller\loader\pyimod03_importers.py”, line 540, in exec_module
File “multiprocessing\reduction.py”, line 16, in
File “PyInstaller\loader\pyimod03_importers.py”, line 540, in exec_module
File “socket.py”, line 49, in
ImportError: DLL load failed while importing _socket: The parameter is incorrect.
[7996] Failed to execute script pyi_rth_multiprocessing
Multiple libraries were found for “esp32cam.h”
Used: C:\Users\User\Documents\Arduino\libraries\esp32cam
Not used: C:\Users\User\Documents\Arduino\libraries\esp32cam-main
Multiple libraries were found for “WiFi.h”
Used: C:\Users\User\AppData\Local\Arduino15\packages\esp32\hardware\esp32\2.0.0\libraries\WiFi
Not used: C:\Program Files (x86)\Arduino\libraries\WiFi
exit status 1
Error compiling for board ESP32 Wrover Module.

foufouner on March 16, 2023 6:57 AM

Is it possible with this technique to move the cursor of a computer mouse?

- Admin on March 16, 2023 10:07 AM
  
  We already developed code and program for that Its posted on our site in the ESP32 CAM Project section.
  
Chirag on March 22, 2023 5:11 PM

Even though COM5 port is connect, I am getting following error:

A fatal error occurred: Could not open COM5, the port doesn’t exist
Failed uploading: uploading error: exit status 2

Jim Walker on April 25, 2023 12:50 PM

lot of nice code here but none of it works…

- Admin on April 25, 2023 7:02 PM
  
  The python library keeps getting updated. That is why it doesn’t work.
  
Jim Walker on April 25, 2023 12:52 PM

Doesn’t work.
Either idle gives me an error that there is no module named cv2
or if I run the code via the command line I get an error from the urllib
Python\Python311\objectdetection.py”, line 27, in
img_resp = urllib.request.urlopen(url)
which leads to an HTTP 404 error: Not Found

Have updates made your code obsolete?

Aiman on May 22, 2023 4:02 AM

how to fix when i get timeout error when connecting stream from esp32 cam to the python

Gregor Peck on June 2, 2023 6:50 AM

Is it hard to provide libs version compatible with this code? Code doesn`t work with :
numpy>=1.21.2
opencv-python-4.7.0.72

jake on June 20, 2023 7:54 AM

can u provide code for multiple colours

Gregg on January 25, 2024 2:51 PM

Although the first board is only $1, you have to pay for shipping from China, which is $10-60 extra, plus it can take up to 20 days to be delivered. And if you make a mistake, the next time you order you need to buy 5 boards, totaling $40 (plus shipping again).

$10+ per board might be worth it if you’re running a class on this topic, but it seems that you need to tell us more about why we’re paying this much and waiting 3 weeks… is it for the experience of learning to order boards, perhaps?