Motion Detection Squid Game using ESP32 CAM & OpenCV

Overview

In this article, we will develop a Motion Detection project based on Squid Game using ESP32 CAM & OpenCV. With the help of the Python program and ESP32 Camera Module, we will develop a Red Light – Green Light Game. This game is inspired from a famous Netflix TV Series “Squid Game”. Here, we will capture the frames of the person moving using ESP32-Cam. If there is any motion detected in the live video stream when the red light is turned on then the person is dead or game over else green light is shown in which the person needs to move.

Red Light = No motion & Green Light = Motion

To get started one must have sound knowledge of Python, Image processing, Embedded Systems, and IoT. In this project, we will understand how to detect the motion of a person, and what all requirements are needed to run the python program. First, we will test the whole python script with a webcam or internal camera of a laptop. Later motion detection program is implemented with the ESP32 CAM. So let’s see how we can build a Motion Detection Project like a Squid Game Red Light Green Light.

You can go through the earlier projects where we did Gesture Recognition and also the Face Recognition project using ESP32 CAM & OpenCV.

Bill of Materials

The following is the list of Bill of Materials for making a ESP32 CAM Motion Detection Project. You can purchase all these components from Amazon.

S.N.	Components	Quantity	Purchase Links
1	ESP32-CAM Board AI-Thinker	1	Amazon \| AliExpress
2	FTDI Module	1	Amazon \| AliExpress
3	Micro-USB Cable	1	Amazon \| AliExpress
4	Jumper Wires	10	Amazon \| AliExpress

Motion Detection Squid Game using PC Camera

Before jumping to the project part, lets build a Motion Detection based Squid Game using PC image recognition technology.

Installing Python & Required Libraries

For the live stream of video to be visible on our computer we need to write a Python script that will enable us to retrieve the frames of the video. The first step is to install Python. Go to python.org and download Python latest version.

Once downloaded and installed open to the command prompt. Now we have to install a few libraries. For that run the following commands below one after another until all the libraries are installed.

pip install numpy

pip install opencv-python

pip install mediapipe

pip install playsound==1.2.2

Once you have entered these commands in the command prompt then these libraries will be installed. Now create a new folder.

Test Code/Program

Inside that folder create a new python file and paste the following code below.

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

import mediapipe as mp

import cv2

import numpy as np

import time

from playsound import playsound

cap = cv2.VideoCapture(0)

cPos = 0

startT = 0

endT = 0

userSum = 0

dur = 0

isAlive = 1

isInit = False

cStart, cEnd = 0,0

isCinit = False

tempSum = 0

winner = 0

inFrame = 0

inFramecheck = False

thresh = 180

def calc_sum(landmarkList):

tsum = 0

for i in range(11, 33):

tsum += (landmarkList[i].x * 480)

return tsum

def calc_dist(landmarkList):

return (landmarkList[28].y*640 - landmarkList[24].y*640)

def isVisible(landmarkList):

if (landmarkList[28].visibility > 0.7) and (landmarkList[24].visibility > 0.7):

return True

return False

mp_pose = mp.solutions.pose

pose = mp_pose.Pose()

drawing = mp.solutions.drawing_utils

im1 = cv2.imread('im1.jpg')

im2 = cv2.imread('im2.jpg')

currWindow = im1

while True:

_, frm = cap.read()

rgb = cv2.cvtColor(frm, cv2.COLOR_BGR2RGB)

res = pose.process(rgb)

frm = cv2.blur(frm, (5,5))

drawing.draw_landmarks(frm, res.pose_landmarks, mp_pose.POSE_CONNECTIONS)

if not(inFramecheck):

try:

if isVisible(res.pose_landmarks.landmark):

inFrame = 1

inFramecheck = True

else:

inFrame = 0

except:

print("You are not visible at all")

if inFrame == 1:

if not(isInit):

playsound('greenLight.mp3')

currWindow = im1

startT = time.time()

endT = startT

dur = np.random.randint(1, 5)

isInit = True

if (endT - startT) <= dur:

try:

m = calc_dist(res.pose_landmarks.landmark)

if m < thresh:

cPos += 1

print("current progress is : ", cPos)

except:

print("Not visible")

endT = time.time()

else:

if cPos >= 100:

print("WINNER")

winner = 1

else:

if not(isCinit):

isCinit = True

cStart = time.time()

cEnd = cStart

currWindow = im2

playsound('redLight.mp3')

userSum = calc_sum(res.pose_landmarks.landmark)

if (cEnd - cStart) <= 3:

tempSum = calc_sum(res.pose_landmarks.landmark)

cEnd = time.time()

if abs(tempSum - userSum) > 150:

print("DEAD ", abs(tempSum - userSum))

isAlive = 0

else:

isInit = False

isCinit = False

cv2.circle(currWindow, ((55 + 6*cPos),280), 15, (0,0,255), -1)

mainWin = np.concatenate((cv2.resize(frm, (800,400)), currWindow), axis=0)

cv2.imshow("Main Window", mainWin)

#cv2.imshow("window", frm)

#cv2.imshow("light", currWindow)

else:

cv2.putText(frm, "Please Make sure you are fully in frame", (20,200), cv2.FONT_HERSHEY_SIMPLEX, 0.8, (0,255,0), 4)

cv2.imshow("window", frm)

if cv2.waitKey(1) == 27 or isAlive == 0 or winner == 1:

cv2.destroyAllWindows()

cap.release()

break

frm = cv2.blur(frm, (5,5))

if isAlive == 0:

cv2.putText(frm, "You are Dead", (50,200), cv2.FONT_HERSHEY_SIMPLEX, 2, (0,0,255), 4)

cv2.imshow("Main Window", frm)

if winner == 1:

cv2.putText(frm, "You are Winner", (50,200), cv2.FONT_HERSHEY_SIMPLEX, 2, (0,255,0), 4)

cv2.imshow("Main Window", frm)

cv2.waitKey(0)

Adding Images & Audio Files

Copy the following images below in the same folder and name them im1.jpg and im2.jpg respectively.

Also, download the following audio files in the same folder and rename them as greenlight.mp3 and redLight.mp3 accordingly.
1. Audio Red Light: Download
2. Audio Green Light: Download

Motion Detection Algorithm & Testing

Now to detect motion we are going to use mediapipe library using this we are able to find landmarks of a person standing in front of the camera as in the image below.

Here to detect motion we are going to use landmark no’s 24,23,28, and 27. Because if the person moves then the distance between his hip and alternate foot increases or decreases. For example, if the person moves then the distance between 23 and 28 or 24 and 27 surely changes. Thus, the motion is detected.

Now after arranging all files in a folder run the python code and the game would work for you.

Now that our Python code is running, then we move towards our firmware.

ESP32 CAM Module

The ESP32 Based Camera Module developed by AI-Thinker. The controller is based on a 32-bit CPU & has a combined Wi-Fi + Bluetooth/BLE Chip. It has a built-in 520 KB SRAM with an external 4M PSRAM. Its GPIO Pins have support like UART, SPI, I2C, PWM, ADC, and DAC.

The module combines with the OV2640 Camera Module which has the highest Camera Resolution up to 1600 × 1200. The camera connects to the ESP32 CAM Board using a 24 pins gold plated connector. The board supports an SD Card of up to 4GB. The SD Card stores capture images.

To learn in detail about the ESP32 Camera Module you can refer to our previous Getting Started Tutorial.

ESP32-CAM FTDI Connection

The board doesn’t have a programmer chip. So In order to program this board, you can use any type of USB-to-TTL Module. There are so many FTDI Module available based on CP2102 or CP2104 Chip or any other chip.

Make a following connection between FTDI Module and ESP32 CAM module.

ESP32-CAM	FTDI Programmer
GND	GND
5V	VCC
U0R	TX
U0T	RX
GPIO0	GND

Connect the 5V & GND Pin of ESP32 to 5V & GND of FTDI Module. Similarly, connect the Rx to UOT and Tx to UOR Pin. And the most important thing, you need to short the IO0 and GND Pin together. This is to put the device in programming mode. Once programming is done you can remove it.

Project PCB Gerber File & PCB Ordering Online

If you don’t want to assemble the circuit on a breadboard and you want PCB for the project, then here is the PCB for you. The PCB Board for ESP32 CAM Board is designed using EasyEDA online Circuit Schematics & PCB designing tool. The PCB looks something like below.

The Gerber File for the PCB is given below. You can simply download the Gerber File and order the PCB from ALLPCB at 1$ only.

Download Gerber File: ESP32-CAM Multipurpose PCB

You can use this Gerber file to order high quality PCB for this project. To do that visit the ALLPCB official website by clicking here: https://www.allpcb.com/.

You can now upload the Gerber File by choosing the Quote Now option. From these options, you can choose the Material Type, Dimensions, Quantity, Thickness, Solder Mask Color and other required parameters.

After filling all details, select your country and shipping method. Finally you can place the order.

You can assemble the components on the PCB Board.

Installing ESP32CAM Library

Here we will not use the general ESP webserver example rather another streaming process. Therefore we need to add another ESPCAM library. The esp32cam library provides an object oriented API to use OV2640 camera on ESP32 microcontroller. It is a wrapper of esp32-camera library.

Go to the following Github Link and download the zip library as in the image

Once downloaded add this zip library to Arduino Libray Folder. To do so follow the following steps:
Open Arduino -> Sketch -> Include Library -> Add .ZIP Library… -> Navigate to downloaded zip file -> add

Source Code/Program for ESP32 CAM Module

The source code/program ESP32 CAM Motion Detection can be found in Library Example. So go to Files -> Examples -> esp32cam -> WifiCam.

Before Uploading the code you have to make a small change to the code. Change the SSID and password variable and in accordance with your WiFi network.

Now compile and upload it to the ESP32 CAM Board. But during uploading, you have to follow few steps every time.

Make sure the IO0 pin is shorted with the ground when you have pressed the upload button.
If you see the dots and dashes while uploading press the reset button immediately
Once the code is uploaded, remove the I01 pin shorting with Ground and press the reset button once again.
If the output is the Serial monitor is still not there then press the reset button again.

Now you can see a similar output as in the image below.

So setting up ESP32-CAM part is done now. The ESP32-CAM is transmitting the live video, make sure that you copy this IP address displayed.

Python Code + Motion Detection ESP32 CAM

Now we again come back to our python code and do some changes by or simply paste the code given.

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

import mediapipe as mp

import cv2

import numpy as np

import time

from playsound import playsound

import urllib.request

#cap = cv2.VideoCapture(0)

url="http://192.168.1.61/cam-hi.jpg"

cPos = 0

startT = 0

endT = 0

userSum = 0

dur = 0

isAlive = 1

isInit = False

cStart, cEnd = 0,0

isCinit = False

tempSum = 0

winner = 0

inFrame = 0

inFramecheck = False

thresh = 180

def calc_sum(landmarkList):

tsum = 0

for i in range(11, 33):

tsum += (landmarkList[i].x * 480)

return tsum

def calc_dist(landmarkList):

return (landmarkList[28].y*640 - landmarkList[24].y*640)

def isVisible(landmarkList):

if (landmarkList[28].visibility > 0.7) and (landmarkList[24].visibility > 0.7):

return True

return False

mp_pose = mp.solutions.pose

pose = mp_pose.Pose()

drawing = mp.solutions.drawing_utils

im1 = cv2.imread('im1.jpg')

im2 = cv2.imread('im2.jpg')

currWindow = im1

while True:

#_, frm = cap.read()

#rgb = cv2.cvtColor(frm, cv2.COLOR_BGR2RGB)

img_resp=urllib.request.urlopen(url)

imgnp=np.array(bytearray(img_resp.read()),dtype=np.uint8)

frm=cv2.imdecode(imgnp,-1)

rgb = cv2.cvtColor(frm, cv2.COLOR_BGR2RGB)

res = pose.process(rgb)

frm = cv2.blur(frm, (5,5))

drawing.draw_landmarks(frm, res.pose_landmarks, mp_pose.POSE_CONNECTIONS)

if not(inFramecheck):

try:

if isVisible(res.pose_landmarks.landmark):

inFrame = 1

inFramecheck = True

else:

inFrame = 0

except:

print("You are not visible at all")

if inFrame == 1:

if not(isInit):

playsound('greenLight.mp3')

currWindow = im1

startT = time.time()

endT = startT

dur = np.random.randint(1, 5)

isInit = True

if (endT - startT) <= dur:

try:

m = calc_dist(res.pose_landmarks.landmark)

if m < thresh:

cPos += 1

print("current progress is : ", cPos)

except:

print("Not visible")

endT = time.time()

else:

if cPos >= 100:

print("WINNER")

winner = 1

else:

if not(isCinit):

isCinit = True

cStart = time.time()

cEnd = cStart

currWindow = im2

playsound('redLight.mp3')

userSum = calc_sum(res.pose_landmarks.landmark)

if (cEnd - cStart) <= 3:

tempSum = calc_sum(res.pose_landmarks.landmark)

cEnd = time.time()

if abs(tempSum - userSum) > 150:

print("DEAD ", abs(tempSum - userSum))

isAlive = 0

else:

isInit = False

isCinit = False

cv2.circle(currWindow, ((55 + 6*cPos),280), 15, (0,0,255), -1)

mainWin = np.concatenate((cv2.resize(frm, (800,400)), currWindow), axis=0)

cv2.imshow("Main Window", mainWin)

#cv2.imshow("window", frm)

#cv2.imshow("light", currWindow)

else:

cv2.putText(frm, "Please Make sure you are fully in frame", (20,200), cv2.FONT_HERSHEY_SIMPLEX, 0.8, (0,255,0), 4)

cv2.imshow("window", frm)

if cv2.waitKey(1) == 27 or isAlive == 0 or winner == 1:

cv2.destroyAllWindows()

#cap.release()

break

frm = cv2.blur(frm, (5,5))

if isAlive == 0:

cv2.putText(frm, "You are Dead", (50,200), cv2.FONT_HERSHEY_SIMPLEX, 2, (0,0,255), 4)

cv2.imshow("Main Window", frm)

if winner == 1:

cv2.putText(frm, "You are Winner", (50,200), cv2.FONT_HERSHEY_SIMPLEX, 2, (0,255,0), 4)

cv2.imshow("Main Window", frm)

cv2.waitKey(0)

Make sure that you have updated the IP address in the URL variable, if done then you are good to go.

Now after connecting your ESP32-Cam module and laptop where we are having our code, with the same local wifi. We run the python code and a similar image is shown below. Thus, our ESP32-Cam-based game is ready.

Here, the frames of the person moving are captured using ESP32-CAM and if there is any motion detected in the live video stream when the red light is turned on then the person is dead or game over else green light is shown in which the person needs to move.

This is how you can build a Motion Detection-based Squid Game using ESP32 CAM & OpenCV can bu be developed.

Video Tutorial & Guide

DIY Red Light Green Light Squid Game with ESP32 Camera & OpenCV

Watch this video on YouTube.

View 1 Comment

1 Comment

pln on December 5, 2022 2:29 AM

Hello, say me pls how fix this error? tnx

line 122, in
mainWin = np.concatenate([cv2.resize(frm, (800, 400)), currWindow], axis=0)
File “”, line 180, in concatenate
ValueError: all the input arrays must have same number of dimensions, but the array at index 0 has 3 dimension(s) and the array at index 1 has 0 dimension(s)