Introduction to Computer Vision¶

This notebook introduces some of the image processing algorithms/functions using the OpenCV library

Function Description
Read Image Read an image using cv.imread
plot_images Plots two images side by side
Convert RGB to grayscale Converts an RGB image to a grascale image
Convert RGB to HSV Converts an RGB image to an HSV image
Corner Detector Uses an algorithm called Harris corner detector to detect edges and corner
Normalize Normalize the values of the image
central_crop Crops the image with the given image dimensions around the center
BGR2RGB Converts a BGR image to an RGB image
resize_shortest_edge While maintaining the aspect ratio of the image, resizes the shortest edge
detect_aruco Detects ArUco markers in an image and draws their corners/ids
detect_and_read_qr Detects and decodes a QR code in an image

Aren't we doing computer vision throughout this bootcamp? Then why have a dedicated session for computer vision?¶

Computer Vision: Traditional method to teach computer understand images

CNNs: human inspired computing paradigm for computers to understand images

OpenCV ¶

OpenCV is an open source cross platform computer vision library. You will be using OpenCV to get familiar with some of the computer vision functions.

Import numpy, OpenCV and matplotlib to visualize images. We will use a mosaic image to demonstrate the effect of some vision operation to such image

In [ ]:
import numpy as np
import cv2 as cv
from matplotlib import pyplot as plt

How to read an image?¶

In [ ]:
img = cv.imread('img/mosaic.jpg')
plt.figure(figsize=(10, 10)), plt.axis("off"), plt.imshow(img);
print(img.shape)

How to plot images next to each other?¶

In [ ]:
def plot_images(original_image, processed_image):
    plt.figure(figsize=(15, 15))
    plt.subplot(121),plt.imshow(original_image),plt.title('Original')
    plt.xticks([]), plt.yticks([])
    plt.subplot(122),plt.imshow(processed_image),plt.title('Processed')
    plt.xticks([]), plt.yticks([])
    plt.show()

How to convert RGB image to Grayscale image?¶

A color image is represented on the RGB color space, however there are many different color spaces. Each of them have a particular purpose. We will explore some of them

A grayscale image has only one channel which represents the amount of light that each pixel contains. One of the main purposes of grayscale images on vision applications is to detect edges on images.

In [ ]:
gray = cv.cvtColor(img, cv.COLOR_RGB2GRAY)
plot_images(img, gray)

How to convert RGB image to HSV format?¶

HSV color space represents an image using the channels hue, saturation and value. This color space aligns a bit better to the way human perceives color-making attributes, and in vision application this color space is useful to detect color more accurately.

In [ ]:
hsv = cv.cvtColor(img, cv.COLOR_BGR2HSV)
plot_images(img, hsv)

Corner Detector¶

Harris Corner Detector¶

The Harris Corner Detector is primarily designed for detecting corner points or keypoints in images and not specifically meant for detecting edges. While it can potentially detect some edges, its primary focus is on identifying corners in the image.

In [ ]:
gray = np.float32(cv.cvtColor(img,cv.COLOR_BGR2GRAY))
harris = cv.cornerHarris(gray,2,3,0.1)
plt.figure(figsize=(10, 10)), plt.axis("off"), plt.imshow(harris, cmap='gray');

How to normalize image?¶

Normalize the image by dividing by 256, then subtracting the mean i.e. 0.5, and then amplifying the values by multiplying by 2.

In [ ]:
def normalize(image):
    image_norm=image/256.0
    image_norm=image_norm-0.5
    image_norm=image_norm*2
    plot_images(image, (image_norm + 1) / 2)  # rescale to [0,1] just for display, avoids imshow clipping warning
    return image_norm
In [ ]:
norm_image = normalize(img)

How to central crop image?¶

Reduce the size of the image around the center.

In [ ]:
def central_crop(image, crop_height, crop_width):
    image_height = image.shape[0]
    image_width = image.shape[1]
    offset_height = (image_height - crop_height) // 2
    offset_width = (image_width - crop_width) // 2
    image_crop = image[offset_height:offset_height + crop_height, offset_width:
                 offset_width + crop_width, :]
    plot_images(image, image_crop)
    return image_crop
In [ ]:
cropped_image = central_crop(img, 213, 320)
print("Size of original image: ", img.shape)
print("Size of cropped image: ", cropped_image.shape)

How to convert BGR image to RGB image?¶

In [ ]:
def BGR2RGB(image):
    B, G, R = cv.split(image)
    processed_image = cv.merge([R, G, B])
    plot_images(image, processed_image)
    return processed_image
In [ ]:
rgb_image = BGR2RGB(img)

How to resize the shortest edge of an image?¶

In [ ]:
def resize_shortest_edge(image, size):
    H, W = image.shape[:2]
    if H >= W:
        nW = size
        nH = int(float(H)/W * size)
    else:
        nH = size
        nW = int(float(W)/H * size)
    processed_image = cv.resize(image,(nW,nH))
    plot_images(image, processed_image)

    return processed_image
In [ ]:
processed_image = resize_shortest_edge(img, 200)
print("Size of original image: ", img.shape)
print("Size of processed image: ", processed_image.shape)

How to detect ArUco markers?¶

ArUco markers are square, black-and-white fiducial markers, each encoding a unique binary id. They are widely used in robotics and augmented reality for camera pose estimation and object tracking because they are fast and reliable to detect, even from far away or at an angle.

Every marker belongs to a dictionary, a set of markers that share the same grid size (e.g. 6x6 bits) and a fixed number of unique ids. OpenCV's cv.aruco module can both generate and detect these markers.

Reference: OpenCV: Detection of ArUco Markers

In [ ]:
# pick a predefined dictionary of 6x6-bit markers (250 unique ids)
aruco_dict = cv.aruco.getPredefinedDictionary(cv.aruco.DICT_6X6_250)

# generate an image of marker id 23 and add a white quiet-zone border around it,
# real markers need this border so the detector can find their black outer edge
marker_image = cv.aruco.generateImageMarker(aruco_dict, 23, 200)
marker_image = cv.copyMakeBorder(marker_image, 20, 20, 20, 20, cv.BORDER_CONSTANT, value=255)

plt.figure(figsize=(4, 4)), plt.axis("off"), plt.imshow(marker_image, cmap='gray')
plt.title('ArUco marker, id 23');

Now let's detect the marker in that image! cv.aruco.ArucoDetector looks for markers from the chosen dictionary and returns the pixel corners and id of each one it finds.

In [ ]:
def detect_aruco(image, aruco_dict=aruco_dict):
    detector_params = cv.aruco.DetectorParameters()
    detector = cv.aruco.ArucoDetector(aruco_dict, detector_params)
    corners, ids, rejected = detector.detectMarkers(image)

    processed_image = cv.cvtColor(image, cv.COLOR_GRAY2BGR) if image.ndim == 2 else image.copy()
    cv.aruco.drawDetectedMarkers(processed_image, corners, ids)
    plot_images(image, processed_image)
    return corners, ids
In [ ]:
corners, ids = detect_aruco(marker_image)
print("Detected marker ids:", ids)

How to detect and read QR codes?¶

QR (Quick Response) codes are 2D barcodes that can encode text, URLs, and other data. OpenCV's cv.QRCodeDetector class can locate a QR code in an image and decode the text stored inside it.

Reference: Detect and read QR codes with OpenCV in Python | note.nkmk.me

In [ ]:
# generate a QR code encoding a URL
qr_encoder = cv.QRCodeEncoder.create()
qr_image = qr_encoder.encode("https://www.amd.com")

# the encoder returns a tiny 1-pixel-per-module bitmap, so scale it up and add a
# white quiet-zone border, the same way a printed QR code would have one
qr_image = cv.resize(qr_image, (200, 200), interpolation=cv.INTER_NEAREST)
qr_image = cv.copyMakeBorder(qr_image, 20, 20, 20, 20, cv.BORDER_CONSTANT, value=255)

plt.figure(figsize=(4, 4)), plt.axis("off"), plt.imshow(qr_image, cmap='gray')
plt.title('QR code');

Now let's detect and read that QR code! cv.QRCodeDetector.detectAndDecode() finds a QR code in an image, decodes the text stored inside it, and gives us back the four corner points of the code.

In [ ]:
def detect_and_read_qr(image):
    qr_detector = cv.QRCodeDetector()
    data, points, _ = qr_detector.detectAndDecode(image)

    processed_image = cv.cvtColor(image, cv.COLOR_GRAY2BGR) if image.ndim == 2 else image.copy()
    if points is not None:
        processed_image = cv.polylines(processed_image, points.astype(int), True, (0, 255, 0), 3)

    plot_images(image, processed_image)
    return data
In [ ]:
decoded_data = detect_and_read_qr(qr_image)
print("Decoded data:", decoded_data)

Summary¶

Let's summarize, in this notebook we

  • learned how to read, display, and plot images side by side with OpenCV and matplotlib
  • converted images between color spaces (RGB to grayscale, RGB to HSV, and BGR to RGB)
  • used the Harris Corner Detector to find corners in an image
  • normalized, center-cropped, and resized images
  • detected ArUco markers and read their ids
  • detected and decoded QR codes

✨ Bonus Challenge! ✨¶

Here is an optional challenge for you to work through!

  • Connect a USB Camera to your board and capture a live video feed instead of using the static mosaic.jpg image. Try applying some of the functions you learned in this notebook (grayscale conversion, HSV conversion, corner detection, normalization, central crop, BGR2RGB, and resize_shortest_edge) to each frame of the live video!
    • hint: you can use cv.VideoCapture(-1) to open the camera, then call cap.read() in a loop to grab each frame, just like the view() function in the PYNQ 201 - MNIST notebook
    • hint: try using IPython.display and ipywidgets (a display_handle and a stop ToggleButton) to show the live video feed in the notebook and give yourself a way to stop the loop, the same way it was done in PYNQ 201 - MNIST
    • hint: instead of showing the original frame, try encoding the processed frame (e.g. the grayscale or HSV version) with cv.imencode('.jpeg', processed_frame) so you can watch the effect update live!