Synthetic Datasets Through 3D Rendering in Blender

Creating a dataset tailored to a specific need can sometimes be difficult and time-consuming. Specifically for image datasets, you may be constrained by copyright and image rights, lack the equipment to take photos, or simply not have the time to label the dataset by hand.

This is where synthetic data comes in, generated from scratch and labeled automatically. Today I'm going to show you how I generated a synthetic dataset using Blender.

Blender is a free and open-source software for 3D modeling, animation, and rendering (among other things).

The goal was to create a training dataset for object detection, specifically for recognizing pedestrians and pallet jack vehicles. This article details the training of a neural network with this dataset: Fine tuning for pedestrian and vehicle recognition in a warehouse

Extract from the created dataset, on the left an input image and on the right the mask describing the objects to recognize

To reproduce this result, you'll need basic knowledge of Blender and the ability to run a python script.

3D Modeling

I started by modeling a warehouse in 3D. I modeled the simplest 3D objects myself and used royalty-free assets for the more complex ones. In particular, I used:

• Mixamo for characters and their animations
• Poly Haven for HDRIs and textures
• Sketchfab for a pallet jack model
• TurboSquid for a pallet model

Animation

I then animated the camera, a character, and a pallet jack so that the final dataset would contain varied viewing angles of the objects to be recognized.

To do this, I first gave the objects a simple animation, then in the graph editor, I applied the noise modifier to the X and Y position.

(1) Graph editor; (2) Add a modifier; (3) Noise

With this modifier, there are some frames where the objects go off-screen or overlap. I had to bake the animation so I could edit only the problematic frames.

Rendering the Images

At this point, I was able to render my images:

I then moved on to rendering the masks.

Compositing the Masks

A mask is a black-and-white image where each pixel's value represents an object to be recognized. So we'll create images where the background has a value of 0, a pallet jack has a value of 1, and a pedestrian has a value of 2.

To do this, the following render settings need to be changed:

Being able to differentiate objects by their materials

View Layer > Passes > Data > Material Index

Removing Blender's artistic color filters

Render > Color Management > View Transform = Standard / Look = None / Sequencer = Non-Color

Switching the render mode to Black and White

Output > Output > Color = BW

I enabled the Material Index pass to differentiate my objects through their materials. This means assigning a different index to the materials of the objects I want to distinguish (0 for the background, 1 for the pallet jack, and 2 for the pedestrian).

By selecting an object: Material > Settings > Pass Index

Now that the material indices are configured, we can use them in the compositing graph.

1. I use the ID Mask node to get masks that separate the material indices.
2. I combine these masks and assign them values of 0, 0.5, and 1. This lets me visualize the masks with visible colors.
3. I map the values 0, 0.5, and 1 to the values 0, 1/255, and 2/255 using a Color Ramp node in continuous mode.

Blender encodes red, green, and blue color values as floating-point values between 0 and 1, but the image we want to create will be encoded in black and white as integer values between 0 and 255. That's why in the Color Ramp node, I divide the desired values by 255.

Unfortunately, when I inspect the rendered masks, I see that the background is represented by pixels with color values 0 and 1; the pallet jack by pixels with color values 12, 13, 14; and the pedestrian by pixels with color values 21, 22, 23.

I didn't find a solution to this problem, so I processed the masks with a python script:

Processing the Masks

mask_processor.py

# imports
from PIL import Image
import os
import argparse

#### Parse arguments
parser = argparse.ArgumentParser(
prog='ProgramName',
description='What the program does',
epilog='Text at the bottom of help')
parser.add_argument('--source', type=str, default="", help='Source folder')
parser.add_argument('--output', type=str, default="processed_masks", help='Output folder')
args = parser.parse_args()
source_folder = args.source
output_folder = args.output
if not os.path.exists(source_folder) or len(os.listdir(source_folder)) == 0:
print('Given source folder does not exist or is empty')
exit
if not os.path.exists(output_folder):
os.makedirs(output_folder)
print(f'Created output folder {output_folder}')

#### Count the occurence of color in the input imagges
color_count = {i: 0 for i in range(256)}
for filename in os.listdir(source_folder):
img = Image.open(os.path.join(source_folder, filename))
px = img.load()
for x in range(img.size[0]):
for y in range(img.size[1]):
color_count[px[x, y]] += 1

#### Use the color count to identify the ranges of clustered colors
ranges = []
current_range = []
for i in range(256):
if color_count[i] > 0:
current_range.append(i)
elif len(current_range) > 0:
ranges.append(current_range)
current_range = []

#### Process the images to map the fuzy values to distinct values
for filename in os.listdir(source_folder):
img = Image.open(os.path.join(source_folder, filename))
px = img.load()
for x in range(img.size[0]):
for y in range(img.size[1]):
for color_range in ranges:
if px[x, y] in color_range:
px[x, y] = ranges.index(color_range)
break

img.save(os.path.join(output_folder, filename))
print(f"Processed {filename}")

#### Display ranges info
for color_range in ranges:
print(f'colors {color_range} have been mapped to {ranges.index(color_range)}')

You can use this script with the following command:

python mask_processor.py --source [path to the folder containing your masks]

Conclusion

I created a synthetic image dataset in Blender for training a neural network to recognize specific objects.

You can find the training details in this article: Fine tuning for pedestrian and vehicle recognition in a warehouse - AI Squad by Reboot Conseil (rebootia.com)