watonomous.github.io

[ Software Division : Data Synthesis ]

Created by [ Rowan Dempster] on Dec 28, 2019

Data Synthesis - OpenGL rendering

This method uses Python bindings for the 3D graphics library OpenGL in order to render 3D models of traffic signs. The PyOpenGL was chosen because it made it easy to integrate with other libraries such as OpenCV and Tensorflow.

Installation:

The package has one system dependency, which is GLFW. It can be installed using the following commands

sudo apt-get update
sudo apt-get install libglfw3
sudo apt-get install libglfw3-dev

It was then made to be installable to virtual environments. To do do, use a tool like pip:

pip install --upgrade /path/to/perception-year-2/tf_object_detection/dataset-preprocessing/render

In doing so, it will also install all other Python dependencies for it.

Usage:

There are some examples on how to use this pacakge in `perception-year-2/tf_object_detection/dataset-preprocessing/render/tests```. In short, the goal is to render 1 traffic sign, and then to composite it on top of another image from another roadway dataset.

To load a sign:

import synthesis.scene as dsScene

mesh_actor = dsScene.loadOBJ(obj_filepath, mtl_filepath, texture_filepath)

To transform it into a location in 3D space, the GLM library can be used:

import glm

mesh_actor.xform = glm.mat4(1.0)
mesh_actor.xform = glm.scale(mesh_actor.xform, (x_scale, y_scale, z_scale))
mesh_actor.xform = glm.rotate(mesh_actor.xform, angle, glm.vec3(x_component, y_component, z_component))
mesh_actor.xform = glm.translate(mesh_actor.xform, (x, y, z))

Then, to get the final image, create a compositor object, and provide the mesh actor. The compositor can be thought of as containing layers that are rendered on top of each other. Each layer can have an image and a 3D scene in it.

import synthesis.composite as dsComp

# Create compositor object
c = dsComp.Compositor(xres=960, yres=540)

# Create layer
sign_layer = dsComp.layer(name='sign', img_path=bg_img_path, xres=960, yres=540)

# Add a 3D model to the scene in the layer
sign_layer.scene.add_actor(mesh_actor)

# Set the compositor's layers
c.layers = [sign_layer]

# The compositor renders out a final image and the bounding box of all elements in the scene.
final_img, bbox = c.comp()

Known bugs :(

UVs on the objects are not working, so the material uses world space to sample the texture. This means all 3D objects should be within the bounding volume of [(0,1),(0,1),(0,1)]
Only supports triangle meshes at this point

Roadmap:

Explore using Unreal Engine to replace GLFW/OpenGL, it is much easier to prototype materials, different 3D models and different lighting.
Maybe shift compositing to OpenGL instead of Python OpenCV to improve performance.

Document generated by Confluence on Dec 10, 2021 04:01

Atlassian