Windows and Doors Detection

As part of a project to create models of large buildings (e.g., schools, hospitals) for energy savings, I developed tools to detect windows and doors from images of exterior walls.  The images came from a variety of sources, including handheld cameras and Google street views.  I initially developed the prototype using Python and Open3D, and then assisted in incorporating the functionality into a Unity app.

An example from Google street view is shown below.  This example consists of one image, but multiple images can be combined into a mosaic.  

The first step was to have the user draw four lines of a rectangular area, so that a rectified “orthophoto” of the wall can be created.

Once the four lines are drawn, I calculate the vanishing points and from that, estimate the focal length of the camera and compute the homography matrix to rectify the image.

The user can then adjust the boundaries of the desired wall in the rectified image.

Finally, I run the rectified image through a neural net to detect windows and doors.  The position of each detected window and door on the wall is now known.  I used the “Faster RCNN Resnet50” neural net model from PyTorch.  I trained it on a public dataset of building facades.