Fire Safety Equipment Detection with Keypoint R-CNN

The problem

Look at an escape plan on a wall. A fire extinguisher symbol seems straightforward — until there is no room to draw it where the extinguisher actually belongs. The icon moves elsewhere, with a line or marker pointing back to its location.

For a computer vision model, that creates a subtle problem: finding the right symbol can still mean returning the wrong position. I explored how to extract both from photographs of escape plans, as a step towards turning visual documents into structured building information.

What I built

A dataset from real plans. I collected and annotated images, largely photographs with varied lighting and viewing angles. The labels captured equipment types and their positions, not just the symbols themselves.
A model for the right question. I adapted an existing Keypoint R-CNN implementation, compared two annotation strategies, and trained and evaluated the resulting models. Keypoints let the model predict specific points alongside a detected region.
A way to inspect the result. I built a graphical interface to load plans, overlay predictions, zoom into details and adjust which detections are shown. This made missed equipment and ambiguous predictions easier to examine.

The decision that changed the approach

My first approach placed a tight box around each symbol and asked the model to predict its associated location. But the implementation predicted keypoints inside the detected box. A location outside that box was out of reach, even when the symbol was detected correctly.

I changed the annotations: the box now covered both the symbol and its location, with five points marking the four symbol corners and the equipment position. This gave the model a region in which it could express the answer.

01 · Find the symbol

02 · Include the location

The same plan, two annotation strategies. The dashed box defines the region available for keypoint prediction. Original schematic, not a floorplan or model prediction.

The lesson went beyond this particular model: the way a problem is labelled can matter as much as the model chosen to solve it. Here, changing the training target addressed a limitation that parameter tuning alone could not fix.

What the results showed

The revised approach could associate offset symbols with their positions on the plan. It also exposed the difficult cases: tightly packed symbols, overlapping regions and less common equipment types. Wall hydrants were underrepresented in the data, making them a particular challenge.

The 2023 evaluation reported 84.4 keypoint AP and 62.5 bounding-box AP for the selected configuration. These are evaluation scores on a 0–100 scale, not percentages of equipment correctly located. The small validation set makes them a useful prototype result, with broader testing still needed.

How to read the evaluation

The report used 169 training images and 10 validation images. The selected configuration scored 84.41 keypoint AP and 62.48 bounding-box AP under an adapted COCO-style evaluation. Keypoint AP evaluates all five annotated points together; it does not measure the equipment-position point alone.

These are the reported validation results from 2023, not a new run of the subsequently modified code or a result on an independent test set. They should not be compared directly with scores from unrelated datasets.

What I took away

This project connected problem definition, data preparation, model evaluation and interface development. A recognised symbol is only useful if the information extracted from it answers the actual task. Building a way to inspect predictions helped make that distinction visible.

The longer-term use case is to support equipment inventories or digital building models. This project reached a detection and review prototype; integration into building models and validation for operational use would be further work.

Foundations and credits

Model foundation: Kaiming He, Georgia Gkioxari, Piotr Dollár and Ross Girshick, Mask R-CNN, 2017, including its extension to keypoint prediction.
Implementation: Torchvision’s Keypoint R-CNN. My contribution was adapting the existing model to this task, building the data and evaluation workflow, and developing the review interface.

Curious about the choices behind the project? Let’s talk. My visual navigation project explores another question about images: what should an agent learn to see before it learns where to go?