Scenario: you're driving to your new job at a university campus and using your Global Positioning System (GPS)-enabled smartphone to tell you-and your car-how to get there. Once you arrive, you begin using building signs to identify your structure of interest. You've parked, made it inside, and are greeted with a building map on a bulletin board that directs you to the particular office you're looking for. Following the map's diagram, you weave through unlabeled corridors, dark stairwells, and a windowless basement hallway to reach your final destination. It's a good thing you arrived early since you got lost a couple of times between the building's entrance and your new office. If only the GPS-enabled device you used to get around outside could tell you where to go (and where you are) inside.
The average person spends 87% of their time indoors1. This likely includes a combination of time inside a residence-such as a private home-or in a commercial or public building such as a business, school, museum, or shopping mall. Yet, the GPS technology we've come to depend on for reliable and accurate location and navigation outdoors is completely ineffective in such buildings due to obstructions from rooftops, walls, and windows. This issue, combined with the lack of widely-available indoor maps, has made indoor navigation capabilities a significant challenge.
For firefighters, this challenge is especially critical; the lack of accurate indoor building maps means firefighters are much more likely to get lost and disoriented. In addition to trying to navigate an unknown building-just as the new hire was in our scenario above-firefighters are moving with limited visibility, wearing up to 50 lbs of gear, and breathing only a finite amount of oxygen. Seconds count.
The world needs reliable indoor maps not just to aid in normal circumstances, but for emergency scenarios in which indoor mapping could prevent injury or death.
Indoor Maps Aren't Always Available
The current process for creating indoor maps of buildings for public safety often begins with pre-incident planning2. Pre-incident planning involves the collection and storage of critical site data and characteristics about target hazard sites by emergency responders, and can improve the effectiveness of responses to terrorist and emergency incidents. In the pre-incident planning process-perhaps during building inspection-firefighters will walk a building and may draw a map by hand as they're moving through. They're noting everything from the building's dimensions, to the locations of elevators and fire pumps, to hazardous materials and water supply information, and more. This manual process is extremely time-consuming, and nearly impossible to replicate every time a building's features change. Commercial building space alone comprises 5.6M commercial buildings and 87B square feet of floor space in the United States3, so there's an enormous amount of ground to cover. While three-dimensional (3D) scanners could help automate the process of making indoor maps, a properly annotated dataset-one tailored to public safety- is still needed.
That's why, in 2018, NIST's Public Safety Communications Research Division (PSCR) launched the Point Cloud City award, intended to accelerate research related to indoor mapping for public safety use cases. PSCR funded three projects-City of Memphis (TN), Enfield Fire Dept. No. 1 (CT), and Hancock County(MS)-with the goal of creating annotated point clouds and 360-degree imagery datasets collected from a variety of building types in the award recipient's local area. Starting in late 2020, the datasets from each project will be made publicly available for the research and development (R&D) community for two years. Learn more about accessing the datasets here.
Point clouds are a collection of spatial 3D data points defined by a given coordinates system (typically x, y, z coordinates) used to represent 3D objects and spaces. They can be created using 3D scanners, which have been effective in creating models for architectural purposes or virtual reality (VR) applications. All three awarded projects used lidar technology for scanning indoor structures in 3D.
"Most of the buildings in the United States are not accurately mapped on the inside. So, for a first responder that's rushing into a building to put out a fire or save people from an active shooter, they're kind of going in blind. They don't know what the inside of the building looks like," says Joel Lawhead, Chief Information Officer of NVision Solutions Inc. Mr. Lawhead is a Hancock County, Mississippi contractor on the Point Cloud City award.
Creating 3D Maps with Lidar
"Lidar has been used to map the outdoors since the 90s," says Mr. Lawhead. "You're actually shooting a laser… and the time it takes [the laser] to come back gives you a distance. Because laser is light, and light is the universal constant, we know how fast it travels." He continues "to map an outdoor space, you combine the GPS that tells you where you are on the Earth and the exact point where a laser shot comes back from. This gives you latitude and longitude coordinates. Then, the distance from the airplane you're shooting the laser from usually indicates elevation. We've mapped the outdoors and outdoor terrain for decades that way. The question is, can we do the same thing inside and use lidar to map the inside of buildings?"
"In this project, we're able to walk through the building with a handheld scanning device and we can map the building in a couple of hours," says Dr. Jason Parent, assistant professor at the University of Rhode Island and principal investigator for the Enfield Fire Dept. No. 1 project. "Of course, we have to take the data back to the computer and analyze and process it, which will take another week or two. But, it's much more efficient than previous methods. And so, it'll be great for researchers to incorporate that into other projects."
Moreover, research into lidar use for public safety may improve the efficiency of emergency response. In a future scenario, a fire chief could have access to a 3D indoor building map, giving them visibility into spaces they may be unfamiliar with or are simply difficult to see. These maps could be used during annual pre-incident planning checks to help efficiently identify objects, obstacles, or structural changes on the floor plan. Imagine if a firefighter, crawling on their hands and knees through smoke, had a 3D indoor map projected on a heads-up display-giving them sight and orientation where there previously was none!
Labeling the Data for Public Safety Use Cases
Until Point Cloud City, research into 3D datasets was not geared to public safety. Much of the available imagery excluded data relevant to first responders such as emergency exits, fire extinguishers, and utility shutoffs. The three PSCR-funded projects taking place in Enfield (CT), Hancock County (MS), and Memphis (TN) set out to pioneer use cases with public safety at the forefront of R&D.
"Although there are many existing image datasets for training object recognition," Dr. Lan Wang of the University of Memphis explained, "these datasets do not contain most of the public safety related objects we are interested in…" Think about an app you'd use to identify the kind of plant you're looking at just by using your smartphone camera; no such experience exists for public safety. "This means that we have to collect and annotate these images ourselves, which is very tedious and time consuming, but the resulting dataset can be of great value to the research community." Future research into automating the annotation process may also be on the horizon.
"It's very hard for us to use existing algorithms to detect the small objects that public safety needs," said Dr. Wang. "We picked the best among the existing algorithms, but they're still not perfect. So, one of our future research goals is to develop better artificial intelligence (AI) or machine learning algorithms to detect or identify these objects. And this is a potential groundbreaking research area for AI researchers."
In addition to research and pre-incident planning, there's a future for this technology to aid in training scenarios. For example, point clouds may be used in VR simulations so that firefighters wearing VR goggles could experience a building walk-through with an accurate 3D map and labeled objects and obstacles. These simulations could also provide other relevant data such as directional cues to help firefighters navigate stairwells, storage rooms, etc. VR helps firefighters train in a safe, controlled, and repeatable environment.
The Point Cloud City projects may result in R&D aimed at integrating object identification technology with augmented reality (AR), which could serve as an intuitive data overlay inside of a firefighter's helmet, giving them sight and intelligence in otherwise dark environments devoid of data inputs. Dr. Parent, a former firefighter himself explains, "they'll be able to bring up a floor plan to see where they're located within the building. There could be some feature telling them the closest way out of the building, the emergency exit, the way they came in, how to get from point A to point B, or do searches for different locations… it could make their process more efficient or help them get out of the building in case they have an emergency." Additionally, the fire chief and firefighters outside the building could have a device with a 3D map showing where firefighters are located indoors and which floor they're on. "So, if anything happens to the firefighters and they need to be rescued themselves, the chief could know where to send the rapid intervention team. This is potentially breakthrough technology in terms of [firefighter] safety and their ability to carry out their jobs."
"It's great having [public safety's] feedback because otherwise we're just wasting time and money on solving problems that aren't actually what they face." -Joel Lawhead, NIST PSCR Award Recipient
At a workshop with the Memphis Fire Department, Memphis Police Department, and the city's Geographic Information System (GIS) team, Dr. Wang's team learned which buildings and objects to prioritize for public safety's needs. "They were very excited, and would actually like to see more buildings mapped. They're very interested in getting models of the schools, because that's one of the first things they want to protect." Collaborating with local public safety stakeholders ensured Dr. Wang's team processed the data from the most important buildings first.
Mr. Lawhead said similarly in his interview, "it's great having [public safety's] feedback because otherwise we're just wasting time and money on solving problems that aren't actually what they face." PSCR funding opportunities are designed to ensure collaboration with public safety for this very reason.
Impacting Public Safety and Beyond
When first responders can more safely and efficiently do their jobs, there's a great benefit to the rest of society. The outcomes of using 3D point cloud building models to enhance first responder situational awareness could include reductions in damage, injury, and even death. "As a former firefighter," said Dr. Parent, "I can very much appreciate the goal of the Point Cloud City project to develop indoor navigation systems and I was very pleased to be able to contribute to this important research."
Moreover, the datasets may help communities better develop building information models to ensure safety features are standardized. Dr. Wang's Memphis team submitted a proposal to the National Science Foundation (NSF) on standardizing new building information models, which could impact building managers, city planners, inspectors, and those in construction.
From an R&D perspective, there are broad applications of these datasets as well. One such application could impact robotics. For example, helping autonomous robotic vacuum cleaners or self-driving cars navigate space more efficiently and effectively. Though for now, the focus remains on providing building models with detailed annotations critical to public safety and improving indoor localization/navigation algorithms for location-based services researchers. The datasets and some buildings will be available for future research access for a period of up to two years after the projects end. Learn more about requesting access here.
In the future, perhaps for your next visit to a new office building or school (whenever that happens), maybe you'll have a 3D indoor map helping you reach your final destination without getting lost. Or maybe, the number of firefighters lost or injured due to disorientation in large structure fires will be greatly reduced. With the Point Cloud City R&D and resulting open source datasets, we hope this future can be realized. This is just the beginning.
