The Potent Mix of Machine Learning, Satellite Imagery, Vehicles, and Roads: Diet Hadrade

A lightweight machine learning codebase for aiding HADR (humanitarian assistance and disaster relief)

Humanitarian crises, both natural and man-made, show no sign of abating in the near term given ongoing changes to the planet and society. Yet ever improving machine learning techniques hold the possibility of blunting some of the harshest impacts of future crises. Diet Hadrade provides an open-source, lightweight mechanism for leveraging remote sensing imagery and machine learning techniques to aid in humanitarian assistance and disaster response (HADR) in austere environments.

In a disaster scenario (be it an earthquake or an invasion) where communications are unreliable, overhead imagery often provides the first glimpse into what is happening on the ground, so analytics that utilize such imagery often provide significant value. Specifically, the rapid identification of both vehicles and road networks from overhead imagery allows a host of problems to be tackled, such as congestion mitigation, optimized logistics, evacuation routing, etc. Such challenges often arise in the aftermath of natural disasters, but are also present in crises like the current invasion of Ukraine where roads are choked with civilians fleeing the fighting.

1. Vehicles as Population Proxy

A reliable proxy for human population density is critical for effective response to natural disasters and humanitarian crises. A multitude of efforts (such as SpaceNet) have striven to improve foundation mapping of building footprints from satellite imagery, which can subsequently be used to estimate population densities. Yet while buildings are useful for longer term planning, their static nature means they provide less information on highly dynamic timescales.

# which can subsequently be used to estimate population density estimates, pre/post disaster baselines, and numerous other analytics. Yet while buildings are useful for longer term planning, they rarely provide information on highly dynamic timescales.

Automobiles provide an alternative and quite attractive proxy for human population. Many buildings (e.g. office parks or industrial zones) remain unoccupied for large portions of the week. Furthermore, entire neighborhoods may be devoid of people during crises, as people move to safer locales to avoid floods, fires, conflict, etc. People tend to stay near their vehicles however, so knowledge of where vehicles are located in real-time provides value in disaster response scenarios. In this project, we deploy the YOLTv5 codebase to rapidly identify and geolocate vehicles over large areas. Geolocating all the vehicles in an area helps responders to prioritize response areas and ascertain where resources should ideally be deployed.

2. Road Networks + Vehicles

Vehicle detections yield significantly greater utility when combined with road network data. We use the CRESI framework to extract road networks with travel time estimates, thus permitting optimized routing. The CRESI codebase is able to extract roads using only overhead imagery, so flooded areas or obstructed roadways will sever the CRESI road graph; this is crucial for post-disaster scenarios where existing road maps may be out of date and the route suggested by navigation services may be impassable or hazardous.

Diet Hadrade provides a number of graph theory analytics that combine the CRESI road graph with YOLTv5 locations of vehicles. This allows congestion to be estimated, as well as optimal lines of communication and evacuation scenarios. In the sections below we provide an example of the analytics that can be performed with this codebase. Of particular note, the test city selected below (Dar Es Salaam) is not represented in the training data for either CRESI or YOLTv5. The implication is that the Diet Hadrade methodology is quite robust and can be applied immediately to unseen geographies whenever a new need may arise.

3. Diet Hadrade Part 1: Building the Enhanced Graph

3.1. SageMaker Studio Lab Environment

The deep learning architectures of both YOLTv5 and CRESI require a GPU for rapid inference. Lightweight CPU-only machines will also work in a pinch, but are significantly slower. Luckily, we can easily use the recently released free Amazon SageMaker Studio Lab to quickly set up our environment and run our examples, see diet_hadrade.ipynb for details.

3.2. Data

The imagery used in this blog is available on AWS for free download, and model weights are available on GitHub as well. For this exercise, we explore SpaceNet Area of Interest #10: Dar Es Salaam. This city was withheld for testing purposes in SpaceNet 5, meaning that neither the CRESI nor YOLTv5 models trained on any data from this city. To access the imagery set up the The AWS CLI tool, then simply execute:

aws s3 cp — recursive s3://spacenet-dataset/AOIs/ AOI_10_Dar_Es_Salaam/PS-MS /local_test_imagery_directory

We clip the testing image to a smaller area to make interpreting the results slightly easier, but the codebase will work on arbitrarily large images. See diet_hadrade.ipynb for full details on data preparation. The final testing image has over 130 million pixels and covers 12 square kilometers (Figure 1).

Figure 1. Raw testing imagery of Dar Es Salaam.

3.3. Execute Deep Learning Inference

As an inaugural step, Diet Hadrade executes both YOLTv5 and CRESI on the desired imagery. Inference is rapid, with the 12 square kilometers of our test area taking ~3 minutes on the NVIDIA T4 GPU freely available through SageMaker Studio Lab. Also note that ~1/3 of that inference time is devoted to making the (optional) plots that take some time to render given the volume of imagery. If only vector outputs are required, the process proceeds at a minimum of ~350 km² per hour on free infrastructure. On the command line within StudioLab, simply execute:

YOLTv5

cd /home/studio-lab-user/geodesic/src/yoltv5/yoltv5
JSON=/home/studio-lab-user/geodesic/cfg/yoltv5_test_studio_lab.yaml
time ./test.sh $JSON

CRESI

cd /home/studio-lab-user/geodesic/src/cresi/cresi
JSON=/home/studio-lab-user/geodesic/cfg/cresi_test_studio_lab.json
time ./test.sh $JSON

We now have road networks and vehicle locations in hand, and in the following section we combine these two items and apply a number of graph analytics.

3.4. Locating Vehicles in the Network

We update the CRESI road graph based upon the locations of the 7,900 vehicles detected within the image. The function below takes a mere 7 seconds to run, and gives us our final graph that we can leverage for numerous analytics.

max_dist_m = 5 # maximum distance from a roadway to a car (meters)
G, df_bbox, g_node_props_dic, kd_idx_dic, kdtree = diet_hadrade.build_graph(graph_path, bbox_path, max_dist_m=max_dist_m)

Figure 2 plots a subset of the road + vehicle predictions overlaid on the test imagery (the full test region is large enough that cars are impossible to see).

Figure 2. Inferred roads (colored by speed as yellow (40 kph) and orange (50kph)), and vehicles (colored as car=green, bus=red, truck=blue)

4. Diet Hadrade Part 2: Congestion, Evacuations, and Critical Nodes

4.1. Traffic and Congestion

We can combine the car detections with the road network to infer how congested certain areas are. Congestion information is important for everyday life, but also crucially important in disaster response scenarios when roads may become impassable due to both natural phenomena as well as traffic. We infer congestion based upon the density of cars on a given roadway (Figure 3). This congestion estimate will likely be less accurate than mobile phone-based estimates (e.g. Google Maps), but unlike solutions based upon the movement of mobile phones, requires no proprietary data, and will still work in austere environments without power or functional communications infrastructure.

# Estimate congestion based upon vehicle density
G, edge_update_dict = diet_hadrade.update_G(G)

Figure 3. Road network colored by inferred speed limits (left), and current estimated traffic speed based upon congestion (right). Yellow is slow while red is fast, note that the red portions of the right plot are yellow (slower) when taking congestion into account. The inset zoom illustrates this, showing the detection of dense traffic along one of the main highways.

4.2. Optimal Routing

We can use our final road network to compute optimal routes between nodes of interest. Furthermore, we have flexibility to define “optimal” as distance traveled, travel time based upon road type, or total travel time taking congestion into account. In Figure 4we show the optimal path from the green node to the red node in the northeast corner for three different edge weights. The best path clearly changes when taking road type and congestion into account.

Figure 4. Optimal (shortest) path from the location in green to the red location to the northeast.

While for this blog we focus on figures as a means to illustrate the functionality of Diet Hadrade, the codebase also provides coordinates along the best path for use in navigation systems:

Optimal path coordinates:
  [[-6.7832424 39.2652306 ]
  [-6.78272858 39.26538534]
  [-6.78270691 39.26539187]
  [-6.7826751 39.26540145]
  [-6.78266211 39.26540536]
  …

4.3. Bulk Evacuation

In this scenario we consider a large number of vehicles that need to exit (or enter) an area, such as following a natural disaster or akin to the current conflict in Ukraine. In Figure 5 below we select trucks as the focus, though could just as easily focus on cars, buses, or all vehicles.

Figure 5. Position of vehicles of interest (trucks) highlighted in green.

In Figure 6, we plot the optimal evacuation routes to the northeast for all the detected trucks in our test image.

Figure 6. Optimal evacuation routes for all detected trucks.

4.4. Critical Intersections

Computing the overlap in routes of Figure 6 allow us to determine the intersection of greatest import for the speedy evacuation of all vehicles. In Figure 7 we plot these critical nodes (i.e. intersections) in the road graph for the evacuation scenario of Figure 6, with node size proportional to importance.

Figure 7. Critical intersections (blue) for the truck evacuation scenario, with size proportional to importance.

5. Conclusions

Combining overhead imagery with advanced computer vision techniques and graph theoretic methods allows for a number of interesting analytics. In this blog we showed how this combination can potentially aid in humanitarian assistance and disaster response (HADR) with a relatively lightweight codebase we call Diet Hadrade. The rapid analysis enabled by this codebase requires only overhead imagery and access to inexpensive (in this example, free) GPU resources. We showed an example of detecting roads, cars, and congestion in the previously unseen city of Dar Es Salaam, and computing critical roadways and optimal evacuation routes that avoid gridlock. Unfortunately, exactly such a scenario is currently playing out in Ukraine. In future posts we will discuss a full-fat (i.e non-diet) version of this codebase that provides a dashboard for realtime analysis and exploration. In the meantime, we encourage the interested reader to explore the open source codebase, and take a look at our brief overview video.