Learning with FR-1 and Rei is a dialogue series where FR-1, an AI robot, and Rei, a university student, explore the World Robot Summit (WRS) 2025 Fukushima.
In this episode, they discuss one of the competitions in the WRS2025 Harsh Environment F-REI Challenge—the Standard Performance Evaluation Drone Challenge (STM Challenge)—explaining its purpose, significance, and rules.
FR-1： Hello again, Rei! Today, let’s dive into the Standard Performance Test Method Challenge, or STM for short.
This one’s a bit different from the flashy races like HEDC.
Rei ： Hi, FR-1! The STM Challenge has “performance test” in its name, but what’s its actual goal?
FR-1： Yes. The primary objective of the STM Challenge is to promote social implementation—practical use— of
drones in harsh environments.
Specifically, it evaluates how well drones can operate under various challenging conditions, such as those
expected at disaster sites.
Rei ：So “practical use” means developing drones that are truly useful when a disaster strikes, right?
Then, what does “standard performance” mean?
FR-1： That is an excellent question, Rei. This challenge is based on the Standard Test Methods (STM) for performance of disaster
response robots and drones being advanced by the U.S. NIST (National Institute of Standards and Technology), and the drone
performance evaluation methods in constrained environments being developed by NEDO's (New Energy and Industrial
Technology Development Organization) ReAMo project.
By combining these methods, we aim to support practical use by evaluating common foundational performance repeatedly,
reproducibly, at low cost, and quantitatively.
In the future, we are considering expanding these evaluation methods to drone schools and promoting them as de
facto standards, as well as collaborating with international standards like ASTM for de jure standardization.
Rei ： I see! So, instead of just vaguely saying "it's an amazing drone," it'll be evaluated by objective criteria to say "this is what's amazing about it!"
So, what are the rules for the drones participating in this STM Challenge? For example, what kind of aircraft can be used?
FR-1： Let me explain the rules in detail. First, there are no restrictions on the robot's form or number.
Any mobile form that can perform the field tasks, such as crawler-type or drone-type, can participate.
Also, remote-controlled or autonomous types are both possible, but you cannot directly observe the robot or the field during the competition.
Rei ： Oh, so I can't look directly? That sounds challenging! What about size or weight?
FR-1： Yes. The reason you cannot see it directly is that we simulate an actual disaster site.
Regarding the drone itself, the total weight must be 10kg or less.
And, it's not just the robot body, but also all attached equipment, including accessories, must fit within a rectangular prism with a total
of 1.5m on its three sides. However, optional equipment can be attached to each field depending on the competition task.
Rei ： So it has to be quite compact. Are there any precautions regarding flight?
FR-1： Yes, drones with rotating wings must always have guards attached.
Also, one pilot and one navigator (pilot's assistant) are allowed. Pilot changes are permitted during the competition.
And, an important point is that robots with environmental resistance will have their scores weighted.
For example, if you have a certificate of waterproofness or explosion-proofness, your score could double, and even with
your own evaluation, it could be 1.2 times higher.
Rei ： So drones that are strong in harsh environments, like waterproof and explosion-proof, will have an advantage!
What other evaluation criteria are there?
FR-1: The STM Challenge primarily evaluates the following four performance aspects:
1. Mobility Performance (MOB): Evaluates the number of round trips within the competition field.
Scores are doubled when flying under communication delay (COM) conditions, and multiplied by 10 for fully autonomous flight.
2. Exploration Performance (EXP): Evaluates the completion rate of missions.
○ Perception missions include thermal image identification, hazardous material label identification, object motion detection, automatic QR code recognition, magnet detection using a proximity magnetometer, and automatic crack and rust recognition and percentage display.
○ Work missions evaluate pair bucket bottom confirmation, Landolt ring reading inside buckets, contact and collection of samples, wiping tasks, and "perching" which involves stationary work on inclined, vertical, or ceiling surfaces. For challenges that are not autonomously explored, scores are doubled for communication delay and multiplied by 10 for fully autonomous operation.
3. Capability (MAP): Evaluates the generation of maps of the competition field, position estimation and semantic understanding
of exploration targets, and the performance of embedding this information into the map.
Evaluation is conducted in two types: "Real-time Map" immediately after the competition ends, and "Detailed Map" after processing
time, basically requiring a 3D map.
4. Autonomy Performance (AUT): Evaluated according to the level of autonomy in MOB and EXP. Scores for autonomous flight
under communication delay (COM) conditions are doubled, and fully autonomous flight scores are multiplied by 10.
Rei ： So the scores really go up with autonomous flight! It seems they really want robots to think and act on their own.
By the way, what kind of specific fields and situations are assumed for "harsh environments"?
FR-1： Yes, The STM Challenge competition fields are broadly categorized into four types.
● Narrow Space Horizontal Slalom (Figure 1): This field evaluates horizontal movement ability within an enclosed space (2-3m wide, 2-3m high) featuring alternating vertical openings (2-3m high, 1-1.5m wide) on the left and right.
● Narrow Space Vertical Slalom (Figure 2): This field assesses the ability to move vertically while passing through openings in a horizontal wall.
● Truss Field (Figure 3): This truss-like field is composed of combined tetrahedrons and triangular prisms made of pipes, requiring the ability to pass through inclined openings.
● Combined Type (Secret Field): This field combines elements from the three basic types mentioned above or uses unique structures, and it will be revealed on the day of the competition.
All of these fields may involve dynamic environmental changes.
Rei ：Wow, so there are all sorts of situations, like going through narrow passages in enclosed spaces,
and moving through slanted areas!
That really is a harsh environment. And I also heard that "harsh environmental factors" might be added?
FR-1： Indeed. The STM Challenge already includes harsh factors like narrow and enclosed spaces, and non-GPS environments.
In addition to these, the following factors are being introduced or considered:
● Narrow Apertures (TRA): Drones will pass through narrow triangular or square apertures based on the shapes of openings created by fire rescue breaching.
● Wind (WND): Fans will be used to create wind in specific segments of the field, introducing disturbances to the drones.
● Enclosed Space: Specific segments will be enclosed by walls, requiring tasks to be performed amidst air turbulence caused by rotor wash.
● Dust (DST): The scenario will involve paper debris and other items being kicked up by propeller downwash.
● Smoke: Fog machines or dry ice will be used to fill the competition field with smoke, creating a low-visibility environment.
● Strong Light: Strong light sources will be placed, or natural light utilized, for operations in bright conditions.
● Darkness: Certain segments will be darkened, requiring movement and exploration in the dark.
● Negotiate (NEG): Movable walls, strings, or flexible cable-like obstacles will be placed in apertures, and the ability to make contact with them to create space while passing through will be evaluated.
● Rain (Drip-proof/Waterproof): The competition will, in principle, proceed even in rain, so waterproofing measures are essential. If drip-proof/waterproof capabilities can be demonstrated, the score will be doubled.
● Flammable Substances (Explosion-proof): Assuming the presence of flammable substances or explosives, if explosion-proof performance can be demonstrated, the score will be doubled.
Rei: That’s impressive—flying through smoke, wind, and even tangled cables.
It really feels like a real disaster site! Are there any other rules for safely competing?
FR-1: Yes. Safety is also extremely important.
Hot and Cold Zones: The competition field has "Cold Zones" and "Hot Zones." Cold Zones are safe areas where drones can take off and land. Repairs, battery changes, and operator changes can be done here, but the competition time won't stop. Hot Zones are disaster areas where repairs or battery changes are not allowed.
Resets and Penalties: If a robot becomes inoperable or it's difficult to continue the competition, a "reset" can be requested, allowing human intervention to get it back on track. However, this comes with a penalty of a 1 to 2 minute interruption in the competition before it can resume.
Damage and Disqualification: Penalties such as point deductions or disqualification will be imposed for damage to the field or drone crashes.
Protective Gear: Team members are required to wear appropriate protective gear such as helmets and goggles.
Rei ：Amazing, so many tough conditions are anticipated!
But that just means it's a true test of technology, right?
I really understand the purpose and rules of the STM Challenge now. Thank you, FR-1!
FR-1：You're very welcome, Rei!
I hope this STM Challenge contributes to the international standardization of drone performance evaluation and the development of future disaster response robots.
Figure 1. Structure and path of the Narrow Space Horizontal Slalom
Figure 2. Structure and path of the Narrow Space Vertical Slalom
Figure 3. Conceptual diagram and example of a truss field (ReAMo exchange meeting)
*Figures 1-3 are quoted from World Robot Summit Tough Environment F-REI Challenge Standard Performance Evaluation Drone Challenge Competition Rules Ver. 1.1.