1. In addition to receiving information from the onboard sensors, the F-35 receives off-board tracks and
measurements from the Link 16 datalink and the Multifunction Advanced Data Link (MADL). Designed for 5th
Generation aircraft, MADL provides fusion-quality data on all air and surface tracks to other members of the flight
group. These data include the track state, track covariance, identification features, and passive RF data.
The amount and fidelity of the off-board information provided by MADL was one of the largest challenges for the
fusion design. The capability of the sensors and information sharing across MADL presented a challenge for sensor
fusion. The challenge was to ensure that the tracks displayed were real and not duplicated, which would result in
display clutter. The last few software builds in the System Development and Demonstration (SDD) phase of the F-35
program were aimed at tackling display clutter problems. The objective was to ensure that the pilot had accurate and
timely information to make real-time tactical decisions in the cockpit.
This paper discusses the design, development, and verification of each of these systems, as well as the system of
systems integrated into the F-35 aircraft.
The EO DAS integration began with a single sensor installation within a pod. This pod was mounted on an F-16 to
support initial testing and data collection for image processing algorithm validation. This podded system was also
mounted on a QF-4 drone to enable testing of the missile warning function. The next step in integration was to mount a
sensor in an integration-representative fashion on the Northrop Grumman-owned BAC 1-11 flying testbed. The first
introduction of multiple EO DAS cameras into the integrated avionics system was performed on the Lockheed Martin
CATB platform. This marked the beginning of integrating the EO DAS sensors into the Lockheed Martin-developed
fusion algorithms. The final step to fully incorporate the EO DAS into the integrated avionics system came in March
2011, with the first flight testing on an F-35.
Key EO DAS operational functions in Block 1 of Flight Test Update B are navigation forward-looking infrared
(NAVFLIR) and missile warning. Block 2 of Flight Test Update B added surface-to-air missile (SAM) launch point
reporting and situational awareness IRST. These EO DAS functions are available simultaneously and serve to enhance
situational awareness and defensive response.
The fusion functionality is divided into two major sub-functions: air target management
(ATM) and surface target management (STM). The purposes of these functions are to optimize the quality of air and
surface target information, respectively. Their functionality is implemented in three primarily software modules: the
A/A tactical situation model (AATSM), the A/S tactical situation model (ASTSM), and the sensor schedule (SS).
The AATSM software module receives data from onboard and off-board sources about air objects in the
environment. It then integrates this information into kinematic and identification estimates for each air object.
Similarly, the ASTSM software module receives data from onboard and off-board sources about surface objects in the
environment. It then integrates this information into kinematic and identification estimates for each surface object.
Objects that are ambiguous between air and surface are sent to both tactical situation models (TSMs). Each TSM
assesses the quality of its tracks to identify any information needs. The system track information needs (STINs) are
sent from the TSMs to the SS software module. The SS prioritizes the information needs by track and selects the
appropriate sensor mode command to issue in order to satisfy the information need. The SS provides the autonomous
control of the tactical sensors to balance the track information need and the background volume search needs.
Measurement and track data is sent to fusion from the onboard sensors (e.g., radar, EW, CNI, EOTS, DAS) and
off-boards sources (e.g., MADL, Link 16). When this information is received at the TSM, the data enter the data
association process. This process determines whether the new data constitute an update for an existing system (fusion)
track or potentially new tracks. After being associated with a new or existing track, data are sent to the state estimation
to update the kinematic, identification, and rules of engagement (ROE) states of the object.
Kinematic estimation refers to the position and velocity estimate of an object. It can also include an acceleration
estimate for maneuvering air track. The kinematic estimate also includes the covariance for the track, an estimate of
the track accuracy. Identification estimation provides an estimate and confidence of the affiliation, class, and type
(platform) of the object.
The identification process also evaluates the pilot-programmable ROE assistant rule to
determine when the sensing states and confidences have been met for declaration. Estimation publishes the updated
track state (kinematic, identification, and ROE statuses) to the system track file. At a periodic rate (about once a
second), each track is prioritized and then evaluated to determine whether the kinematic and identification content
meets the required accuracy and completeness. Any shortfall for a given track becomes STINs. The STIN message
for the air and surface tracks are sent to the SS to make future tasking decisions for the onboard sensor resource. The
process continues in a closed-loop fashion with new pieces of data from the sensors or datalinks. Figure 15 illustrates
this process
2. The terms data fusion, sensor fusion, and information fusion are often used interchangeably, and yet these terms
have subtle distinctive connotations within the community. The Joint Directors of Laboratories (JDL) Data Fusion
Model defines a useful categorization of fusion algorithms and techniques used in the solution of many general fusion
problems [11]. They define data fusion as the combining of information to estimate or predict the current or future
state of the environment. Level 1 fusion is focused on object assessment. Level 1 fusion algorithms include: (1) data
association algorithms, which determine whether information from multiple sources describes the same object; and (2)
state estimation algorithms, which estimate the current (and, in some cases, future) state of the physical object in the
environment. The estimate includes both the kinematic state (e.g., position, velocity) and an estimate of the object’s
identification (ID). Level 2 fusion focuses on aggregating the Level 1 objects, inferring relationships between/among
the objects and corresponding events, and assessing the unfolding situation. Level 3 fusion assesses the impact of
perceived, anticipated, or planned actions in the context of the unfolding situation, for instance, in terms of lethality
and survivability. Level 4 fusion is focused on process refinement, including sensor resource management or sensor
feedback to modify sensor actions and refine the overall situational picture.
5th Generation aircraft are designed to process the sensor measurements rather than the sensor tracks, resulting in
an integrated system track containing the most precise track accuracy and enabling cooperative sensing across aircraft.
Measurement-level processing can provide earlier discovery of objects in the environment that are hard to detect. By
processing the measurement-level data, the system can use detections from any sensor (or aircraft) to confirm a track
before any single sensor can make the declaration. The focus on the measurement data rather than track data also means
that combat ID information from a sensor is retained by the system track, even when the track is no longer in the
sensor’s field of view since the system track can be maintained by other sensors or aircraft.
In addition to improved accuracy and detection performance, the introduction of an Autonomous Sensor
Management capability provided the ability to react and refine objects in the environment much faster than any human
could respond [14]. The addition of the Autonomous Sensor Manager is referred to as Closed Loop Fusion. This
capability provides the fusion process a feedback loop to coordinate the actions of the sensors in a complementary way
to detect, refine, and maintain tracks based on system priorities [15]. The sensor management capability evaluates each
system track, determines any kinematic or ID needs, assesses those needs according to system track prioritization, and
cues the sensors to collect the required information. Analogous to John Boyd’s Observe, Orient, Decide, and Act
(OODA) Loop [16], which expressed the engagement advantage related to the pilot’s ability to understand and react
to an adversary, closed loop fusion accelerates the ability of the pilot to understand and respond to an object in space
faster and often at a much greater range than legacy systems.
IV. The F-35 Information Fusion Approach
Prior to the introduction of the 5th Generation fusion systems, fusion historically only referred to the data
association and estimation processes. The earliest partitioning of the F-35 fusion capability envisioned the sensor
management capability to be independent of the fusion process. However, there was already strong evidence that the
autonomous sensor manager was fundamental to efficient fusion performance and sensor optimization. During the
early stages of design, the sensor manager was repartitioned to the F-35 fusion design. Figure 4 shows the top-level
functional architecture of the F-35 fusion design, highlighting the data association, estimation (both kinematic and ID),
and sensor management functionality
The F-35 Information Fusion design isolates fusion algorithms from both the sensor and datalink inputs, as well as
any consumers of fused data. Essentially, the fusion algorithms comprise a black box, known internally as the fusion
engine, and sensor inputs and data consumers are encapsulated in external software objects known as virtual interface
models (VIMs). For incoming data, the sensor-specific or datalink-specific VIMs fill in missing data (e.g., navigation
state, sensor bias values), preprocess the information, and translate it into a standard form for the fusion process. For
data leaving fusion, the outgoing VIM, known internally as the fusion server, provides data to the various consumers
of fused information, both onboard and off-board. The fusion server isolates users of the fused information from both
the fusion process and data sources. Legacy fusion implementations reported fusion tracks as a monolithic block (i.e.,
one size fits all) where all data consumers received the same message. Any propagation of the data or conversion was
the responsibility of the recipient. This created a coupled interface between fusion and the data consumers. When a
new data source was introduced to fusion, the interface changes to make this data available impacted all consumers of
that message, whether the data was used or not, making changes to fusion very costly. The fusion server sends each
information consumer a tailored message that contains only the information required to support that consumer. This
isolates that consumer from changes to any data source or to the fusion algorithm. The use of VIMs enables the fusion
architecture to be extensible to new sensors and data sources, as well as new data consumers, over its lifetime.
One of the key architecture decisions for F-35 fusion is how to share information among aircraft. Independent data
can be incorporated optimally into a filter for the highest accuracy. However, if dependent data is incorporated under
the assumption of independence, the result will be track instability and, eventually, track loss [18]. Data consumers on
the F-35, including the pilot, receive the kinematic and ID estimate of each track based on all available data sources,
both onboard and off-board. This is referred to as the Tier 3 solution. However, when sharing information with other
aircraft, each F-35 shares the information describing a track based solely on measurements from onboard sensors. This
is referred to as the Tier 1 solution. By ensuring that the information received from MADL is independent, the track
information can be converted into equivalent measurements [19] by the recipient supporting both track-to-track and
measurement-to-track of the information. The sharing of Tier 1 data ensures that the information is not coupled to any
specific fusion algorithm and provides a method for dissimilar fusion platforms to share optimal fusion data in the
future (Fig. 5). In late 2016, Lockheed Martin and the U.S. government used this technique to share an F-35 fused track
of a target drone across MADL to a surface-based weapons system that had no line of sight to the drone....
