Senior Research Scientist

Matthew Johnson is a Senior Research Scientist focusing on human-machine teamwork. He has been at IHMC since 2002 and applied his efforts to technologies such as robotics, artificial intelligence, software agents, drones, and autonomous vehicles, in a variety of domains including disaster response, space applications, aviation, and military operations. His research has made advancements in understanding teamwork, building systems that team, and measuring team performance.

Understanding and Analyzing Teamwork
The essential starting point for human-machine teamwork is understanding teamwork. Johnson’s thesis work proposed teamwork is about managing interdependence between team members[1]. He developed a novel design approach, called Coactive Design, that identified three key interdependence relationships: observability, predictability, and directability. He also developed a unique analysis approach to conducting analysis and an associated tool, called the Interdependence Analysis Table [2][3][4], to identify and understand the interdependence within an activity for a given team. To help jump start one’s thinking from an interdependence perspective, Johnson has provided an Introduction to Coactive Design video, as well as Introduction to Interdependence Analysis video.

Building Systems that Team Effectively
With a proper understanding, the next challenge is to build effective human-machine systems. Johnson and his colleague Micael Vignati have developed Joint Activity Graphs (JAGs) [5][6] as a tool to help understand, model, and implement effective team behavior. JAGS have been used to drive automated teammates’ behavior (e.g., DARPA CREATE). They have also been used to track and predict human teammates’ behavior (e.g., DARPA ASIST).

Measuring Team Performance
Though significant research effort has been focused on measuring team performance, it remains an open challenge that is critical to interpreting the effectiveness of any human-machine team. Johnson and Vignati have instrumented JAGs to not only track team behavior, but measure team performance through a novel measure called Joint Activity Efficiency, which has proven to be an excellent predictor of team performance [7].

Background
Matt came to IHMC out of the military, where he served as a naval aviator, flying both fixed wing aircraft and helicopters. He continued in the Navy Reserves, retiring after 20 years of service. Matt earned his undergraduate degree in aerospace engineering at the University of Notre Dame (go Irish!) prior to entering the Navy. He earned a master’s degree in computer science from Texas A&M – Corpus Christi while on active duty. Matt completed his Ph.D. from Delft University in the Netherlands.

In addition to his research, Matt is the IHMC coordinator for the joint UWF/IHMC PhD program on Intelligent Systems and Robotics and is a UWF Director of Research Professor.

Current PhD students

• Micael Vignati
• Dale Perkins

Current Projects:

• ONR Team Aware Collaborative Assistants
• ARL VERSATILE project focused on assessing technology fluency
• Penn State Applied Research Lab – Cognitive Agents for Command and Control
• ARL Valuation of Expertise with Relevance for Skillful Adaptation of Technologies with Intelligent Learning Elements (VERSATILE)
• NCIA Collaborative Autonomy
• ONR assessing viability of shipboard robotic maintenance
• Virtual Reality Workbench: Leverage VR to derisk new technology concepts
• Working with local SWAT to set up a drone program
• Working with local police department to set up a drone program

Previous Projects:

• DARPA Artificial Social Intelligence for Successful Teams (ASIST)
• COEUS: A Co-Training Methodology for Improved Performance in Human-Machine Teams
• Collaboration and Secure Tasking for Multi-Agent Swarms
• DARPA Context Reasoning for Autonomous Teaming (CREATE)
  • Capture-the-Flag teaming simulation video
  • Scaling Teamwork in Captue-the-flag video
  • Capture-the-flag hardware video
  • Reinforcement Learning simulation video
  • Reinforcement Learning transfer to hardware video
• Worked with Navy Post Graduate School on how to design AI to team more effectively with people
• Worked with Modus Operandi on developing team performance metrics
• Worked with ONR on developing fast behaviors for humanoid robots (Human-Robot Teamwork video)
• Worked on a NASA project to use AI to inform novice technicians how to more effectively use complex sensors
• Worked with AeroVironment to explore the complexities of people controlling large numbers of high altitude vehicles
• Worked with NASA on characterizing the nature of foundational and applied research that need to evolve in order to develop advanced intelligent technologies that reduce cost, enhance performance and improve safety.
• National Robotics Initiative: Toward Humanoid Avatar for Co-exploration of Hazardous Environments. Developing humanoid behaviors and advanced interface concepts to enable complex work using both Boston Dynamics Atlas robot and NASA’s Valkyrie robot.
• Worked with Xerox PARC to brainstorm human-machine teaming concepts
• DARPA Aircrew Labor In-Cockpit Automation System (ALIAS) to develop a robotic copilot (video)
• Nissan Fleet Management for Autonomous Vehicles
• The DARPA Robotics Challenge (DRC) for disaster response (video)
• AFRL Micro-Air Vehicles to develop a novel UAV interface (video)
• NASA rotorcraft noise minimization
• OZ flight display (video)
• DARPA Augmented Cognition project to develop technology to enhance human performance
• NASA Human-Robot Teamwork for exploration
• DARPA Learning Locomotion quadrpedal walking over complex terrain (walking video, technical summary video)
• ARL applying semantic technologies to support information gathering and sharing for the soldier (large video so be patient)
• ONR Coordinated Operations for multiple people working with multiple robots (video)

Presentations available on the web:

• 2022 – DARPA Forward – Advancing the Horizons of National Security- “Systems of Autonomy”
• 2021 – NASA presentation titled “The Unseen m:N Challenge” as part of the NASA m:N working Group
• 2018 – NASA webinar titled “Automation Myths and Human-Machine Teaming Virtues“

Invited Talks:

• 09 NOV 2023: Designing, Building, and Evaluating Machines as Team Players invited speaker at OSU
• 09 MAY 2023: Hybrid Intelligence Center Workshop invited speaker: “Coactive Design” in Vlieland, NL
• 15 MAR 2023: Hybrid Societies keynote: “What it means to work together”
• 10 NOV 2022: SCSP-RUSI Human-Machine Collaboration and Teaming “Current and Future Capabilities”
• 31 OCT 2022: Expert Workshop on the Responsible Use of AI in Military Systems
• 13 SEP 2022: DARPA Forward – Advancing the Horizons of National Security- “Systems of Autonomy”
• 19 OCT 2021: Invited Panelist for NATO Workshop on Meaningful Human Control of AI-based Systems
• 15 SEP 2021: Invited Panelist for NASA Safe Human Expeditions Beyond Low Earth Orbit Workshop
• 30 NOV 2017: Embry-Riddle President’s Symposium: “The Future Role of AI and Autonomy for UAS”
• 22 AUG 2017: HRT Summer School invited speaker: “Human-Machine Teaming”
• 25 MAY 2017: John’s Hopkins Applied Physics Lab: “The Future of Humans & Machines”
• 10 MAY 2017: Organized and ran AFRL Robotics Bootcamp
• 09 APR 2017: Georgia Tech invited speaker
• 30 MAR 2017: CogSIMA invited speaker: “Interdependence as a Framework for Situation Awareness”
• 02 MAR 2017: John’s Hopkins Applied Physics Lab invited speaker
• 29 JUL 2016: Workload Assessment workshop: “New Perspectives on Workload in Human/Machine Teams”
• 17 MAY 2016: FLAIRS Keynote – “No AI is an Island” – Key Largo, FL
• 29 FEB 2016: Presented on Humanoids and Robotics at the PCC Engineering Forum
• 20 FEB 2016: Presented “Rise of the Humanoid Robots” at PensaCon
• 12 NOV 2015: Presented about humanoid robots at Border Sessions – the Netherlands
• 15 OCT 2015: Organized and presented Human & Machine Teamwork in the Air Force Context
• 29 AUG 2015: Tutorial on Coactive Design for NPS – IHMC
• 18 MAR 2014: Presented Coactive Design to NASA’s Future In-Space Operations workgroup – telecon
• 03 FEB 2014: Presented Coactive Design to Marine Corps Warfighting Lab and Naval Post Graduate School
• 19 NOV 2014: Presented at Coactive Design to National Robotics Initiative PI meeting – Washington DC
• 02 OCT 2014: Invited Speaker at Cooperation of Robots and Sensor Networks Summer School – Germany
• 10 JUN 2014: Presented at Interactive Intelligence workshop – the Netherlands
• 28 APR 2014: Class lecture for TU Delft Context project – the Netherlands
• 17 APR 2014: Presented DRC work to NASA Ames Intelligent Robotics Group – NASA Ames
• 04 APR 2014: Plenary talk at Tulane Engineering Forum – New Orleans
• 01 APR 2014: Presented DRC to local Rotary club – Pensacola
• 25 MAR 2014: Presented Coactive Design at HRI – Germany
• 27 MAR 2014: Presented DRC work to II and 3ME groups at Delft – The Netherlands
• 31 JAN 2014: Presented DRC work Association for Unmanned Vehicle Systems International – Pensacola
• 13-17 JAN 2014: Organized and ran Coactive Design workshop for DSO National Laboratories – Singapore

Additional Qualifications:

• Commercial Pilot Ratings
  • Airplane Single Engine Land
  • Rotorcraft-Helicopter
  • Instrument Airplane and Helicopter
• Small Unmanned Aircraft Systems

You can find Dr. Johnson’s CV here.

Outside Interests
When not working on robots for IHMC he enjoys time with his family, playing soccer, traveling, and supporting the Children of Christ Home orphanage in Galette Chambon, Haiti. Matt also developed and maintains medical record software for the medical clinic adjacent to the orphanage. This software is now in two different clinics in Haiti, one in Uganda and another in Liberia.