CSS-IMDE

How might we help federal, state, and local agencies improve their communication and ability to plan and schedule for inter and intra agency missions?

 

My Role

User Researcher

Puget Sound was selected as the trial area for the CSS-IMDE system as it is home to an international boarder, large scale ports, and shipping lanes.

Puget Sound was selected as the trial area for the CSS-IMDE system as it is home to an international boarder, large scale ports, and shipping lanes.

Product

Coastal Surveillance System and Integrated Maritime Domain Enterprise (CSS-IMDE) is a web based enterprise architecture application that is being developed by the Department of Homeland Security Science and Technology (DHS S&T) division and the Information Sharing Environment (ISE).  The goal of the CSS-IMDE system is to integrate existing federal, state, local, tribal, territorial, and foreign surveillance infrastructure and assets to detect, track, identify, and interdict maritime threats.  The end state of CSS-IMDE will result in increased small vessel interdiction efficiency and effectiveness, which will assist in not only finding more threats, but also allowing DHS to save money by pursuing threats more efficiently.

Processes

Project Declaration

The purpose of this usability study was to introduce CSS-IMDE to regional local, state, and federal agencies to identify opportunities for improvement that might prevent adaptation of the system, bugs, and future usability research areas and topics.

Stakeholders at DHS S&T and the ISE commissioned CSS-IMDE to be developed by SRI International, formally known as Stanford Research Institute, a research and development company that works with U.S. Government agencies, agencies across the globe, and others to develop groundbreaking innovations.

Discover

CSS-IMDE is a new program being introduced to potential users that have been using their current systems or methods to go about their business.  This study would help the DHS S&T, ISE, and SRI International become aware of current issues with the system and how it meets the needs of the various Puget Sound operators.   These research goals were prompted from previous research the DHS S&T, ISE, SRI International, and the University of Washington completed during the life cycle of the CSS-IMDE program.  

University of Washington's role in the CSS-IMDE program was built off of two years of previous research in the area of researching how local, state, federal, and tribal agencies communicated and built relationships with each other through formal and informal means.

Research Goals

  • Evaluate the planning and scheduling widget
  • Identify important operationally relevant information that is missing
  • Assess users ease of planning and scheduling on the CSS-IMDE system compared to their current method
  • Understand user satisfaction and function of system to ensure that CSS-IMDE meets their need
  • Identify software bugs and glitches that would be a show stopper during operational demonstration
 

Methods

Interviewing:  To recruit participants for our usability study we had very specific domain experts we had to target.  Unfortunately a screener would not work.  To find potential users we had a subject mater expert who was able to direct us and help us engage regional players.  Interviews were conducted on 5 diverse, but representative users in which we learned key workflow information which allowed us to engage with other users who would fit our target personas.

Usability Study with Think-Aloud Protocol: By using this method we will be able to observe the necessary steps that they go through to complete each assigned task. The think-aloud method increases the understanding and insight into what the participant was reading, thinking, and expecting from each step they took to complete each task. It would also allow us to follow up with questioning if the participants say something interesting. To ensure that each member of the community was in a comfortable setting in which they could potentially use stimuli outside of the system, we performed each usability test at their place of work.  The usability study allowed us to clearly evaluate the prototype's ease of usage and content.

System Usability Scale & Satisfaction and Function Questionnaire: Participants completed a System Usability Scale (SUS).  The SUS is a well-validated instrument for obtaining a single numeric score to indicate users’ subjective assessment of system usability. Additional subjective measures included participants’ overall satisfaction with the design concept presented by the demo, their behavioral intent to use such a design, and their feelings on trust, privacy, and information sharing and safeguarding associated with the design concept. The open-ended questions focused on what participants liked or found difficult about using the demo and what they might change about this current version. 

Click to expand

Participants

For our study, we tested eight participants. Nielsen's research shows that we will begin to see patterns at three participants and need at least five participants for a study.  Yet, part our research goals was to understand if it met user needs, we decided seven was enough.  Our eighth participant was used as the pilot for our study.

Personas

Our target personas for the CSS-IMDE study was local, state, federal, and tribal government agency operatives who's position put them in the position to plan and schedule the daily activities and larger scale operations of the rest of the department.  Research showed that these were typically male, between the ages of 35 and 60, with very limited exposure to technology (depending on their jurisdiction and funding).

Research Process

Scenario: You receive a phone call requesting support for an event due to occur tomorrow.  This event is the movement of the M/V Polar Pioneer Mobile Offshore Drilling Unit, The Polar Pioneer, which is currently anchored at Terminal 3, Everett, will begin transit toward Alaska.  The Polar Pioneer will transit under tow and will have a 500 yard safety zone around it and a 100 yard safety zone while moored at Terminal 3.  Current planning for tomorrow is available in the CSS-IMDE application.  This event has been given the name “Operation Seagull.”

The scenario was broken down into four tasks:

  • Task 1: Review their agency’s resource schedule for the current day.
  • Task 2: Familiarize themselves with the plan and schedule for an interagency operation scheduled for a future date.
  • Task 3: Decide whether and how their own agency’s resources might be allocated in support of the operation in task 2.
  • Task 4: Schedule any resources they wish to allocate in support of this mission, including assignment of the desired starting location for the mission.

Design Recommendations

While we found many unique usability findings here are some we prioritized:

Priority ratings are an estimate of the issue’s importance to field operations and were based on the frequency with which the issue came up in test sessions and the degree to which it inhibited users in moving through the test case.  These scores do not take technical feasibility into account.

High Priority

  • Finding/User Comment: Users need feedback that tells them that newly assigned resources appear on the map.  Many did not notice this, which brings the risk that they will leave the resource at its default starting position rather than moving it to its accurate starting position.  Also, moving resources around on a map is not always critical for single agency operations at the LLE or USCG station level, so adjusting the position of resources on a map was not intuitive to all users.
    Recommendation: When a new resource is assigned, users should be notified that an icon representing the resource has appeared on the map, and they should be prompted to move it to the correct position, or they should be given the option of adding the resource to the map and adjusting its position.  We need to ask users to help us envision a design that will be intuitive to them. 
     
  • Finding/User Comment: Planners make staffing decisions based on whether certain crew members have the right training and qualifications to use certain resource capabilities (e.g., Rad/Nuc detection equipment) or perform certain operations (e.g., diving).
    Recommendation: In resource details, consider adding what skills and training are needed to operate the resource.This should link to a database that holds personnel training information.  Such a system could support decision making about which crew to assign to a resource or mission by narrowing the list of crew members by availability AND capability.  This requires adding rules re: the training and skills required to undertake certain missions and operate resources.

Moderate Priority

  • Finding/User Comment: To assign a resource to a mission, users frequently tried to drag resources onto the mission name or elsewhere in the mission row outside of the scheduled time window.
    Recommendation: Allow users to assign a resource to a mission by dragging and dropping the resource to anywhere within the mission row. 
     
  • Finding/User Comment: Users wanted to know the status of people in the directory.  I.e., are they online now and available to chat. 
    Recommendation: Consider borrowing Facebook’s convention of a green dot showing who is online now and also indicating how long it has been since a user logged in. 

Low Priority

  • Finding/User Comment: Consider borrowing Facebook’s convention of a green dot showing who is online now and also indicating how long it has been since a user logged in. 
    Recommendation: Add rules to encode fatigue limits into resource inventory.  As resource fatigue limits will change as time progresses and weather changes, this will need to be tied to business rules that have yet to be articulated.
     
  • Finding/User Comment: Often, the same people are scheduled to crew together for weeks at a time. 
    Recommendation: Users need the ability to create and assign set crew groups.

Impact

Challenges and Constraints

  • Stakeholders had their own idea of the requirements that users would need, and had never been introduced to user-centered design.
  • The development team was in Florida and was made up of PM's and Engineer's who where very "requirements" focused.
  • We had to do the tests fast, stakeholders were planning an operational demonstration 3 months after we started.  We where scheduled to run 4 iterations before the demonstration.
  • Participants were hard to schedule.  They each were very busy with their jobs.
  • The participants were generally weary of the system as they, "liked their way of doing it," but even so participants were generally excited about being involved in the process of developing the system unlike every other system the federal government tried to push on them.

Stakeholder Feedback

The significant value of this information has been recognized by SRI International and the sponsors of the greater CSS-IMDE project.  As a result, sponsors have recommended that the iterative prototyping team continue this activity into fiscal year 2016.  The iterative prototyping team completed a project plan for completing no less than four additional iterations in the design of IMDE/CSS planning and scheduling capabilities to be delivered as part of an operational demonstration in late fiscal year 2016.

What would we do differently

  • Bring stakeholders into the usability tests.  Unfortunately, this is very difficult when you go to their offices and the stakeholders are in a different state.

Next Steps

The following is a high-level list of issues to be addressed in future iterations:

  • Interagency resource management
  • Support of mission flow from planning to scheduling to monitoring
  • Transparency and access management of mission components
  • Mission-based communication
  • Geospatial components of planning and scheduling
  • Deconfliction of multiple missions
  • Mission staffing