Recruiting users for participation is an important part of usability testing. The recruitment piece could be handled internally or through a vendor. Outlined below are key tasks and considerations for someone recruiting users for a usability testing study: Determine the number of participants needed for the usability test Ideally, a test should be conducted with a minimum of six users to get quantifiable and representative feedback from all user types. However, you need to balance your need for acquiring participants with other practical constraints such as time, resources, and availability of participants. Create (and provide the vendor with) a list of user types User types should be defined based on your site audiences. If the user types are not known or you have difficulty understanding who the end users of your site are, you can look for information about users by:

• reviewing requirements and specification documents that were created when building your website (since your site was created to achieve certain goals for certain types of users);
• referencing marketing studies / documentation related to user profiles for your organization such as the organizational annual report;
• referencing personas for your website, which can inform you about behavioral and habitual patterns about the end users of your site (you may also reference the IEEE personas (PDF, 145KB);
• reaching out to the IEEE Strategic Research team for member-related information.

Create a testing schedule with session slots This schedule is useful for scheduling participants during the recruiting process. While creating a testing schedule, ensure that:

• each test is ideally no longer than one hour;
• there is a sufficient gap between two sessions to allow participants to complete the session, adjust the session script if needed, prepare for the next participant, allow some flexibility for the remote (local or international) participant to set up the remote connection, and allow the participant to arrive to the venue; and
• there are no more than five or six participants scheduled on one day of testing to allow flexibility for script changes and sufficient time for the facilitator to recover between sessions.

Decide honorarium type and value to thank participants for their time Typically, participants need to be compensated with an honorarium for volunteering their time for a usability study. Keep the following guidelines in mind when deciding an honorarium type and value.

• An honorarium can be an incentive in the form of cash, a gift certificate, a gift card/gift checks, or money transfer.
• Honorarium value should be fair and equal, irrespective of age, gender, gender identity, race, ethnicity, culture, national origin, religion, sexual orientation, disability, socioeconomic status, or any basis prescribed by law. The value of the honorarium should be decided based on the length of the session. The budget of the project may also be a factor.
• The method of payment can be determined based on the easiest way to get the payment to the participant based on their geographic location.
• Typically, IEEE employees and volunteers would not receive honorariums; however there may be special situations in which this is appropriate.

Other considerations

• If using a vendor for recruitment, budget more money for the project. During the RFP stage of selecting a vendor, clarify roles and expectations related to recruitment of participants.
• Plan to start recruiting users as soon as the type of testing method and scope are decided.
• Decide incentive/honorarium value before you start recruiting.
• Methods for generating a pool of participants interested include:
  • publishing a sign-up form that can be emailed/advertised to select people from membership or customer base lists;
  • advertising the website sign-up form on the website, official social networks, print magazines; - See sample social media advertising copy for recruiting users (PDF, 78.3 KB)
  • asking employees to reach out to people they know who may qualify and ask them to fill out the form.
• Create a testing schedule with session slots for scheduling participants. (If applicable, collaborate with the vendor to create the test schedule and recruiting criteria.
• From the pool of possible participants (if available), select appropriate people to schedule for testing by sending them a screening questionnaire. The screening questionnaire may contain the following criteria for selection as appropriate:
  • experience with using the Internet or certain computer-related technologies
  • user type (member, non-member, business user, etc.)
  • frequency of use of your website
  • availability
  • See sample screening questionnaire (PDF, 85.8 KB)
• Send screening and recruitment emails from an IEEE employee email address to prove the authenticity of the study. (If recruitment communications are originating from a vendor, ensure that contact information of an IEEE staff member is included in the communication so the authenticity and seriousness of the study is not lost.)
• Send confirmation emails containing session timings, meeting connection instructions, and directions to the study location (unless conducting remote test), to participants selected to participate.
• Schedule backup participants (“floaters”) to cover multiple time-slots in case a participant fails to arrive for the user test session.
• Leave sufficient time between test sessions to avoid more than one participant joining a session while another session is in progress.

These requirements define IEEE expectations for proper conduct with end users and are intended to protect the privacy and interests of those who agree to participate in studies related to IEEE websites. Be honest, fair, and maintain integrity

• Share the same information with all participants to ensure they are informed about the scope and procedures involved in a study.
• Users must not be exposed to any physical, mental, or emotional stress during the study.
• Ensure that the mix of participants represents the audience for that project/website such as age, gender, race, ethnicity, culture, national origin, geographic region, disability, and socioeconomic status.
• No derogatory comments may be made regarding the user, either in public or private.
• Feedback received from users should be fairly reported without bias or attribution to others.
• Necessary precautions and all applicable regulations must be adhered to when working with participants who are elderly, disabled, or children.

Respect confidentiality, privacy, and anonymity

• Information that identifies participants should not be disclosed without their permission and all reasonable precautions should be taken to avoid such information from being disclosed.
• Users must provide informed/written consent before the start of the study. See sample informed consent form (PDF, 79.8 KB).
• Reports, writings, teaching materials, or other public media should not contain information about users or IEEE employees unless reasonable steps have been taken to disguise the identity of the user or IEEE employees, or unless permission has been granted to disclose their identity.

Honorarium and benefits for participation Users who participate in IEEE website studies are usually provided an honorarium. The following requirements must be met if an honorarium is provided:

• Honorarium amount should be fair and equal, irrespective of age, gender, gender identity, race, ethnicity, culture, national origin, religion, sexual orientation, disability, socioeconomic status, or any basis prescribed by law.
• Participants should be aware of whether or not* they will receive an honorarium, and in what form.
• All participants must sign a receipt for any honorarium received from or on behalf of IEEE. See sample honorarium receipt (PDF, 88.7 KB).

* Users who do not complete the study are not eligible to receive an honorarium. Some studies may not be required to provide honorariums, in which case, this should be clearly noted during the recruitment process.

These guidelines should be followed by those who observe* usability tests, either during the session or after (by seeing a recording). Ethical considerations for observing Having observers present during a usability testing session can be stressful for users. Hence, it is important to keep in mind the following considerations:

• Users should be treated with respect at all times.
• Observers should have their phones and microphones muted and should not participate in the discussion at any point.
• Do not discuss users’ personal attributes since it is inappropriate to discuss anything that is not directly related to the work we are carrying out.
• User names should not be included in any notes.
• Be cautious when using chat features within the online meeting tool (for remote studies); ensure that your comments will only be shared with the facilitator and keep the comments appropriate and professional.

Analyzing observed user behavior and comments

• Usability testing is one of several available methods for evaluating websites. Test findings should always be looked at alongside other available site research, data, and analytics and in the context of overall business and user objectives for the site.
• Listen to users as they think and talk aloud. This helps you understand why they are performing certain actions on the site(s) being tested.
• You may see behavior that suggests particular design elements do not work well. However, refrain from making any inferences until all the sessions are complete and the data analyzed.
• You may come across users who do unexpected things or make mistakes. Do not judge them for their inexperience or lack of intelligence based on this, but remember that the purpose of observing users is to learn what parts of the site might be difficult to use or ineffective because of faulty site design.
• Watch for patterns based on responses from all users. Just because one user has a problem with something, that doesn’t mean every user will. Carefully consider why the single user had the problem and consider discarding that finding if it can be easily explained, otherwise, recognize that the site design or functionality may be faulty.
• Users in usability tests with a higher level of understanding about the study will sometimes make observations based on what they think other users would think or find challenging rather than what they themselves are experiencing. Such feedback may be noted as a hypothesis, but should not be considered a task failure for that particular user and should be validated with other users.

Exception: If a single user test uncovers a functional bug, typo, or other site defect, further testing is not required to conclude that action is needed. * An observer could be a business stakeholder or a member of the project team conducting the test. Besides the facilitator who conducts the testing session, there can be any number of observers for a usability test session.