Overview

The Quality Score is a feature that assesses Learner answering patterns and engagement during the  Aptem Assess cognitive assessments. This feature enables administrators to detect problems that can negatively impact cognitive assessment outcomes and indicate unnecessary additional learning support.

Measurements for Quality Score

Each test area’s (e.g. Symbols) quality score is based on five measurements:

1. Response has consistent alternating pattern: Checks for consistent alternating response patterns

2. Percentage of completed items: Verifies if very few responses were provided

3. Response is the same: Flags responses are consistently the same

4. Response click speed: Verifies if response times are below 1.5 seconds per item

5. Instruction time is low: Checks if instruction viewing times are below 2.5 seconds per item

Scoring Method for Measurements

Issues are produced where there is an indication of concern in the quality of the answering style or engagement.

Warnings are produced where there is an indication the quality or engagement is marginally less than expected.

Warnings and issues both contribute to the overall quality score. The overall quality score provides a single score from 0% to 100%. Higher scores indicate better engagement/answering quality.

Each characteristic is scored as follows:

• 2 points = Issue: When it exceeds the set threshold.
• 1 point = Warning: When it is marginally below the threshold.
• 0 points = Nothing is flagged: When there is no issue or warning detected.

Exceptions include: Response has consistent alternating pattern, which is binary and is always assigned either an issue or nothing value.

Note: 'Response is the same' and 'Response click speed' have higher weightings (3x and 2x, respectively).

Measurement Quality Score Calculation

1. Response has consistent alternating pattern: Checks for consistent alternating response patterns - Issue (2 points) recorded if there is an alternating pattern anywhere

2. Percentage of completed items: Verifies if very few responses were provided -  =>50% = nothing is flagged (0 points), 34%-49% = warning (1 point), anything below is an issue (2 points)

3. Response is the same: Flags responses are consistently the same - <70% = nothing is flagged (0 points), 70%-84% = warning (3 points), =>85% = issue (6 points)

4. Response click speed: Verifies if response times are below 1.5 seconds per item – (formula = minimum answer time of 1.5 seconds, then get the number of answers that have an answer time less than 20% greater than the minimum, then divide that number by the number of items) <60% = nothing is flagged (0 points), 60%-70% = warning (2 points), >70% = issue (4 points)

5. Instruction time is low: Checks if instruction viewing times are below 2.5 seconds per item -(formula = minimum answer time of 2.5 seconds, then get the number of answers that have an answer time less than 20% greater than the minimum, then divide that number by the number of items) <60% = nothing is flagged (0 points), 60%-70% = warning (1 point), >70% = issue (2 points)

Note: Number Recall, Missing Numbers and Objects contain multiple choice questions and are therefore  excluded from 'Response has consistent alternating pattern' and 'Response is the same' scoring.

The maximum possible scores per test area are as follows:

Overall Quality Score Calculation

The overall Quality score for the cognitive assessment is calculated and converted to a percentage using the following formula:

100 - (total measurement score / maximum possible measurement score * 100)

Where total measurement score is a sum of the points allocated of all the issues and warnings detected (if any); maximum possible measurement score is a sum of the maximum possible points allocated for the applicable measurements.

Notes:

1. ‘total measurement score’ and ‘maximum possible measurement score’ are both a sum of the 8 related test area values.
2. If a Learner completes the long version of a test area, this will be used for the quality score, rather than the diagnostic version.

Example:

Cognitive Assessment test area scoring results:

Total measurement score = 0 + 4 + 8 + 0 + 16 + 5 + 3 + 1 = 37

Maximum possible measurement score = 16 + 16 + 8 + 16 + 16 + 8 + 8 + 16 = 104

Quality score = 100 - (37 / 104 * 100) = 64.43 = 64%