Tests is standardized series of problems that assess a person's knowledge, skill, abilities, or other characteristics.

• Legally in the US, the Uniform Guidelines on Employee Selection Procedures (1978) defines a test as any method used to make an employment decision (including hiring, promotion, licensing, and certification)
• E.g., resume reviews are considered test from a US legal standpoint. However, these are typically not standardized in such a way to be considered a test by people involved in the assessment.

Psychometrics of Predictors

Predictor: any variable used to forecast a criterion.

Two psychometric terms that are critical to the use of tests: reliability and validity.

Standardization is also important. The aspects that should be standardized are the following: administration, scoring, and interpretation of scores.

Reliability

Reliability is the degree of dependability, consistency, or stability of scores on a measure (either predictors, criteria, or other variables) used in selection research.

The reliability of a measure is determined by the degree of consistency between two sets of scores on the measure.

• There is an inverse relationship between reliability and error.
• To estimate reliability, examine the relationship between two sets of scores.
  • Index varies between 0.00 to 1.00
  • Higher coefficient means higher reliability.
  • A coefficient hovering near zero indicates a high degree of error, and therefore a low reliability estimate.

What makes a test unreliable?

• testing conditions
• transitory personal factors
• test administration effects
• test scoring (e.g., subjectivity)

Reliability Types

The four types of reliability:

• test-retest reliability (coefficient of stability)
• equivalent-form reliability (coefficient of equivalence)
• internal-consistency reliability (homogeneous content, split half reliability)
• inter-rater reliability (conspect reliability)

Type: test-retest reliability, or the "coefficient of stability". It's an easy and simple way to calculate reliability. However, there's practice effects.

• one group of individuals
• one measure
• administered two times
• correlate scores from time 1 with score from time 2

Type: parallel (equivalent) forms reliability, or the "coefficient of equivalence". It reduces practice effects, but its difficult/expensive to develop equivalent forms.

• one group of individuals
• two (equivalent) forms of measure
• administered 1/2 times
• correlate score on form 1 with the score on form 2

Type: internal consistency reliability looks at reliability from a different angle focused on the homogeneity of items (similarity)

• All these methods can be done in a single administration.
  • Split half reliability. Pros: can use if there's no equivalent forms and no time to retest, Cons: underestimates reliability because it is half of a test.
    • one group of individuals
    • one measure
    • divide item into 2 halves (odd, even)
    • correlate scores from half 1 with scores from half 2
  • K-R 20 (kuder-richardson) used for item scored dichotomously (2 categories). It's the the average of all possible ways of subdividing a measure (every possible split-half). Pro: easiest method, requires single administration. Con: not appropriate for speed tests.
  • Coefficient Alpha: similar to K-R 20 but for items that are scored on a continuum.

Type: Interrater Reliability, or "conspect reliability". It determines consistency among raters; "to what degree do raters agree in their ratings?"

• one measure
• single administration
• give to 2+ raters in one session
• used in subjective situation (rating/judgement)

Reliability Estimates

Reliability estimates can be interpreted as "the extend (in percentage) to which individual differences in scores on a measure are due to "true" differences in the attribute and the extend to which they are due to chance error" $r_{xx} = .90$ means that 90% of the differences in test scores among individuals who took the test is due to true variance and 10% is due to error.

What's a good level of reliability? The reliability coefficient, $r_{xx}$, should be $.70$ or higher for early stages of research, $.80$ for basic research, and $.85$ is the minimum for applied settings. Higher than $.85$ should be the standard.

What are the factors influencing reliability of a measure?

• method of estimated reliability
• individual differences among respondents
• length of a measure
• test question difficulty
• homogeneity of a measure's content
• response format
• administration of a measure.

Validity

Validity is the degree to which evidence supports inferences made from scores on predictors.

• NOT a property of measure itself
• selection measure is neither valid or invalid
• it is the inferences we draw from scores that are valid or invalid
• there could be many validities.

What is the interplay between reliability and validity:

• a test can be reliable but not valid
• a test can never be valid without being reliable
• the square root of a test's reliability sets the upper limit for its potential validity for any use.
• validity could be higher than reliability, but that is never the case.
  • Why? In general, correlating a measure with itself (i.e., reliability) will result in a higher coefficient than correlating a measure with something else (i.e., validity)

Validation Types

Criterion-related validation: correlate scores on a predictor with an evaluative standard for successful job performance. The two approaches: concurrent validation study, predictive validation study.

• Approach: concurrent validation study: first, conduct a thorough job analysis. Then, administer the predictor to current employees. Then, obtain criterion data from the company records (existing data). Lastly, correlate predictor and criterion scores

  • pro: short time frame for results (you have their evaluation data right there)
  • disadvantage: motivation/experience of employees. (are they taking the test seriously if nothing is on the line?)

• Approach: predictive validation study: first, conduct a thorough job analysis. Then, administer predictor to applicants. Select applicants by the existing method, DO NOT USE the predictor. Once job performance is stable (like in a year), then criterion (like evaluation) scores are obtained. Correlate predictor scores from when the person was an applicant with the criterion scores from when they are an employee.

  • pro: less biased sample
  • con: takes > 6 months.

Content validation: whether the predictor contains a representative sample of knowledge, skills, or behaviors performed on the job (job content domain).

• a judgement by subject matter experts, not a statistical procedure
• most psychologists believe content validity is more appropriate as a method of predictor construction, not for validating an existing measure.

The steps to do this:

• conduct a thorough job analysis
• select experts (e.g., people with experience with job)
• experts determine whether the items for a given predictor adequately tap all the important aspects of the job.
• more difficult to use content validity for abstract job functions.

Face validity: appearance of whether a measure is measuring what is intended.

• A selection measure has "face validity" if it appears to the job applicants taking the test that it is related to the job.
• Don't confuse with content validity
• it's not a form of validity technically.
• It's still important! why?

Construct validation: the extent to which a predictor measures the underlying characteristic intended to measure. Construct is a hypothetical attribute which determines overt behavior, which are behaviors that can't be directly observable, like extroversion, motivation, intelligence.

• Can't be accomplished in a single study
• Look for
  • convergent validity: positive correlations with scores on similar constructs
  • divergent validity: zero or negative correlations with scores on opposite constructs.

Neuroticism is an outdated term. Now we use the term "emotional stability." High emotional stability makes good law enforcement officers or air traffic controllers. Low score on neuroticism = high emotional stability, this is an example of divergent validity.

To conduct a study to validate construct:

1. the construct must be carefully defined and a hypotheses is formed concerning the relationships between the constructs and other variables
2. a measure hypothesized to assess the construct is developed
3. studies test the hypothesized relationships between the constructed measure and other relevant variables are conducted.

Validity Coefficient

Interpreting validity coefficient, which can vary from -1.00 to +1.00, summarizes the relationship between scores on a predictor and scores on a criterion.

• the sign dictates the direction of the relationship
• magnitude is the strength of the relationship
• zero means there is no relationship
• the square of the validity coefficient is the coefficient of determination
• can be interpreted as representing the percentage of variance in the criterion that can be explained by variance associated with the predictor.
• For example, $r_{xy} = .60$ and $r_{xy}^2 = .36$ means that 36% of the differences in job performance can be explained by their differences in test scores.

The factors that affect the size of validity coefficients are:

• the reliability of the predictor and criterion
• restriction of range
• criterion contamination.

Incremental Validity

Incremental validity: is the amount of predictive validity one assessment contributes above and beyond that of another. It seeks to answer if a new assessment provides additional value after accounting for what is captured by existing methods

• general cognitive ability tests are often used as the baseline comparison because they are regarded as the most predictive of job performance across a variety of job
• e.g., wonderlic motivating potential assessment has incremental validity over WPT-Q

Ability Test

Selection methods include

• personal interviews (super common)
• ability tests
• personality questionnaires
• behavioral simulations

A predictor measure is an evaluative device in which a sample of behavior is obtained and scored using a standardized process.

• E.g., tests have right or wrong answers while surveys (also called questionnaires or inventories) do not have right or wrong answers.

Test administration issues, there are many different areas to test on.

• speed vs power. The Wonderlic test is a 12 minute test and its very rare that the test is finished. It doesn't mean that someone can't do it, it tests speed not power. In real life, is it common that you are under such a time limit? On the other hand, the SAT is a power test. They chose a time that most people would be able to finish.
• individual vs groups?
• paper and pencil vs performance

Employment tests are used because

• economic return on investment
• objectivity and fairness
• demonstrated predictive validity. But maybe as not as much as previously thought. Recently discovered that previous corrections based on range restriction may have inflated correlation.

Tests have three aspects. And when these three aspects are considered, it should make a good determination of fit for the role.

• abilities (engine, the force within us, how do we do something)
• interests (rudder, direction)
• personality (hull, how we appear) and motivation (fuel)

Issues relating to validity and fairness of ability testing

• psychometric concerns: is the test a good predictor for the performance outcome of interest
  • differential validity of the test. is the test a better predictor of performance for certain groups
  • situational specificity vs validity generalization.
• legal concerns:
  • differential (adverse) impact
  • differential treatment
  • physical ability and medical test use with ADA
• practical concerns:
  • cheating
  • equal access to preparation
  • proctored vs unproctored

Types of Employment Tests

Types of pre-employment tests

• general cognitive ability (verbal, numerical, logical reasoning, problem solving).
  • Pre-2023, it was a general consensus that this is probably the best robust predictor for everything (job, long term success, health outcomes). It's validity was the highest according to some meta-analysis.
  • After, 2023 they re-did the meta analysis and found that they their corrections inflated the correlation. This means that we don't know
• physical ability (strength, endurance, movement quality, static and explosive strength, for gross body coordination, stamina)
  • Results of work analysis should determine how much importance the physical ability test has in overall assessment.
• spatial reasoning
• clerical aptitude
• numerical aptitude
• mechanical comprehension
• psychomotor/mechanical performance
• job knowledge
• situational judgement
• emotional intelligence
• integrity
• personality (self report, projective but this is not used in the industry)

Diving into one of the pre-employment tests: General Cognitive Ability is the ability to learn. It assess a person's aptitude or potential to solve job-related problems.

• Examples: Wonderlic, hogan business test, ONET, ASVAB, SAT, etc.
• What they measure: problem solver, numerical ability, memory, reasoning, perception, verbal ability, problem solving. (Gottfredson, 1997)
• Mental tests or scholastic tests have been criticized as unfair.
• Traditionally recognized as the best overall predictor (r=.51 across jobs). Hesitant now.
• Validity: good predictors of job performance and training success for a wide variety of jobs. They have high criterion-related validity. The more complex the job, the better the test works.
• Face validity: test developed to refer explicitly to specific jobs or types of jobs within the hiring organization are perceived as highly related to the job (high face validity) than commercially developed tests.
• Administration method: paper, pencil, electronic
• Subgroup differences: cognitive ability test typically produce racial/ethnic differences larger than other valid predictors of job performance like biodata, personality tests, and structured interviews. Use in combination with other assessment types (like interviews or biodata instrument) to lower any potential adverse impact.
• Development cost: the cost of purchasing a cognitive test is less expensive than developing a customized test
• Administration cost: generally inexpensive, requires few resources for administration, and does not require skilled administrators
• Utility: high return on investment if you need applicants who possess particular cognitive abilities or have high potential to acquire job knowledge or benefit from training. Cost effectiveness of developing own test over a commercial test is lower when face validity is not an issue.
• Common uses: best used for jobs requiring particular cognitive abilities for effective job performance and complex jobs.
• Advantages: produces valid inferences for organization outcomes, have been demonstrated to predict job performance for more complex jobs, can be administered to large numbers in a cost effective way and doesn't require skilled administers, not influenced by test taker attempts to impression manage or fake responses
• Disadvantages: more likely to differ in results by gender and race than other tests, can be time-consuming to develop

Job Knowledge Tests include knowledge about critical job tasks, procedures etc. needed for the job. Some assess whether you have done each of the tasks or have trained someone else. Credential and licensure tests may indicate job knowledge.

• advantages: useful for roles that involve acquisition of knowledge from specific qualifications or experience. High validity coefficient, not influenced by test taker attempt to impression manage or fake responses. Licensures and credentials are easy to verify
• disadvantages: demonstrates what a candidate already knows, doesn't tell you how fast they will learn or how fast they can apply what they know to a new situation. Like cognitive ability tests, there are significant differences in scores of black-white applicant subgroups.

Example: "The Knowledge" is a London taxi driver test that is one of the most difficult test in the world. It takes 4 years to study all the 320 trips in the "blue book". It has been compared to getting a law or medical degree, and passing gets you the "green badge" which allows you to drive the famous London black taxies.

Personality

Personality, integrity, EQ tests.

Organizations should not depend on one factor when selecting people. They should include all three parts for a holistic assessment:

• IQ predicts a lot of life outcomes.
• EQ
• Personality

Personality Tests

Personality tests measure traits, characteristics, dispositions, tendencies, stable mental and behavioral patterns.

• These are traits that are not temporary. Over time, you would probability be similar
• The strength of your personality can change, but it shouldn't change too much.
• Mood is not a personality

It's popularity has varied over the years.

• The first personality test was to test who would be most likely to have an emotional breakdown.

Personality is measured in two categories:

• Projective techniques (not used in inventory): think of the Rorschach test. These are derived out of psychoanalytic theories. The trend is moving away.
• Self-report inventories: you answer yes/no or 1-5 scale rating.

Self-report inventories: Employers cannot use medical tests/devices, so this limits the options.

• Examples of self-report inventories:
  • The Big Five Dimensions (OCEAN acronym) including the NEO-PI which consists of: 1) openness to experience (arts, entrepreneurship), conscientiousness (double check their work, research, cross t's and dot their i's), 2) extroversion (gregarious, outgoing, easily chat with strangers, ability and enjoyment to interact with others), 3) agreeableness (positive disposition, someone who people want as the greeter of clients), 4) neuroticism/emotional stability.
  • Big 5 project personality test, PIP (big 5 factor marker), Mini-marker test
  • Jungian Personality Typology including the MBTI (Keirsey Temperament Sorter) has extroverted/introverted, sensing/intuitive, thinking/feeling, judging (broadcast EI/SN) /perceiving.
  • Hogan Personality Inventory (HPI), California Psychology Inventory (CPI), Fundamental Interpersonal Relationship Orientation (FIRO-B), Comprehensive Personality Profile (CPP), Guiford-Zimmerman Temperament Survey..
• Good for team building or developmental purpose, but bad for selecting people for a job

Predictive validity for job performance:

• Overall across all jobs, conscientiousness is the highest validity coefficient (people who check their work, dependent, etc.). I The other traits are inconsistent or job specific.
• Some roles have higher dependency on personalities, like:
  • client facing
  • team roles

Limitations:

• difficult to measure
• self-report makes it possible to 'fake good'
• potential legal challenges if questions are invasive or of such a nature that the instrument can be considered a medical device.

Integrity Tests

Overt measures: asses attitude towards and frequency of theft/cheating. Personality based measures: measures traits linked to several related employee behaviors that are detrimental to the organization.

• looks at attitudes towards behavior that would be considered bad

Integrity construct is not well defined:

• may be similar to reliability, conscientiousness, adjustment, sociability
• uncorrelated with general cognitive ability tests.

Validities are difficult to obtain; hard to how how useful it is. However, it is used in banking and retail. Limitations: faking issue similar to personality tests. No adverse impact for specific groups of test takers

Emotional Intelligence

Emotional intelligence or EI is the ability to understand and manage your own emotions, and those of the people around you.

• people with a high degree of EI know what they're feeling, what their emotions mean, and how these emotions affect other people.
• Valuable for supervision and leadership positions
• Some components are: self awareness, self regulation, motivation, empathy, social skills. Each component is explored below:

Self-awareness: the ability to recognize one's emotion and their effect. People who are self-aware...

• Are capable of recognizing and understanding emotions as they experience them
• Realize the links between their feelings and what they think, do and say
• Recognize how their feelings affect their performance
• Work on their weaknesses so they can perform better
• Have a guiding awareness of their values and goals

Regulating/Managing emotions: The ability to control one’s emotions and impulses People who manage their emotions:

• Don’t make impulsive decisions; they think before they act
• Deep disruptive emotions in check
• Are flexible in handling change—they can and will adapt as needed
• Are capable of postponing immediate results for long-term success

Motivation: The ability to use an emotional system to begin a process and keep it going People who are self-motivated:

• Are results-oriented with a high drive to meet their objectives
• Set challenging goals and take calculated risks
• Are ready to seize opportunities
• Operate from hope of success rather than fear of failure
• Are usually more productive

Empathy is the ability to relate well: The ability to engage with people and demonstrate an understanding of their views and feelings. People who relate well:

• Are typically team players and collaborate well
• Balance a focus on tasks with attention to relationships
• Handle difficult people and tense situations with diplomacy and tact
• Are excellent communicators and listen without judgment
• Identify with and understand the wants, needs and viewpoints of others

Social skills: The ability and willingness to help other people learn, grow and become more self-aware through emotional mentoring. Emotional mentors do the following:

• Provide an objective view in a supportive way
• Offer useful feedback
• Know how to help others calm down when emotions are taking over
• Are excellent listeners who can identify the emotion behind the words

Work Samples, SJT Assessment Center

Work Samples

Performance tests/work samples types include: performance or motor (wiring, reading blue prints), written or verbal (oral directions, job knowledge), trainability tests (mini-course designed to determine how well they can learn the material).

• E.g., for the position of a maintenance mechanic a work sample may be to install pulleys and belts or disassemble and repair a gearbox.

Validity of work samples (AVG. r=.38) has high content validity.

• applicants accept the instrument as having "face validity"
• fewer complaints about appropriateness
• realistic job previews result in less turnover
• less adverse impact

Situational Judgement Tests present applicants with a description of a work problem or critical situation related to the job. Then, they ask them how they would handle it. Taps into the following: conflict management, interpersonal skills, problem solving, cultural awareness, good for managers. Other considerations include

• Validity: the tasks and activities described in SJT scenarios are representative of the tasks and activities found on the job (high degree of content validity). Performance on the tests moderately relates to performance on the job (moderately high degree of criterion-related validity).
• Face validity/applicant reactions: applicants often perceive SJTs as being very fair (high face validity)
• Administration Method: paper and pencil, computer-base, or video-based
• Subgroup differences: typically moderate. Racial differences in test scores may be smaller than those typically observed for cognitive ability tests.
• Development costs: less than high-fidelity alternatives like assessment centers. Depends on the cost related to the use of SME.
• Administration costs: typically are low when delivered via paper and pencil, but may be more costly via computer or video. No special administrator expertise is needed.
• Utility/ROI: high return on investment if you need applicants who possess a high level of social/interpersonal skills upon entry into job; if the skills measured by the test can be learned on the job or are not highly critical, then the return on investment will be significantly lower.

Assessment Centers

Assessment Centers are work samples for managers/leaders. Characteristics of assessment centers:

• assessments done in groups
• evaluation done in groups
• uses multiple methods like standardized tests, situational exercises, in-depth interview, case analysis.

The following are situations that can be given out in assessment centers:

In-basket situational exercise presents applicants with a description of a manager's job. You get an "in-basket"full of memos, letters, requests, and various action items, that range from situations where no actions is appropriate to complex problems requiring immediate action. These are 1-2 hours long and impossible to complete. The manager must decide what to do and typically responds right on the original memo. Then, a follow up interview is performed to get details and rationale on why the manager made those decisions. Taps into the following criterias: organizing/planning, creativity, accuracy of decisions.

Leaderless group discussions: with 4-6 managers, each person has a position to uphold with no leader identified. Roles can also be defined/undefined. The discussion is often competitive in nature and each manager wants his or her position adopted by the group. Taps into communication, leadership, energy.

Oral presentation forms consist of press conference exercises, formal presentation, informal presentations (e.g., briefing exercise). Candidates are given a brief period of time to plan and organize their thoughts and make notes. Follow questions are asked. Taps into: communication, poise, confidence, job knowledge.

Case analysis/management game: interviewee is given capital to set up business, they can buy raw materials, manufacture products, and make decisions. The goal is to maximize profit. Taps into: planning, organizing, decisiveness, resilience, analysis and problem solving.

Less used methods of assessment:

• interest inventories
• drug testing
• medical exam (post-job offer)
• graphology

Considerations:

• Validity: good job predicting occupational success (criterion-related validity), but the level of predictive validity can depend on the purpose, assessor training, and assessment methods used. There is little evidence that assessment centers provide useful information about the relative strength and weaknesses. They are useful for making selection decisions but not helpful for comprehensive developmental feedback.
• Face validity/applicant reaction: applicants react favorably and perceive it as fair. The exercises simulating actual job tasks provide effect job previews.
• Administration method: assess groups at the same time so you can assess team environment.
• Subgroup differences: no performance difference found between gender or race.
• Development cost: costly to develop in terms of time and money. Requires frequent updates because they are remembered by the applicants.
• Administration costs: expensive to administer because requires several accessors.
• Utility/ROI: productivity gains realized by selecting managers and skilled individuals average well above administrative costs.
• Common use: promotion or selection purposes. Most commonly used to assess candidates for leadership, managerial, customer service, and sales positions. Requires pre-screen to limit the number of applicants scheduled for labor-intensive assessment center process.