Development of a New Personality-Oriented Work Analysis Questionnaire: First Steps Towards Validation
DOI:
https://doi.org/10.24837/pru.v19i1.484Keywords:
job analysis, personality traits, job performance, frame of reference trainingAbstract
In this paper we detail the construction and validation process for a new personality-oriented work analysis instrument, in the form of a standardized questionnaire, based on extant research that shows that personality traits are good predictors of job performance. We present the process of item development, frame of reference training, rating scale creation, and the selection of subject matter experts. By administering the instrument to three distinct positions, the interrater reliability coefficients resulted between .80 and .94. We also investigated the instrument’s ability to discriminate between the same rated positions, and the results for this indicator were quite low. Conclusions provide some possible explanations for the lower resulted discriminability. Practical and theoretical implications are discussed as well as other future research for general improvement of data quality.
Downloads
References
Aguinis, H., Mazurkiewicz, M. D., & Heggestad, E. D. (2009). Using web-based frame-of-reference training to decrease biases in personality-based job analysis: An experimental field study. Personnel Psychology, 62(2), 405—438. https://doi.org/10.1111/j.1744-6570.2009.01144.x
Arumugam, S., Ramachandran, K., Bhattacharyya, A. (2014). Suitability screening test for air traffic controllers. Global Journal of Human-Social Science Research, 14(4), 11–18.
Atkins, S. G. (2012). Smartening-up voluntourism: SmartAid's expansion of the Personality focused Performance Requirements Form (PPRF). International Journal of Tourism Research, 14(4), 369-390. https://doi.org/10.1002/jtr.875
Barrick, M. R., & Mount, M. K. (1991). The Big Five personality dimensions and job performance: A meta-analysis. Personnel Psychology, 44(1), 1–26. https://doi.org/10.1111/j.1744-6570.1991.tb00688.x
Brannick, M. T., Pearlman, K., & Sanchez, J. I. (2017). Work analysis. In J. L. Farr & N. T. Tippins (Eds.), Handbook of employee selection (2nd ed., pp. 134–162). Routledge. https://doi.org/10.4324/9781315690193-6
Brutus, S., & Facteau, J. D. (2003). Short, simple, and specific: The influence of item design characteristics in multi-source assessment contexts. International Journal of Selection and Assessment, 11(4), 313–325. https://doi.org/10.1111/j.0965-075X.2003.00254.x
Castille, C. M., Castille, A. M. R., & Smith, R. W. (2019). Assessing ideal personalities at work: Is it all just a little bit of history repeating? Industrial and Organizational Psychology, 12(2), 133-137. https://doi.org/10.1017/iop.2019.25
Cecil, C. A. (2015). Effects of information processing strategies on rater motivation in job analysis [Doctoral dissertation, Cleveland State University]. EngagedScholarship@CSU. https://bit.ly/2OysTq2
Conrad, M., & Schweizer, K. (2018). Personality oriented job analysis to identify non-cognitive factors for a doctor of physical therapy program in the United States. Journal of educational evaluation for health professions, 15, 34-34. https://doi.org/10.3352/jeehp.2018.15.34
Costa Jr, P. T., McCrae, R. R., & Kay, G. G. (1995). Persons, places, and personality: Career assessment using the Revised NEO Personality Inventory. Journal of Career Assessment, 3(2), 123-139. https://doi.org/10.1177/106907279500300202
Costa, P. T., Jr. (1996). Work and personality: Use of the NEO PI-R in industrial/organizational psychology. Applied Psychology: An International Review, 45(3), 225–241. https://doi.org/10.1111/j.1464-0597.1996.tb00766.x
Costa, P. T., Jr., & McCrae, R. R. (1992). Revised NEO Personality Inventory (NEO–PI–R) and NEO Five Factor Inventory (NEO–FFI): Professional manual. Psychological Assessment Resources.
Costa, P. T., Jr., & McCrae, R. R. (1995). Domains and facets: Hierarchical personality assessment using the Revised NEO Personality Inventory. Journal of Personality Assessment, 64(1), 21–50. https://doi.org/10.1207/s15327752jpa6401_2
Cucina, J. M., Martin, N. R., Vasilopoulos, N. L., & Thibodeuax, H. F. (2012). Self-serving bias effects on job analysis ratings. The Journal of Psychology, 146(5), 511-531. https://doi.org/10.1080/00223980.2012.656155
Cucina, J. M., Vasilopoulos, N. L., & Sehgal, K. (2005). Personality-based job analysis and the self-serving bias. Journal of Business and Psychology, 20(2), 275-290. https://doi.org/10.1007/s10869-005-8264-2
Debusscher, J., Hofmans, J., & De Fruyt, F. (2017). The multiple face(t)s of state conscientiousness: Predicting task performance and organizational citizenship behavior. Journal of Research in Personality, 69, 78-85. https://doi.org/10.1016/j.jrp.2016.06.009
Dierdorff, E. C., & Rubin R. S. (2007). Carelessness and discriminability in work role requirement judgments: Influences of role ambiguity and cognitive complexity. Personnel Psychology, 60(3), 597–625. https://doi.org/10.1111/j.1744-6570.2007.00085.x
Dierdorff, E. C., & Wilson M. A. (2003). A meta-analysis of job analysis reliability. Journal of Applied Psychology, 88(4), 635–646. https://doi.org/10.1037/0021-9010.88.4.635
Dierdorff, E. C., & Morgeson, F. P. (2009). Effects of descriptor specificity and observability on incumbent work analysis ratings. Personnel Psychology, 62(3), 601-628. https://doi.org/10.1111/j.1744-6570.2009.01151.x
Dudley, N. M., Orvis, K. A., Lebiecki, J. E., & Cortina, J. M. (2006). A meta-analytic investigation of conscientiousness in the prediction of job performance: Examining the intercorrelations and the incremental validity of narrow traits. Journal of Applied Psychology, 91(1), 40. https://doi.org/10.1037/0021-9010.91.1.40
DuVernet, A. M. Dierdorff, E. C., & Wilson, M .A. (2015). Exploring factors that influence work analysis data: A meta-analysis of design choices, purposes, and organizational context. Journal of Applied Psychology, 100(5), 1603-1631. https://doi.org/10.1037/a0039084
Fiske, S. T., & Pavelchak, M. A. (1986). Category-based versus piecemeal-based affective responses: Developments in schema-triggered affect. In R. M. Sorrentino & E. T. Higgins (Eds.), Handbook of motivation and cognition: Foundations of social behavior (pp. 167–203). Guilford Press. https://bit.ly/3rwGhcZ
Foster, J. L., Gaddis, B. H., & Hogan, J. (2012). Personality-based job analysis. In M. A. Wilson, W. Bennett, Jr., S. G. Gibson, & G. M. Alliger (Eds.), The handbook of work analysis: Methods, systems, applications and science of work measurement in organizations (pp. 247-264). Routledge. https://doi.org/10.4324/9780203136324-24
Goffin, R. D., Rothstein, M. G., Rieder, M. J., Poole, A., Krajewski, H. T., & Powell, D. M. (2011). Choosing job-related personality traits: Developing valid personality oriented job analysis. Personality and Individual Differences, 51(5), 646–651. https://doi.org/10.1016/j.paid.2011.06.001
Goldberg, L. R. (1990). An alternative "description of personality": The Big-Five factor structure. Journal of Personality and Social Psychology, 59(6), 1216–1229. https://doi.org/10.1037/0022-3514.59.6.1216
Guder, E. J. (2012). Identifying appropriate sources of work information. In M. A. Wilson, W. Bennett, Jr., S. G. Gibson, & G. M. Alliger (Eds.), The handbook of work analysis. Methods, systems, applications and science of work measurement in organizations (pp. 31-40). Routledge. https://doi.org/10.4324/9780203136324-12
Harari, M. & Viswesvaran, C. (2018). Individual job performance. In D. S. Ones, N. Anderson, & C. Viswesvaran (Eds.). The SAGE handbook of industrial, work and organizational psychology (2nd ed. pp. 55-72). SAGE. https://doi.org/10.4135/9781473914940.n4
Hauenstein, N., & McCusker, M. E. (2017). Rater training: Understanding effects of training content, practice ratings, and feedback. International Journal of Selection and Assessment, 25(3), 253-266. https://doi.org/10.1111/ijsa.12177
Hogan, J. & Holland, B. (2003). Using theory to evaluate personality and job-performance relations: a socioanalytic perspective. Journal of Applied Psychology, 88(1), 100–112. https://doi.org/10.1037/0021-9010.88.1.100
Hunter, J. E. (1983). A causal analysis of cognitive ability, job knowledge, job performance, and supervisor ratings. In F. Landy, S. Zedeck, & J. Cleveland (Eds.), Performance measurement and theory (pp. 257– 266). Routledge. https://doi.org/10.4324/9781315211947-13
Hurtz, G. M., & Donovan, J. J. (2000). Personality and job performance: The big five revisited. Journal of Applied Psychology, 85(6), 869–879. https://doi.org/10.1037/0021-9010.85.6.869
Iliescu, D., Minulescu, M., Nedelcea, C., & Ispas, D. (2008). NEO PI-R, Manual tehnic [Technical manual for the NEO PI-R]. Sinapsis.
Ispas, D. (2010). The role of rater motivation in personnel selection validation studies [Doctoral dissertation, University of South Florida]. University of South Florida libraries. https://bit.ly/3vicGX5
Ispas, D., Iliescu, D., Ilie, A., & Johnson, R. E. (2014). Exploring the cross-cultural generalizability of the five-factor model of personality: The Romanian NEO PI-R. Journal of Cross-Cultural Psychology, 45(7), 1074-1088. https://doi.org/10.1177/0022022114534769
Jenkins, M., & Griffith, R. (2004). Using personality constructs to predict performance: Narrow or broad bandwidth. Journal of Business and Psychology, 19(2), 255-269. https://doi.org/10.1007/s10869-004-0551-9
Jordan, J., Linden, J.A., Maculatis, M.C., Hern, H.G., Schneider, J.I., Wills, C.P., Marshall, J.P., Friedman, A., & Yarris, L.M. (2018). Identifying the emergency medicine personality: A multisite exploratory pilot study. AEM Education and Training, 2(2), 91–99. https://doi.org/10.1002/aet2.10078
Judge, T. A., Rodell, J. B., Klinger, R. L., Simon, L. S., & Crawford, E. R. (2013). Hierarchical representations of the Five-Factor Model of personality in predicting job performance: Integrating three organizing frameworks with two theoretical perspectives. Journal of Applied Psychology, 98(6), 875–925. https://doi.org/10.1037/a0033901
Kaiser, R.B., & Craig, S.B. (2005). Building a better mouse trap: Item characteristics associated with rating discrepancies in 360-degree feedback. Consulting Psychology Journal: Practice and Research, 57(4), 235–245. https://doi.org/10.1037/1065-9293.57.4.235
Kulik, C. T. (1989). The effects of job categorization on judgments of the motivating potential of jobs. Administrative Science Quarterly, 34(1), 68–90. https://doi.org/10.2307/2392986
Lee, Y., Berry, C. M., & Gonzalez-Mulé, E. (2019). The importance of being humble: A meta-analysis and incremental validity analysis of the relationship between honesty-humility and job performance. Journal of Applied Psychology, 104(12), 1535–1546. https://doi.org/10.1037/apl0000421
Lubelski, D., Healy, A. T., Friedman, A., Ferraris, D., Benzel, E. C., & Schlenk, R. (2016). Correlation of personality assessments with standard selection criteria for neurosurgical residency applicants. Journal of Neurosurgery, 125(4), 986-994. https://doi.org/10.3171/2015.7.JNS15880
Manea, A. I. (2020). Selecting subject matter experts in job and work analysis surveys: Advantages and disadvantages. Academic Journal of Economic Studies, 2(6), 52–61. https://bit.ly/3rw8Iri
McFarland, L. A., Ryan, A. M., & Ellis, A. (2002). Item placement on a personality measure: Effects on faking behavior and test measurement properties. Journal of Personality Assessment, 78(2), 348-369. https://doi.org/10.1207/S15327752JPA7802_09
Morgeson, F. P., Brannick, M. T., & Levine, E. L. (2019). Job and work analysis: Methods, research, and applications for human resource management (3rd ed.). SAGE. https://bit.ly/3sQM4dZ
Morgeson, F. P., & Campion, M. A. (1997). Social and cognitive sources of potential inaccuracy in job analysis. Journal of Applied Psychology, 82(5), 627–655. https://doi.org/10.1037/0021-9010.82.5.627
Morgeson, F., & Campion, M. A. (2012). A framework of potential sources of inaccuracy in job analysis. In M. Wilson, W. Bennett, Jr., S. Gibson, & G. Alliger (Eds.), The handbook of work analysis: The methods, systems, applications, and science of work measurement in organizations (pp. 593–601). Routledge. https://doi.org/10.4324/9780203136324-43
Morgeson, F. P. & Dierdorff, E. C. (2011). Work analysis: From technique to theory. In S. Zedeck (Ed.), APA handbook of industrial and organizational psychology, Vol 2: Selecting and developing members for the organization (pp. 3–41). American Psychological Association. https://doi.org/10.1037/12170-001
Morgeson, F. P., Spitzmuller, M., Garza, A. S., & Campion, M. A. (2016). Pay attention! The liabilities of respondent experience and carelessness when making job analysis judgments. Journal of Management, 42(7), 1904-1933. https://doi.org/10.1177/0149206314522298
Nisbett, R. E., & Ross, L. (1980). Human inference: Strategies and shortcomings of social judgment. Prentice Hall.
Noua clasificare a ocupaţiilor din România [The new classification of occupations in Romania] (2020). Rentrop&Straton.
O’Neill, T. A., Goffin, R. D., & Rothstein, M. G. (2013). Personality and the need for personality-oriented work analysis. In N. Christiansen & R. P. Tett (Eds.), Handbook of personality at work (pp. 226–253). Routledge. https://doi.org/10.4324/9780203526910.ch11
Raymark, P. H., Schmit, M. J., & Guion, R. M. (1997). Identifying potentially useful personality constructs for employee selection. Personnel Psychology, 50(3), 723–736. https://doi.org/10.1111/j.1744-6570.1997.tb00712.x
Reiter-Palmon, R., Brown, M., Sandall, D. L., Buboltz, C., & Nimps, T. (2006). Development of an O*NET web-based job analysis and its implementation in the U.S. Navy: Lessons learned. Human Resource Management Review, 16(3), 294–309. https://doi.org/10.1016/j.hrmr.2006.05.003
Roch, S. G., Woehr, D. J., Mishra, V., & Kieszczynska, U. (2012). Rater training revisited: An updated meta-analytic review of frame-of-reference training. Journal of Occupational and Organizational Psychology, 85(2), 370-395. https://doi.org/10.1111/j.2044-8325.2011.02045.x
Salgado, J. F. (1997). The five factor model of personality and job performance in the European community. Journal of Applied Psychology, 82(1), 30–43. https://doi.org/10.1037/0021-9010.82.1.30
Salgado, J. F., Moscoso, S., & Berges, A. (2013). Conscientiousness, its facets, and the prediction of job performance ratings: Evidence against the narrow measures. International Journal of Selection and Assessment, 21(1), 74-84. https://doi.org/10.1111/ijsa.12018
Salgado, J. F., Moscoso, S., Sanchez, J. I., Alonso, P., Choragwicka, B., & Berges, A. (2015). Validity of the five-factor model and their facets: The impact of performance measure and facet residualization on the bandwidth-fidelity dilemma. European Journal of Work and Organizational Psychology, 24(3), 325-349. https://doi.org/10.1080/1359432X.2014.903241
Sanchez, J. I., & Levine, E. L. (1994). The impact of raters’ cognition on judgment accuracy: An extension to the job analysis domain. Journal of Business and Psychology, 9(1), 47–57. https://doi.org/10.1007/BF02230986
Sanchez, J. I., & Levine, E. L. (2012). The rise and fall of job analysis and the future of work analysis. Annual Review of Psychology, 63, 397–425. https://doi.org/10.1146/annurev-psych-120710-100401
Shaffer, J. A., & Postlethwaite, B. E. (2012). A matter of context: A meta-analytic investigation of the relative validity of contextualized and noncontextualized personality measures. Personnel Psychology, 65(3), 445–494. https://doi.org/10.1111/j.1744-6570.2012.01250.x
Spilberg, S. W., & Corey, D. M. (2014). Peace officer psychological screening manual. California Commission on Peace Officer Standards and Training. https://bit.ly/3clztIR
Stetz, T. A., Button, S. B., & Quist, J. (2012). Rethinking carelessness on job analysis surveys: Not all questions are created equal. Journal of Personnel Psychology, 11(2), 103–106. https://doi.org/10.1027/1866-5888/a000061
Suresh, A., Ramachandran, K., & Srivastava, A. (2012). Personality based job analysis of air traffic controller. Indian Journal of Aerospace Medicine, 56(2), 21-31.
Tett, R. P., Jackson, D. N., & Rothstein, M. (1991). Meta-analysis of personality–job performance relationships. Personnel Psychology, 47(1), 157–172. https://doi.org/10.1111/j.1744-6570.1994.tb02415.x
Tett, R. P., Jackson, D. N., Rothstein, M., & Reddon, J. R. (1999). Meta-analysis of bidirectional relations in personality-job performance research. Human Performance, 12(1), 1-29. https://doi.org/10.1207/s15327043hup1201_1
Tett, R. P., Steele, J. R., & Beauregard, R. S. (2003). Broad and narrow measures on both sides of the personality–job performance relationship. Journal of Organizational Behavior, 24(3), 335-356. https://doi.org/10.1002/job.191
Tsai, M. H., Wee, S., & Koh, B. (2019). Restructured frame‐of‐reference training improves rating accuracy. Journal of Organizational Behavior, 40(6), 740-757. https://doi.org/10.1002/job.2368
Vinchur, A. J., Schippmann, J. S., Switzer, F. S., & Roth, P. L. (1998). A meta-analytic review of predictors of job performance for salespeople. Journal of Applied Psychology, 83(4), 586–597. https://doi.org/10.1037/0021-9010.83.4.586
Voskuijl, O. F., & van Sliedregt, T. (2002). Determinants of interrater reliability of job analysis: A meta-analysis. European Journal of Psychological Assessment, 18(1), 52–62. https://bit.ly/3vi3TnU
Walmsley, P. T., Sackett, P. R., & Nichols, S. B. (2018). A large sample investigation of the presence of nonlinear personality–job performance relationships. International Journal of Selection and Assessment, 26(2-4), 145–163. https://doi.org/10.1111/ijsa.12223
Weekley, J., Labrador, J., & Campion, M. A. (2019). Job analysis ratings and criterion-related validity: Are they related and can validity be used as a measure of accuracy? Journal of Occupational and Organizational Psychology, 92(4), 764-786. https://doi.org/10.1111/joop.12272
Woo, S. E., Chernyshenko, L. S., Stark, S. E., & Conz, G. (2014). Validity of six openness facets in predicting work behaviors: A meta-analysis. Journal of Personality Assessment, 96(1), 76–86. https://doi.org/10.1080/00223891.2013.806329
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2021 Psihologia Resurselor Umane
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal,it also allows for use of the work for non-commercial purposes and if others remix, transform or build upon the works found in this journal they must distribute the contributions under the same licence as the original.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See: The Effect of Open Access).
