The purpose of this study was to evaluate the psychometric properties of a newly developed instrument intended to measure faculty competence as it pertains to their role as advisors, particularly in medical and professional programs. A total of 166 students completed the Faculty Advisor’s Skills and Behaviors Inventory (FASBI). The psychometric properties of the FASBI were evaluated using the Rasch Rating Scale Model. Results indicate the FASBI is a psychometrically-sound instrument capable of producing valid and reproducible measures.
}, number={1}, journal={Journal of Educational and Developmental Psychology}, publisher={Canadian Center of Science and Education}, author={Royal, Kenneth D. and Gonzalez, Liara M.}, year={2016}, month={Apr}, pages={195} } @article{royal_kedrowicz_snyder_2016, title={Do All Dogs Go to Heaven? Investigating the Association between Demographic Characteristics and Beliefs about Animal Afterlife}, volume={29}, ISSN={["1753-0377"]}, DOI={10.1080/08927936.2016.1189748}, abstractNote={ABSTRACT The purpose of this research was to explore American’s beliefs about animal afterlife based on key demographic factors such as sex, race/ethnicity, age, geographic region, religion/faith, and pet ownership. We attained a large and diverse sample of respondents (n = 800) using Amazon’s Mechanical Turk, and attempted to make the data fairly representative of the United States population by applying post-stratification weights based on auxiliary statistics obtained from US Census data. Results of the study suggest that many people perceive animal lives as similarly sacred as human lives. Evidence also suggests that one’s membership in a particular demographic category may have considerable bearing on views about animal afterlife. The authors recommend veterinarians remain cognizant that some people extend their own views on issues such as spirituality to their pets and those beliefs and values can impact veterinary care and decision-making.}, number={3}, journal={ANTHROZOOS}, publisher={Informa UK Limited}, author={Royal, Kenneth D. and Kedrowicz, April A. and Snyder, Amy M.}, year={2016}, pages={409–420} } @article{walker_engelhard_hedgpeth_royal_2016, title={Exploring Aberrant Responses Using Person Fit and Person Response Functions}, journal={Journal of Applied Measurement}, author={Walker, A.A. and Engelhard, G. and Hedgpeth, M.W. and Royal, K.D.}, year={2016}, month={Jan} } @article{snyder_royal_2016, title={Investigating the Financial Awareness and Behaviors of Veterinary Medical Students}, DOI={10.5539/ijef.v8n7p201}, abstractNote={A sample of veterinary medical students were administered a modified version of the Financial Fitness Quiz (FFQ) to gain insights about current veterinary students money management behaviors (e.g., developing financial goals, having a spending plan, accumulating savings, etc.). Results demonstrate students possess a general awareness and conscientiousness toward personal finance. In some cases, veterinary students engaged in positive financial behaviors at a higher rate than average consumers in the United States, and most reported having more cash on hand to cover an emergency than average consumers in the United States. While students generally appear to understand the negative impact of debt, results indicate most students do not fully appreciate the value of personal budgeting or annual calculations of net worth.
}, journal={International Journal of Economics and Finance}, author={Snyder, Amy M. and Royal, Kenneth D.}, year={2016}, month={Jun} } @article{stockdale_royal_2016, title={Missing data as a validity threat for medical and healthcare education research: problems and solutions}, volume={2}, DOI={10.5430/ijh.v2n2p67}, abstractNote={At present, the problem of missing data has received virtually no attention by medical and healthcare education researchers. This is a significant problem for the education research community because when missing data are disregarded or handled inappropriately it can result in serious validity threats. This article discusses the problem of missing data in the context of medical and healthcare education research and recommends appropriate methods for handling missing data.}, number={2}, journal={International Journal of Healthcare}, publisher={Sciedu Press}, author={Stockdale, Myrah and Royal, Kenneth}, year={2016}, month={Jun} } @misc{royal_cannedy_dent_2016, title={More on Diversity and Inclusion in Academic Medicine: The Isolation of Black Females}, volume={91}, ISSN={["1938-808X"]}, DOI={10.1097/acm.0000000000001215}, abstractNote={To the Editor: We recently read the research report by Person and colleagues1 on the topic of measuring diversity and inclusion in academic medicine. The topic of diversity and inclusivity is an important one for all the health professions, and other scholars will no doubt benefit from the useful tool (the Diversity Engagement Survey) the authors have generously shared with us. The authors found “black, Hispanic/Latino, female, and LGBTQ (lesbian, gay, bisexual, transgender, queer) respondents reported lower degrees of engagement than their counterparts,” and noted these findings are consistent with extant literature on these topics. We would like to briefly call attention to one key finding of this study that we have evidenced at both a medical school and a veterinary medical school in the United States as we believe it is a particularly important topic that deserves additional research. At the University of North Carolina School of Medicine we conducted a study involving social networks among medical students. Whereas most studies rely on the use of self-reported measures in which students provide ratings of their “sociability,” we utilized a study design in which students were presented a survey and the items consisted of a series of photo vignettes of other students in the same cohort. Students were asked to rate the strength of their social relationships with each of the other students. The most striking finding was that students reported significantly weaker social ties with black female students. It is important to note that all other demographic groups, including black males, reported similar levels of social ties with all other subpopulations. Relatedly, at North Carolina State University’s College of Veterinary Medicine we have observed a similar finding. Qualitative data have indicated that black females tend to be perceived as somewhat isolated from their peers, and black females also report feeling isolated. Again, this finding is not necessarily true for other demographic populations, including black males. Unquestionably, more research is needed in the area of diversity and inclusivity in medical education. However, we challenge researchers to devote particular attention to investigating black female subpopulations as this group of students appears to be particularly vulnerable and there is virtually a complete absence of literature on this subpopulation of medical students. Kenneth D. Royal, PhD, MSEdAssistant professor of educational assessment and outcomes, Department of Clinical Sciences, North Carolina State University College of Veterinary Medicine, Raleigh, North Carolina, and adjunct associate professor, Department of Family Medicine, University of North Carolina School of Medicine, Chapel Hill, North Carolina; [email protected] Allan L. Cannedy, DVMDirector for diversity and multicultural affairs, North Carolina State University College of Veterinary Medicine, Raleigh, North Carolina. Georgette A. Dent, MDAssociate dean for student affairs, University of North Carolina School of Medicine, Chapel Hill, North Carolina.}, number={7}, journal={ACADEMIC MEDICINE}, publisher={Ovid Technologies (Wolters Kluwer Health)}, author={Royal, Kenneth D. and Cannedy, Allan L. and Dent, Georgette A.}, year={2016}, month={Jul}, pages={896–896} } @misc{royal_2016, title={More on standard setting}, volume={13}, ISSN={["1743-498X"]}, DOI={10.1111/tct.12482}, abstractNote={As a professional psychometrician I read Richard Hays’ paper on standard setting with great interest. 1 I appreciated how Hays discussed the three broad categories of standardsetting methods and urged caution for the use of normreferenced methods, as basing performance measures on students’ relative standing compared with their peers is rarely appropriate in medical education. 2 I also appreciated Hays’ efforts to convey the message that standard setting is becoming increasingly popular in medical education, but that actually conducting these exercises properly can be a very challenging task.}, number={1}, journal={CLINICAL TEACHER}, publisher={Wiley-Blackwell}, author={Royal, Kenneth}, year={2016}, month={Feb}, pages={82–82} } @article{royal_munana_neel_2016, title={Prohibiting Students from Asking Questions during Exams: A Guideline for Promoting Fairness and Preserving Score Validity}, volume={8}, url={http://www.scopus.com/inward/record.url?eid=2-s2.0-85024494960&partnerID=MN8TOARS}, DOI={10.3138/jvme.0316-054r}, abstractNote={ Many medical and professional programs implement policies that prohibit students from asking questions during examinations. The reasoning behind these policies remains unclear to some, as there is a lack of literature addressing this topic. The purpose of this article is to present the rationale behind such policies and to discuss why these policies may help promote fairness and preserve score validity. }, number={2}, journal={Journal of Veterinary Medical Education}, publisher={University of Toronto Press Inc. (UTPress)}, author={Royal, Kenneth D. and Munana, Karen R. and Neel, Jennifer A.}, year={2016}, month={Aug}, pages={1–3} } @article{royal_hecker_2016, title={Rater Errors in Clinical Performance Assessments}, volume={43}, ISSN={["1943-7218"]}, DOI={10.3138/jvme.0715-112r}, abstractNote={ Rater errors are some of the most significant validity threats to any performance assessment. Veterinary medical education routinely uses raters to assess student performance in a variety of scenarios (e.g., clinical assessments, OSCEs, etc.). The purpose of this “teaching tip” is to introduce veterinary medical educators to the notion of rater error, identify a list of common rater errors, and discuss how these errors can be addressed and minimized so as to produce accurate and defensible measures of student performance. }, number={1}, journal={JOURNAL OF VETERINARY MEDICAL EDUCATION}, publisher={University of Toronto Press Inc. (UTPress)}, author={Royal, Kenneth D. and Hecker, Kent G.}, year={2016}, pages={5–8} } @article{kedrowicz_royal_flammer_2016, title={Social Media and Impression Management: Veterinary Medicine Students' and Faculty Members' Attitudes toward the Acceptability of Social Media Posts}, volume={4}, url={http://jamp.sums.ac.ir/index.php/JAMP/article/view/605}, number={4}, journal={Journal of Advances in Medical Education and Professionalism}, author={Kedrowicz, A.A. and Royal, K.D. and Flammer, K.}, year={2016}, pages={155–162} } @article{royal_hedgpeth_mcwhorter_2016, title={Students' Perceptions of and Experiences With Educational Technology: A Survey}, DOI={10.2196/mededu.5135}, abstractNote={Background It is generally assumed that incoming students in medical education programs will be better equipped for the “digital age” given their younger age and an educational upbringing in which technology was seemingly omnipresent. In particular, many assume that today's medical students are more likely to hold positive attitudes and increased comfortability with technology and possess greater information technology (IT) skills. Objective The purpose of this study was to compare responses of incoming veterinary medical students to a series of IT-related questions contained in a common questionnaire over the course of a 10-year period (2005-2015) to discern whether students’ attitudes have improved and uses and comfortability with technology have increased as anticipated. Methods A survey measuring attitudes and preferences, computing experience, and technology ownership was administered each year for the past 10 years to incoming veterinary medical students at a large veterinary school in the United States. Students' responses to survey items were compared at 3 data points (2005, 2010, and 2015). Results Today's incoming veterinary medical students tend to indicate the same desire to improve skills using spreadsheets and web page design as incoming students from 10 years ago. It seems that despite technological advances and increased exposure to such applications and skills, there remains a challenge for students to “keep up” with the ever evolving technology. Moreover, although students continue to report they are very comfortable with using a computer (and related devices), many use their computers as typewriters or word processors, as opposed to a means for performing more advanced computing functions. Conclusions In general, today's medical students are not expert computer users as many assume. Despite an upbringing in a digitized world, many students still lack many basic computing skills.}, journal={JMIR Medical Education}, author={Royal, Kenneth and Hedgpeth, Mari-Wells and McWhorter, Dan}, year={2016}, month={May} } @article{royal_hedgpeth_mulkey_fremer_2016, title={The 10 most wanted test cheaters in medical education}, volume={50}, ISSN={["1365-2923"]}, DOI={10.1111/medu.13096}, abstractNote={Abstract}, number={12}, journal={MEDICAL EDUCATION}, publisher={Wiley-Blackwell}, author={Royal, Kenneth and Hedgpeth, Marian-Wells and Mulkey, Jamie and Fremer, John}, year={2016}, month={Dec}, pages={1241–1244} } @article{royal_2016, title={The Impact of Item Sequence Order on Local Item Dependence: An Item Response Theory Perspective}, volume={9}, url={http://surveypractice.org/index.php/SurveyPractice/article/view/344/html_78}, note={Retrieved from}, number={5}, journal={Survey Practice}, author={Royal, K.}, year={2016} } @article{royal_rinaldo_2016, title={There's Education, and Then There's Education in Medicine}, volume={4}, number={3}, journal={Journal of Advances in Medical Education and Professionalism}, author={Royal, K.D. and Rinaldo, J.C.B.}, year={2016}, pages={150–154} } @misc{royal_hecker_2016, title={Understanding Reliability: A Review for Veterinary Educators}, volume={43}, ISSN={["1943-7218"]}, DOI={10.3138/jvme.0315-030r}, abstractNote={ Veterinary medical faculty and administrators routinely administer student assessments and conduct surveys to make decisions regarding student performance and to assess their courses/curricula. The decisions that are made are a result of the scores generated. However, how reliable are the scores and how confident can we be about these decisions? Reliability is one of the hallmarks of validity evidence, but what does this mean and what affects the reliability of scores? The purpose of this article is to provide veterinary medical educators and administrators with fundamental information regarding the concept of reliability. Specifically, we review what sources of error reduce the reliability of scores and we describe the different types of reliability coefficients that are reported. }, number={1}, journal={JOURNAL OF VETERINARY MEDICAL EDUCATION}, publisher={University of Toronto Press Inc. (UTPress)}, author={Royal, Kenneth D. and Hecker, Kent G.}, year={2016}, pages={1–4} } @article{royal_guskey_2015, title={A Case for Differentiated Grades}, volume={25}, DOI={10.1007/s40670-015-0127-5}, abstractNote={The recent court ruling in favor of Case Western Reserve University that denied a medical degree to a student based on a lack of professionalism shows the clear need for differentiated grading in medical education. In the Ohio case [1], a threejudge panel concluded that evaluations regarding professionalism are academic judgments that fall within the purview of the degree-granting institution. Thus, medical students with strong academic records based on assessments of their medical knowledge and skills may legally fail a course, fail a program, or otherwise be denied a degree based on unprofessional conduct. Although this case attracted great attention, the situation is not unusual. Stories abound of students who make outstanding grades and put forth maximum effort, but are lacking in the area of professional behavior. As Chen [2] points out, many medical school faculty members would like to fail students who exhibit extremely unprofessional behavior, but often believe such a decision would be indefensible given the student’s adequate performance on examinations of medical knowledge and skills. Other faculty members attempt to address this issue by allocating sufficient weight to professionalism indictors in the grade so that students with high assessment scores, but poor professionalism, could not receive a passing course grade. In this article, we discuss an alternative solution through differentiated grading that is designed to make grades more meaningful, accurate, and educationally sound. Amalgamated Versus Differentiated Grades}, number={3}, journal={Med.Sci.Educ.}, publisher={Springer Science \mathplus Business Media}, author={Royal, Kenneth D. and Guskey, Thomas R.}, year={2015}, month={Apr}, pages={323–325} } @article{royal_hedgpeth_smith_kirk_2015, title={A Method for Investigating "Instructional Familiarity" and Discerning Authentic Learning}, volume={5}, ISSN={["2277-9205"]}, DOI={10.4103/2141-9248.177990}, abstractNote={Background: Presently, most medical educators rely exclusively on item difficulty and discrimination indices to investigate an item's psychometric quality and functioning. We argue “instructional familiarity” effects should also be of primary concern for persons attempting to discern the quality and meaning of a set of test scores. Aim: There were four primary objectives of this study: (1) Revisit Haladyna and Roid's conceptualization of “instructional sensitivity” within the context of criterion-referenced assessments, (2) provide an overview of “instructional familiarity” and its importance, (3) reframe the concept for a modern audience concerned with medical school assessments, and (4) conduct an empirical evaluation of a medical school examination in which we attempt to investigate the instructional effects on person and item measures. Subjects and Methods: This study involved a medical school course instructor providing ratings of instructional familiarity (IF) for each mid-term examination item, and a series of psychometric analyses to investigate the effects of IF on students' scores and item statistics. The methodology used in this study is based primarily on a mixed-method, “action research” design for a medical school course focusing on endocrinology. Rasch measurement model; correlation analysis. Results: The methodology presented in this article was evidenced to better discern authentic learning than traditional approaches that ignore valuable contextual information about students' familiarity with exam items. Conclusions: The authors encourage other medical educators to adopt this straightforward methodology so as to increase the likelihood of making valid inferences about learning.}, number={6}, journal={ANNALS OF MEDICAL AND HEALTH SCIENCES RESEARCH}, publisher={Medknow}, author={Royal, K. D. and Hedgpeth, M. W. and Smith, K. W. and Kirk, D.}, year={2015}, pages={428–434} } @article{royal_hedgpeth_2015, title={A Novel Method for Evaluating Examination Item Quality}, volume={7}, DOI={10.5539/ijps.v7n1p17}, abstractNote={Background and aims: Poor quality examination items may result in invalid test scores that potentially misrepresent what a student actually knows about a given content area. Thus, if an examination consists largely of poor quality items it is plausible that an individual with minimal content knowledge could perform reasonably well and receive a score that erroneously inflates his or her measure of ability. This study builds on this premise by introducing a novel method for evaluating item quality and demonstrating its utility. Method: We sought to understand the extent to which medical school examination items were vulnerable to good test-taking skills and guessing strategies by administering an examination to a group of medical education professional staff. The extent to which persons with no formal medical training could perform above the odds of random guessing were used to identify zones in which items may be vulnerable to guessing strategies. Results: The performance of professional staff was able to provide excellent diagnostic information regarding which items may be particularly vulnerable to guessing strategies. Conclusions: The proposed methodology was demonstrated to be successful, thus we encourage other medical educators to adopt this model for evaluating item and examination quality in a variety of contexts.}, number={1}, journal={IJPS}, publisher={Canadian Center of Science and Education}, author={Royal, Kenneth D. and Hedgpeth, Mari-Wells}, year={2015}, month={Feb} } @article{royal_2015, title={A cautionary note about focus groups in medical education}, volume={26}, DOI={10.1080/14739879.2015.1101862}, abstractNote={ISSN: 1473-9879 (Print) 1475-990X (Online) Journal homepage: http://www.tandfonline.com/loi/tepc20 A cautionary note about focus groups in medical education Kenneth Royal To cite this article: Kenneth Royal (2015) A cautionary note about focus groups in medical education, Education for Primary Care, 26:6, 449-449, DOI: 10.1080/14739879.2015.1101862 To link to this article: http://dx.doi.org/10.1080/14739879.2015.1101862}, number={6}, journal={Education for Primary Care}, publisher={Informa UK Limited}, author={Royal, Kenneth}, year={2015}, month={Nov}, pages={449–449} } @article{latessa_beaty_royal_colvin_pathman_heck_2015, title={Academic outcomes of a community-based longitudinal integrated clerkships program.}, volume={37}, url={http://europepmc.org/abstract/med/25693796}, DOI={10.3109/0142159X.2015.1009020}, abstractNote={Abstract Background: Longitudinal integrated clerkships (LICs) receive recognition internationally as effective, innovative alternatives to traditional block rotations (TBRs) in undergraduate medical education. No studies of LICs in the USA have assessed how students perform on all the standardized exams. Aim: To compare performance on standardized tests of students in the first four years of LICs at the University of North Carolina School of Medicine–Asheville (UNC SOM–Asheville) with students from UNC SOM’s Chapel Hill main campus in TBRs. Methods: LIC and TBR students’ previous academic performance was considered using Medical College Admissions Test (MCAT) and United States Medical Licensing Examination (USMLE) Step 1 scores. Step 1 exam tests students’ pre-clinical, basic science knowledge. Outcome measures included all eight standardized National Board of Medical Examiners (NBME) Subject Shelf Examinations and USMLE Step 2 Clinical Knowledge (CK) examinations, which are used widely in the US to assess students’ progress and as prerequisites to eventual licensure. TBR students were selected using propensity scores to match LIC students. Groups were also compared on the required core clinical conditions documented, and on residency specialty choice. Results: Asheville LIC students earned higher scores on the Step 2 CK examination and the six shelf examinations linked to longitudinal clerkships than the matched TBR students (Step 2 CK exam, Family Medicine and Ambulatory Medicine shelf exams reached statistical significance). LIC students logged greater percentages of core conditions than TBR students and more often chose primary care residencies. Conclusions: UNC School of Medicine medical students participating in a longitudinal integrated curriculum in a community setting outperformed fellow students who completed a more TBR curriculum within the school’s academic medical center. Differences were found in performance on standard tests of clinical knowledge (six NBME exams and Step 2 CK exam), documented breadth of clinical experiences, and likelihood of choosing primary care residency programs.}, number={9}, journal={Medical Teacher}, publisher={Informa UK Limited}, author={Latessa, R and Beaty, N and Royal, K and Colvin, G and Pathman, DE and Heck, J}, year={2015}, month={Sep}, pages={862–867} } @article{royal_hedgpeth_2015, title={Balancing Test Length with Sufficiently Reliable Scores}, volume={7}, DOI={10.5959/eimj.v7i1.321}, abstractNote={One of the most important indicators of a quality examination is the reliability of the scores. In low to moderate stakes scenarios it is desirable for scores to achieve a minimum estimate of .70. Unfortunately, for many medical educators it is difficult to attain this minimum threshold for “acceptable” statistical reproducibility. A common approach is to include additional items to an exam, but this process can become cumbersome and misguided without clear direction. Fortunately, the Spearman-Brown Prophecy formula can help medical educators identify approximately how many additional items will be necessary to achieve a minimum reliability estimate of .70. This article describes a case in which we were presented with a less than desirable reliability estimate from a previous medical school examination, utilized the Spearman-Brown Prophecy formula, and was able to achieve the minimum estimate of .70 on the next iteration of the examination. We encourage others to make use of this technique rarely utilized outside of the psychometric arena as well.}, number={1}, journal={EIMJ}, publisher={KKMed Publications}, author={Royal, Kenneth D and Hedgpeth, Mari-Wells}, year={2015}, month={Mar} } @article{magallanes_stoskopf_royal_2015, title={Building Collaborative Learning Opportunities between Future Veterinary and Design Professionals}, volume={3}, DOI={10.11114/jets.v3i5.852}, abstractNote={Positive inter-professional collaborations and interactions facilitate the effectiveness of veterinarians working on professional teams addressing a wide range of societal challenges. The need for these interactions extend far beyond the different medical professions, which is the limit of many discussions of inter-professional relations for veterinary medicine. Methods of problem solving differ across professions, and clinical veterinary problem solving is quite distinct from the approaches of non-medical professions. These distinctions are particularly obvious when looking at the approaches of design professionals. Yet, most veterinarians work in collaboration with architects and landscape architects at some time in their careers, with varying degrees of success. This case study describes a collaboration between a College of Veterinary Medicine and a College of Design in which a course was offered across both colleges that allowed students to build a real life design to suit the needs of a veterinary medical college considering how to best utilize space for animals, animal and human interaction and the holistic development of healthy spaces for humans. Results of the collaboration indicated that despite initial struggles, students from very different disciplinary backgrounds were able to effectively work in teams, problem solve, communicate and exchange ideas, learn to appreciate their peer counterparts, and produce a high-quality design that may ultimately be adopted by the university.}, number={5}, journal={Journal of Education and Training Studies}, publisher={Redfame Publishing}, author={Magallanes, Fernando and Stoskopf, Michael K. and Royal, Kenneth D.}, year={2015}, month={Jun}, pages={43–51} } @article{royal_schoenfeld-tacher_flammer_2015, title={Comparing Veterinary Student and Faculty Perceptions of Academic Misconduct}, volume={1}, ISSN={2380-9205 2380-9183}, url={http://dx.doi.org/10.5430/irhe.v1n1p81}, DOI={10.5430/irhe.v1n1p81}, abstractNote={A study was conducted to assess veterinary students’ and faculty perceptions of a variety of academic and classroom behaviors, and the degree to which these are acceptable or not. Two instruments were developed for this purpose: 1) The Exams and Assignments Scale (EAS), consisted of 23 items measuring the extent to which a variety of examination and assignment-related actions and behaviors constituted misconduct, and 2) The Classroom Behavior Scale (CBS), consisted of 8 items and measured the extent to which a variety of classroom behaviors were acceptable professional behaviors. Student responses were pooled for analysis and compared to faculty perceptions. This comparison resulted in statistically significant differences on 11 of the 23 EAS indicators. In all cases, faculty felt the described actions were more severe offenses than students with most of these discrepancies centering on situations regarding examinations and clinical skills. Both groups felt that offenses pertaining to cheating in exam settings were the most severe infractions, and the least severe issues pertained to laboratories/assignments and other out of class behaviors. On the CBS scale, there were statistically significant differences on 3 of the 8 measures, with faculty reporting the behaviors as less acceptable. The most concerning behaviors involved actions that could potentially impact people other than the offender, while the most acceptable behaviors seemed to only impact the person committing them.}, number={1}, journal={International Research in Higher Education}, publisher={Sciedu Press}, author={Royal, Kenneth D. and Schoenfeld-Tacher, Regina M. and Flammer, Keven}, year={2015}, month={Nov} } @article{schoenfeld-tacher_royal_flammer_2015, title={Differences Among Veterinary Students’ and Faculty Members’ Perceptions of Appropriate Interactions with Corporate Entities at One US College of Veterinary Medicine}, volume={26}, ISSN={2156-8650}, url={http://dx.doi.org/10.1007/s40670-015-0213-8}, DOI={10.1007/s40670-015-0213-8}, number={1}, journal={Medical Science Educator}, publisher={Springer Science and Business Media LLC}, author={Schoenfeld-Tacher, Regina M. and Royal, Kenneth D. and Flammer, Keven}, year={2015}, month={Nov}, pages={35–38} } @article{royal_guskey_2015, title={Does Mathematical Precision Ensure Valid Grades? What Every Veterinary Medical Educator Should Know}, volume={42}, ISSN={["1943-7218"]}, url={http://europepmc.org/abstract/med/26075627}, DOI={10.3138/jvme.0115-005r1}, abstractNote={ Veterinary medical educators use a variety of methods for calculating student grades. Most educators assume grades will be valid as long as the mathematical calculations underpinning each grading method are correct. The purpose of this “teaching tip” is to demonstrate and discuss how mathematical precision does not necessarily equate to truthful scores that accurately reflect what students know or can do. We contend that this topic is of considerable importance to all veterinary educators who assign grades, and to academic deans who make decisions about grading policies. }, number={3}, journal={JOURNAL OF VETERINARY MEDICAL EDUCATION}, publisher={University of Toronto Press Inc. (UTPress)}, author={Royal, Kenneth D. and Guskey, Thomas R.}, year={2015}, pages={242–244} } @article{schoenfeld-tacher_kogan_meyer-parsons_royal_shaw_2015, title={Educational Research Report: Changes in Students' Levels of Empathy during the Didactic Portion of a Veterinary Program}, volume={42}, ISSN={0748-321X 1943-7218}, url={http://dx.doi.org/10.3138/jvme.0115-007R}, DOI={10.3138/jvme.0115-007r}, abstractNote={ Empathy can be defined as the ability to understand and connect with the emotional state and frame of mind of another. Establishing connections with others is a crucial skill for veterinarians who must earn the trust and compliance of their clients to provide quality care for their patients. This is a longitudinal study using the Davis Interpersonal Reactivity Index (IRI) to assess changes in veterinary students' self-reported empathy as they progressed through the didactic portion of the DVM program at Colorado State University. The IRI consists of four subscales that measure cognitive and affective components of empathy: Perspective Taking, Fantasy, Emphatic Concern, and Personal Distress. Data were collected at three time points, corresponding to the start of the first year, the fourth semester (second year), and the fifth semester (third year). Results showed an overall decline in Perspective Taking scores, with a significant decrease between first and second year. There was an overall increase in students' levels of Personal Distress as they progressed through the program. The significant difference was found between years 1 and 3. This is especially concerning because the enhanced unease in interpersonal interactions coincides with students entering clinical rotations in the third year—the very time when these traits are necessary for quality client and patient care. Veterinary educators are urged to use this information as a basis for investigating novel curricular and programmatic interventions to counteract these changes in student empathy. }, number={3}, journal={Journal of Veterinary Medical Education}, publisher={University of Toronto Press Inc. (UTPress)}, author={Schoenfeld-Tacher, Regina M. and Kogan, Lori R. and Meyer-Parsons, Beatrice and Royal, Kenneth D. and Shaw, Jane R.}, year={2015}, month={Sep}, pages={194–205} } @article{webb_royal_nash_2015, title={Examining Job Satisfaction among Kentucky Head Principals Using the Rasch Rating Scale Model}, volume={5}, DOI={10.5296/jse.v5i3.8152}, abstractNote={Effective principals can impact student learning and other vital outcomes; thus, it is important to retain effective school leaders. This article examines the perceived sources of Kentucky principals’ satisfaction and dissatisfaction with their work and implications for policies and practices to increase principal retention. Survey research methods were used and Kentucky principals’ satisfaction with specified job facets was measured using the Rasch Rating Scale Model (RRSM). Data indicated that head principals sampled in Kentucky were generally dissatisfied with the amount of time spent on tasks that have nothing to do with their primary responsibility of improving student outcomes.
}, number={3}, journal={JSE}, publisher={Macrothink Institute, Inc.}, author={Webb, Xavier J. and Royal, Kenneth D. and Nash, John B.}, year={2015}, month={Jun}, pages={135} } @article{stonkus_royal_2015, title={Further Validation of the Inventory of Mental Toughness Factors in Sport (IMTF-S)}, volume={7}, DOI={10.5539/ijps.v7n3p35}, abstractNote={The purpose of this study was to provide further validation a new measure of mental toughness in sport. The Inventory of Mental Toughness Factors in Sport (IMTF-S) was originally developed and validated using principal component analysis. For the present study, the psychometric properties of the IMTF-S were again evaluated, but by way of the Rasch Rating Scale Model using the same sample (n=329) of athletes, and the same 42-item instrument measuring mental toughness on a 5-point Likert-type scale (always-never). Results indicate the IMTF-S is a psychometrically-sound instrument capable of producing valid and reproducible measures of mental toughness in sports.}, number={3}, journal={IJPS}, publisher={Canadian Center of Science and Education}, author={Stonkus, Mark A. and Royal, Kenneth D.}, year={2015}, month={Aug} } @article{royal_hedgpeth_bynum_colford_2015, title={How Much Content Knowledge Do Graduating Medical Students Retain from Their Pre-Clinical and Clinical Courses?}, volume={26}, DOI={10.1007/s40670-015-0218-3}, number={1}, journal={Medical Science Educator}, publisher={Springer Nature}, author={Royal, Kenneth D. and Hedgpeth, Mari-Wells and Bynum, Debra and Colford, Cristin}, year={2015}, pages={117–122} } @article{royal_schoenfeld-tacher_kedrowicz_hardie_flammer_2015, title={Measuring Factors that Influence Decisions to Become a Veterinarian}, volume={4}, ISSN={2322-2719}, url={http://dx.doi.org/10.15171/rdme.2015.004}, DOI={10.15171/rdme.2015.004}, abstractNote={Introduction: Veterinary medical education; Psychometrics; Career influence; The purpose of the present study was twofold. First, we sought to evaluate the psychometric properties of a newly developed instrument designed to measure factors incoming veterinary medical students indicated were important (or not) in their decision to become a veterinarian. Second, we sought to produce a measure of these factors by utilizing a state-of-the-art psychometric modeling technique to produce an empirical hierarchy that illustrates the construct of interest and make appropriate inferences about the findings. Methods: The Career Influence Factors Survey (CIFS) was administered to an incoming class of veterinary medical students. Data were analyzed via the Rasch Rating Scale Model. The psychometric properties of the instrument were evaluated according to Messick’s framework of construct validity. Results: Results indicate the Career Influence Factors Survey is a psychometrically-sound instrument capable of accurately and reliably measuring veterinary medical students’ motivations for becoming a veterinarian. Substantive results indicated issues pertaining to animal welfare were the most important in making the decision to become a veterinarian. Previous exposure to a particular animal or family pet was only moderately influential in students’ decision to pursue a career in veterinary medicine. Factors such as a desire to work on food production issues and anticipated earning potential tend to be of lesser importance to incoming students. Conclusion: We encourage others to utilize the instrument and/or adopt the methodology to evaluate the psychometric properties of other instruments. Substantive findings generally corroborated findings from previous studies, but may identify new insights for improving the recruitment of veterinary medical students.}, number={1}, journal={Research and Development in Medical Education}, publisher={International Society for Phytocosmetic Sciences}, author={Royal, Kenneth D. and Schoenfeld-Tacher, Regina and Kedrowicz, April A. and Hardie, Elizabeth M. and Flammer, Keven}, year={2015}, month={Jul}, pages={23–29} } @article{royal_walter_2015, title={On Requiring Medical Student Participation on Course and Faculty Evaluations: Ethical and Methodological Implications}, volume={5}, DOI={10.5958/2231-6728.2015.00040.2}, abstractNote={Many medical schools have policies that require student participation on various course and faculty evaluations. Consequences for non-participation often include threats to withhold grades and prohibiting students from further advancing with their studies. The rationale behind these policies is that mandatory participation is the only way to ensure a high response rate, which many faculty and administrators erroneously perceive as the primary requirement for valid feedback. Unfortunately, there are a number of ethical problems and methodological concerns underscoring these practices. The purpose of this opinion article is to briefly discuss some of the problems associated with this evaluation practice and encourage medical educators to conduct evaluations in more ethically responsible and methodologically appropriate ways.}, number={3}, journal={Journal of Research in Medical Education & Ethics}, publisher={Diva Enterprises Private Limited}, author={Royal, Kenneth D. and Walter, Rachael A.}, year={2015}, pages={195} } @article{royal_guskey_2015, title={On the appropriateness of norm- and criterion-referenced assessments in medical education}, volume={94}, number={7}, journal={Ear, Nose, & Throat Journal}, author={Royal, K. D. and Guskey, T. R.}, year={2015}, month={Jul}, pages={252-} } @article{royal_guskey_ear_2015, title={On the appropriateness of norm- and criterion-referenced assessments in medical education.}, volume={94}, url={http://europepmc.org/abstract/med/26214661}, number={7}, journal={& throat journal}, author={Royal, K.D. and Guskey, T.R. and Ear, nose}, year={2015}, month={Jul}, pages={252–254} } @article{royal_henderson_hedgpeth_2015, title={Post-Exam Reviews: A Consideration of Costs and Unintended Consequences}, volume={25}, DOI={10.1007/s40670-015-0140-8}, number={3}, journal={Med.Sci.Educ.}, publisher={Springer Science \mathplus Business Media}, author={Royal, Kenneth D. and Henderson, Ashley G. and Hedgpeth, Mari-Wells}, year={2015}, month={Jun}, pages={327–329} } @article{bynum_colford_royal_2015, title={Teaching medical students the art of the 'write-up'.}, volume={12}, url={http://europepmc.org/abstract/med/26036763}, DOI={10.1111/tct.12304}, abstractNote={The creation of a complete ‘write–up’ continues to be essential to the clinical learning experience for medical students. The ability to document a clinical encounter is a key communication skill and Core Entrustable Professional Activity for entering residency.}, number={4}, journal={Clinical Teacher}, author={Bynum, D and Colford, C and Royal, K}, year={2015}, month={Aug}, pages={246–249} } @article{thomas r. o'neill_sun_peabody_royal_2015, title={The Impact of Repeated Exposure to Items}, volume={27}, ISSN={["1532-8015"]}, url={http://europepmc.org/abstract/med/26507998}, DOI={10.1080/10401334.2015.1077131}, abstractNote={Theory: When test developers have a limited number of test questions available or when the equating design requires some item overlap across forms, psychometricians worry that examinees who encounter previously seen questions on subsequent test forms may be able to inflate their test score due to their familiarity with the repeated test questions. Hypotheses: Prior exposure to test questions may lead to contamination and inflated scores. This research seeks to detect if examinees' scores were inflated due to prior exposure to test questions and, if so, whether those increases were significant. Method: The sample for this study consisted of candidates who took the American Board of Family Medicine's certification examination twice in a single year (n = 988). Examinees were randomly assigned one of two forms for their first attempt and received the other form for their repeat test. There were 99 questions in common across both forms. The Rasch model was used to estimate examinee ability. Performance changes on the common questions and unique questions were compared and repeated measures t tests were performed to establish whether score changes were likely to have occurred by chance. Results: On average, the examinees increased their overall ability estimate by .187 logits on the repeat attempt. The repeated measures t tests indicate this difference was statistically significant, t(987) = −25.298, p < .001, α = .05. The mean difference between the examinees' ability estimate on common and unique items for their first attempt was not statistically significant, t(987) = .264, p = .792, α = .05; however, the mean difference between common and unique items on the second attempt (0.029 logits) was statistically significant, t(987) = 3.28, p = .001, α = .05. Conclusions: Some of the increase in the examinees' overall ability estimate may attributed to a general increase in the latent trait; however, there was a small but detectable increase that could be attributed to prior exposure to the questions. On average, about 15% of the repeated questions were changed from wrong to right, but about 11% of questions were changed from right to wrong, suggesting that examinees may occasionally be using prior exposure to their benefit but general guessing accounts for more of the changes. The impact of the mean difference between the common and unique item scores (0.029 logits) is trivial at the individual level; however, such a bias among the population of repeat testers could be problematic if a small subset of examinees were using a “remember–research–retest” strategy to obtain nontrivial score increases.}, number={4}, journal={TEACHING AND LEARNING IN MEDICINE}, author={Thomas R. O'Neill and Sun, Letao and Peabody, Michael R. and Royal, Kenneth D.}, year={2015}, month={Oct}, pages={404–409} } @article{o'neill_li_peabody_lybarger_royal_puffer_2015, title={The Predictive Validity of the ABFM's In-Training Examination.}, volume={47}, url={http://europepmc.org/abstract/med/25905876}, number={5}, journal={Family medicine}, author={O'Neill, TR and Li, Z and Peabody and Lybarger, M and Royal, K and Puffer, JC}, year={2015}, month={May}, pages={349–356} } @article{royal_stockdale_2015, title={The myth of learning styles: What medical educators need to know}, volume={94}, url={http://www.scopus.com/inward/record.url?eid=2-s2.0-84949487901&partnerID=MN8TOARS}, number={4-5}, journal={Ear, Nose and Throat Journal}, author={Royal, K.D. and Stockdale, M.R.}, year={2015}, pages={132–134} } @article{royal_stockdale_ear_2015, title={The myth of teaching styles: What medical educators need to know [Editorial].}, volume={94}, url={http://europepmc.org/abstract/med/25923265}, number={4-5}, journal={& throat journal}, author={Royal, K.D. and Stockdale, M.R. and Ear, nose}, year={2015}, pages={132–134} } @article{o’neill_li_peabody_lybarger_royal_puffer_2015, title={The predictive validity of the ABFM’s in-training examination}, volume={47}, url={http://www.scopus.com/inward/record.url?eid=2-s2.0-84929091868&partnerID=MN8TOARS}, number={5}, journal={Family Medicine}, author={O’Neill, T.R. and Li, Z. and Peabody, M.R. and Lybarger, M. and Royal, K. and Puffer, J.C.}, year={2015}, pages={349–356} } @article{royal_2015, title={Using the Nudge and Shove Methods to Adjust Item Difficulty Values}, volume={42}, url={http://europepmc.org/abstract/med/26200700}, DOI={10.3138/jvme.0115-008r}, abstractNote={ In any examination, it is important that a sufficient mix of items with varying degrees of difficulty be present to produce desirable psychometric properties and increase instructors' ability to make appropriate and accurate inferences about what a student knows and/or can do. The purpose of this “teaching tip” is to demonstrate how examination items can be affected by the quality of distractors, and to present a simple method for adjusting items to meet difficulty specifications. }, number={3}, journal={Journal of Veterinary Medical Education}, publisher={University of Toronto Press Inc. (UTPress)}, author={Royal, Kenneth D.}, year={2015}, month={Sep}, pages={239–241} } @article{madanick_brown_royal_palmer_isaacs_2014, title={436 Feasibility and Acceptability of a Mobile Application for Point-of-Care Data Collection in Assessing Trainee Skills During Colonoscopy Training: a Pilot Study}, volume={79}, DOI={10.1016/j.gie.2014.02.097}, abstractNote={Background & Aims: The Mayo Colonoscopy Skills Assessment Tool (MCSAT) is an instrument designed to improve competency assessment during colonoscopy training by defining a trainee's learning curve. Because the MCSAT was integrated into a single institution's endoscopy database, dissemination for routine use by other GI training programs has been limited. As mobile computers (tablets, smartphones) become more widely used in medical settings, a mobile application (APP) could allow dissemination of the MCSAT or other evaluation instruments to other GI training programs for point-of-care (POC) data collection. The aim of this study is to assess the feasibility and acceptability of an APP for POC data collection by GI faculty for assessment of trainees' colonoscopy skills.}, number={5}, journal={Gastrointestinal Endoscopy}, publisher={Elsevier BV}, author={Madanick, Ryan D. and Brown, Daniel L. and Royal, Kenneth and Palmer, Jeff and Isaacs, Kim L.}, year={2014}, month={May}, pages={AB139–AB140} } @article{royal_dent_2014, title={A Novel Approach for Investigating Curricular Redundancy}, volume={6}, DOI={10.5959/eimj.v6i4.285}, abstractNote={Curricular redundancy can be a major problem for medical schools, especially those with a highly integrated curriculum. Although some redundancy is desirable, many forms of redundancy are not. Unfortunately, evaluating curricular redundancy and discerning desirable redundancy from undesirable redundancy can be quite challenging. Further, pinpointing undesirable redundancy and quantifying it so as to produce an estimate of inefficiency is even more difficult. This brief article describes a novel , and promising, student-led strategy for evaluating redundancy in a highly integrated medical school curriculum.}, number={4}, journal={EIMJ}, publisher={KKMed Publications}, author={Royal, Kenneth D and Dent, Georgette A}, year={2014} } @article{royal_gilliland_dent_2014, title={A student-led methodology for evaluating curricular redundancy}, volume={10}, number={23}, journal={Journal of MultiDisciplinary Evaluation}, author={Royal, K.D. and Gilliland, K.O. and Dent, G.A.}, year={2014}, month={Mar}, pages={1–8} } @article{royal_kernick_gilliland_2014, title={Are Repeat Exam Scores Trustworthy? A Note about Measurement Error and the Need for Genuine Instructional Remediation}, volume={24}, DOI={10.1007/s40670-014-0011-8}, number={1}, journal={Med.Sci.Educ.}, publisher={Springer Science \mathplus Business Media}, author={Royal, Kenneth D. and Kernick, Edward T. and Gilliland, Kurt O.}, year={2014}, month={Feb}, pages={39–41} } @article{royal_stockdale_2014, title={Are Teacher Course Evaluations Biased Against Faculty That Teach Quantitative Methods Courses?}, volume={4}, DOI={10.5430/ijhe.v4n1p217}, abstractNote={The present study investigated graduate students’ responses to teacher/course evaluations (TCE) to determine if students’ responses were inherently biased against faculty who teach quantitative methods courses. Item response theory (IRT) and Differential Item Functioning (DIF) techniques were utilized for data analysis. Results indicate students in non-methods courses preferred the structure of quantitative courses, but tend to be more critical of quantitative instructors. Authors encourage consumers of TCE results to investigate item-level results, as opposed to summative results, when making inferences about course and instructor quality.}, number={1}, journal={IJHE}, publisher={Sciedu Press}, author={Royal, Kenneth D. and Stockdale, Myrah R}, year={2014}, month={Oct} } @article{schulte_mannino_royal_brown_peterson_puffer_2014, title={Community size and organization of practice predict family physician recertification success.}, volume={27}, url={http://europepmc.org/abstract/med/24808117}, DOI={10.3122/jabfm.2014.03.130016}, abstractNote={Objective: Health disparities exist between rural and urban areas. Rural physicians may lack sufficient medical knowledge, which may lead to poor quality of care. Therefore, we sought to determine whether medical knowledge differed between family physicians (FPs) practicing in rural areas compared with those practicing in metropolitan areas. Methods: We studied 8361 FPs who took the American Board of Family Medicine maintenance of certification (MOC) examination in 2009. Data sources were examination results and data from a demographic survey of practice structure and activities, completed as part of the examination application process. FPs' location of practice was categorized as either rural or metropolitan using a moderate and conservative definition based on reported community size. Univariate statistics assessed differences in FP characteristics between rural and metropolitan areas. Logistic regression analyses determined the adjusted relationship between rural status and the odds of passing the MOC examination. Results: Metropolitan FPs were less likely than their rural counterparts to pass the MOC examination using both the moderate (odds ratio, 0.67; 95% confidence interval, 0.54–0.83) and conservative (odds ratio, 0.56; 95% confidence interval, 0.42–0.74) definitions. Physicians in solo practice were less likely to pass the examination than physicians in group practice. Conclusion: Rural physicians were more likely to pass the MOC examination, suggesting that rural health disparities do not result from a lack of provider knowledge.}, number={3}, journal={Journal of the American Board of Family Medicine}, author={Schulte, BM and Mannino, DM and Royal, KD and Brown, SL and Peterson, LE and Puffer, JC}, year={2014}, pages={383–390} } @article{royal_akers_2014, title={Social Network Analysis and Rasch Measurement as Complementary Methods}, volume={28}, number={2}, journal={Rasch Measurement Transactions}, author={Royal, K.D. and Akers, L.}, year={2014}, pages={1466–1467} } @article{street_gilliland_mcneil_royal_2014, title={The Flipped Classroom Improved Medical Student Performance and Satisfaction in a Pre-clinical Physiology Course}, volume={25}, DOI={10.1007/s40670-014-0092-4}, number={1}, journal={Med.Sci.Educ.}, publisher={Springer Science \mathplus Business Media}, author={Street, Sarah E. and Gilliland, Kurt O. and McNeil, Cheryl and Royal, Kenneth}, year={2014}, month={Nov}, pages={35–43} } @article{royal_puffer_2014, title={The consequential validity of ABFM examinations}, volume={27}, url={http://www.scopus.com/inward/record.url?eid=2-s2.0-84900529537&partnerID=MN8TOARS}, DOI={10.3122/jabfm.2014.03.140089}, abstractNote={Measurement scholar Samuel Messick,[1][1] defines validity as “an integrated evaluative judgment of the degree to which empirical evidence and theoretical rationales support the adequacy and appropriateness of inferences and actions based on test scores… ” (p. 13). Messick's definition of}, number={3}, journal={Journal of the American Board of Family Medicine}, author={Royal, K.D. and Puffer, J.C.}, year={2014}, pages={430–431} } @article{royal_puffer_2014, title={The consequential validity of ABFM examinations.}, volume={12}, url={http://europepmc.org/abstract/med/24821902}, DOI={10.1370/afm.1647}, abstractNote={Measurement scholar, Samuel Messick, defines validity as “an integrated evaluative judgment of the degree to which empirical evidence and theoretical rationales support the adequacy and appropriateness of inferences and actions based on test scores….”[1][1] (p13) Messick’s definition of}, number={3}, journal={Annals of Family Medicine}, author={Royal, KD and Puffer, JC}, year={2014}, pages={280–282} } @article{royal_gilliland_kernick_2014, title={Using Rasch measurement to score, evaluate, and improve examinations in an anatomy course.}, volume={7}, url={http://europepmc.org/abstract/med/24431324}, DOI={10.1002/ase.1436}, abstractNote={Any examination that involves moderate to high stakes implications for examinees should be psychometrically sound and legally defensible. Currently, there are two broad and competing families of test theories that are used to score examination data. The majority of instructors outside the high‐stakes testing arena rely on classical test theory (CTT) methods. However, advances in item response theory software have made the application of these techniques much more accessible to classroom instructors. The purpose of this research is to analyze a common medical school anatomy examination using both the traditional CTT scoring method and a Rasch measurement scoring method to determine which technique provides more robust findings, and which set of psychometric indicators will be more meaningful and useful for anatomists looking to improve the psychometric quality and functioning of their examinations. Results produced by the more robust and meaningful methodology will undergo a rigorous psychometric validation process to evaluate construct validity. Implications of these techniques and additional possibilities for advanced applications are also discussed. Anat Sci Educ 7: 450–460. © 2014 American Association of Anatomists.}, number={6}, journal={Anatomical Sciences Education}, author={Royal, KD and Gilliland, KO and Kernick, ET}, year={2014}, pages={450–460} } @article{royal_smith_foster_hedgpeth_2014, title={You Say Tomato, I Say “Tomahto”: the Semantics of Blended Learning}, volume={25}, DOI={10.1007/s40670-014-0093-3}, abstractNote={The notion of a “flipped classroom” has garnered considerable attention in medical education in recent years. Unfortunately, many medical educators find the term confusing. Multiple definitions exist, and instructors across multiple disciplines and education levels (e.g., K-12 to higher education) use the term inconsistently. Further, there is no single authoritative definition of a flipped classroom. Many educators use something similar to the following as their working definition: School work is done at home and home work is done at school. In actuality, the flipped classroom is very similar to other education models including reverse instruction, inverted classroom, hybrid learning, and blended learning; however, the flipped classroom has become the most recognized term. All of these education models are closely related because they share a common feature: students are required to prepare for class through the use of text, videos, or other educational materials. Ambiguous terminology presents three major problems for medical educators. First, it imposes unnecessary and undesirable constraints on an instructor. Adhering to a single pedagogical approach limits an instructor’s ability to adapt content delivery to the learner’s needs and nature of the content. Specialized terms often compel faculty to think of instructional approaches as a strict dichotomy, as opposed to a continuum that offers maximum flexibility to meet learners in a variety of ways. Second, many medical educators find terms such as reverse instruction, inverted classroom, hybrid learning and flipped classroom to be vague. This results in many medical educators erroneously assuming that these techniques ask students to acquire and understand large volumes of information independently and/or asynchronously and without any oversight. Third, ambiguous terminology makes the implementation and study of these concepts difficult. Specifically, educators will find it difficult to aptly characterize their instructional approaches when discussing with others, and researchers will find it difficult to conduct comparative and replication studies given the ambiguity. These issues could potentially impede our understanding of these concepts, result in confusion among colleagues, and waste researchers’ time. Margulieux and colleagues [1] have recognized similar problems with ambiguous terminology and produced a twodimensional taxonomy that attempts to clarify the distinctions between the terms hybrid, blended, flipped, and inverted learning.While the taxonomy appears promising, it is unlikely that the typical medical educator with minimal formal educational training will use such a taxonomy without considerable faculty development. Thus, even potentially promising solutions for understanding these issues as they are currently framed have substantial limitations in the context of medical education in which most medical educators are medical content experts and practitioners first and educators second. We believe making a semantic distinction is critical for purposes of objectivity. Given that medical schools have historically relied primarily on lectures to convey instruction, many medical educators have deeply rooted opinions that lectures are the “right” or preferred method of instruction. Terms such as flipped, reverse, and inverted imply a significant deviation from the norm, which could potentially invoke negative emotions among skeptical faculty. Given the complex and ambiguous terminology, associated instructional constraints, and potential for adverse emotional responses, we believe the term “blended learning” is more appropriate. Graham [2] and Osguthore and Graham [3] precisely define blended learning as “combining online and faceto-face instruction.” Because combining instructional methodologies is not new, this definition acknowledges the K. D. Royal (*) North Carolina State University, Raleigh, NC, USA e-mail: kdroyal2@ncsu.edu}, number={1}, journal={Med.Sci.Educ.}, publisher={Springer Science \mathplus Business Media}, author={Royal, Kenneth D. and Smith, Kathryn W. and Foster, Johanna H. and Hedgpeth, Mari-Wells}, year={2014}, month={Nov}, pages={81–82} } @article{royal_raddatz_2013, title={A cautionary tale about item equating with fluctuating samples}, volume={27}, number={2}, journal={Rasch Measurement Transactions}, author={Royal, K.D. and Raddatz, M.M.}, year={2013}, pages={1417} } @article{royal_puffer_2013, title={A closer look at recertification candidate pass rates.}, volume={26}, url={http://europepmc.org/abstract/med/23986926}, DOI={10.3122/jabfm.2013.04.130134}, abstractNote={We continue to receive a number of inquiries from physicians regarding the low pass rate statistics for recertification candidates for the reporting periods from 2010 to 2012. Many physicians fear the examination has changed in some way, putting those attempting to maintain their certification at a}, number={4}, journal={Journal of the American Board of Family Medicine}, author={Royal, KD and Puffer, JC}, year={2013}, pages={478–489} } @article{royal_puffer_2013, title={A closer look at recertification candidate pass rates.}, volume={11}, url={http://europepmc.org/abstract/med/23991467}, DOI={10.1370/afm.1556}, abstractNote={We continue to receive a number of inquiries from physicians regarding the low pass rate statistics for recertification candidates for the reporting periods from 2010 to 2012. Many physicians fear the examination has changed in some way, putting those attempting to maintain their certification at a}, number={4}, journal={Annals of Family Medicine}, author={Royal, KD and Puffer, JC}, year={2013}, pages={389–390} } @article{royal_brosh_2013, title={A more accurate pain scale?}, volume={26}, number={4}, journal={Rasch Measurement Transactions}, author={Royal, K. and Brosh, A.}, year={2013}, pages={1398} } @article{royal_thompson_2013, title={A psychometric validation of the beliefs toward mental illness scale.}, volume={21}, url={http://europepmc.org/abstract/med/24620521}, DOI={10.1891/1061-3749.21.3.516}, abstractNote={Background and Purpose: This study measured Protestant Christians’ beliefs about mental illness using the Beliefs Toward Mental Illness Scale (BMI). Methods: This study used a state-of-the-art item response theory technique, namely the Rasch rating scale model (RRSM), to evaluate the psychometric properties of the BMI and to produce a ruler of the construct hierarchy. Results: Using Messick’s framework for evaluating construct validity, sufficient evidence was obtained that suggests the BMI is psychometrically sound regarding substantive, content, structural, and generalizable aspects of validity. Conclusions: The BMI is a psychometrically sound instrument that is appropriate for measuring beliefs toward mental illness.}, number={3}, journal={Journal of Nursing Measurement}, author={Royal, KD and Thompson, JM}, year={2013}, pages={516–524} } @article{hart_mueller_royal_jones_2013, title={Achievement Goal Validation Among African American High School Students: CFA and Rasch Results}, volume={31}, url={http://www.scopus.com/inward/record.url?eid=2-s2.0-84878162205&partnerID=MN8TOARS}, DOI={10.1177/0734282912466726}, abstractNote={ Achievement goal theory helps describe how and why students engage in various academic behaviors. Historically, achievement goals have been examined almost exclusively with undergraduate, nonminority samples, and predominately with factor analytic techniques. The present study adds to a growing literature by providing initial validation of a leading achievement goal measure, the Achievement Goal Questionnaire-Revised (AGQ-R; Elliot & Murayama, 2008), among rural ( N = 186) and urban ( N = 197) African American high school students. Collectively, results from both confirmatory factor and Rasch analyses highlight issues that should be considered when using the AGQ-R among African American high school students. }, number={3}, journal={Journal of Psychoeducational Assessment}, author={Hart, C.O. and Mueller, C.E. and Royal, K.D. and Jones, M.H.}, year={2013}, pages={284–299} } @article{royal_puffer_2013, title={Criterion-referenced examinations: implications for the reporting and interpretation of examination results.}, volume={11}, url={http://europepmc.org/abstract/med/23508610}, DOI={10.1370/afm.1524}, abstractNote={The purpose of the American Board of Family Medicine (ABFM) certification/maintenance of certification examination is to measure the basic knowledge necessary to deliver high quality care to patients and their families. More than 25 years ago, the ABFM became the first American Board of Medical}, number={2}, journal={Annals of family medicine}, author={Royal, KD and Puffer, JC}, year={2013}, pages={185–187} } @article{royal_eli_2013, title={Developing a psychometric ruler: An alternative presentation of Rasch Measurement output}, volume={8}, number={3}, journal={Journal of Applied Quantitative Methods}, author={Royal, K.D. and Eli, J.A.}, year={2013}, pages={1–10} } @article{royal_puffer_2013, title={Dimensionality of the maintenance of certification for family physicians examination: Evidence of construct validity}, volume={26}, url={http://www.scopus.com/inward/record.url?eid=2-s2.0-84877751509&partnerID=MN8TOARS}, DOI={10.3122/jabfm.2013.03.130079}, abstractNote={The American Board of Family Medicine (ABFM) Maintenance of Certification for Family Physicians (MC-FP) examination is designed to measure a single construct: clinical decision-making abilities within the scope of practice of family medicine. Implied in the construct of clinical decision-making}, number={3}, journal={Journal of the American Board of Family Medicine}, author={Royal, K.D. and Puffer, J.C.}, year={2013}, pages={342–344} } @article{royal_puffer_2013, title={Dimensionality of the maintenance of certification for family physicians examination: evidence of construct validity.}, volume={11}, url={http://europepmc.org/abstract/med/23829001}, DOI={10.1370/afm.1536}, abstractNote={The American Board of Family Medicine (ABFM) Maintenance of Certification for Family Physicians (MC-FP) examination is designed to measure a single construct: clinical decision-making abilities within the scope of practice of family medicine. Implied in the construct of clinical decision-making abilities is the ability to recall relevant elements from a large fund of pertinent medical knowledge. While clinical decision-making abilities could be perceived as comprising several separate constructs (eg, based upon clinical categories, organ systems, etc), that approach would require the development of multiple assessment scales with a passing criteria specific to each. Instead, the overarching construct of clinical decision-making ability, which encompasses those more specific areas, has been selected by the ABFM because it more closely mirrors the pass-fail decision process used to discern which candidates receive certification. In any instance, the construct that the ABFM attempts to measure needs to be sufficiently unidimensional in order to produce precise, error-free estimates of a candidate’s performance. This brief article will discuss the dimensionality of the MC-FP examination and its implications for construct validity, namely the validation that the examination accurately measures the ability of family physicians to make appropriate clinical decisions.}, number={3}, journal={Annals of Family Medicine}, author={Royal, KD and Puffer, JC}, year={2013}, pages={286–288} } @article{royal_2013, title={G. ENGELHARD, JR. (2013) Invariant Measurement: Using Rasch Models in the Social, Behavioral, and Health Sciences.: New York: Routledge. 312 pages. US$44.95. ISBN: 978-0-415-87125-9}, volume={79}, url={http://www.scopus.com/inward/record.url?eid=2-s2.0-84919932654&partnerID=MN8TOARS}, DOI={10.1007/s11336-013-9398-1}, number={4}, journal={Psychometrika}, author={Royal, K.D.}, year={2013}, pages={733–735} } @article{royal_hedgpeth_2013, title={Investigating guessing strategies and their success rates on items of varying difficulty levels}, volume={27}, number={1}, journal={Rasch Measurement Transactions}, author={Royal, K.D. and Hedgpeth, M.W.}, year={2013}, pages={1407–1408} } @article{royal_lybarger_2013, title={Is now the time for a Rasch measurement MOOC?}, volume={27}, number={3}, journal={Rasch Measurement Transactions}, author={Royal, K.D. and Lybarger, M.}, year={2013}, pages={1437–1438} } @article{royal_gregg_2013, title={Objectively Measuring Change across Time in Higher Education Assessment}, volume={3}, ISSN={2160-6765}, url={http://dx.doi.org/10.5325/jasseinsteffe.3.1.0019}, DOI={10.5325/jasseinsteffe.3.1.0019}, abstractNote={Abstract}, number={1}, journal={Journal of Assessment and Institutional Effectiveness}, publisher={The Pennsylvania State University Press}, author={Royal, Kenneth and Gregg, Justin}, year={2013}, pages={19} } @article{jones_mueller_royal_shim_hart_2013, title={Social Achievement Goals: Validation Among Rural African American Adolescents}, volume={31}, url={http://www.scopus.com/inward/record.url?eid=2-s2.0-84887536026&partnerID=MN8TOARS}, DOI={10.1177/0734282913483982}, abstractNote={ Little extant research attempts to understand why rural African Americans engage in social relationships with peers in school. This is somewhat surprising as rural students’ peer interactions often affect their scholastic desires, and peers can alter African Americans’ academic performance. Hence, the current study examined both the presence and psychometric validity of social achievement goals among rural African American high school students. Results suggest the presence of three reasons for engaging in social relationships in school: social development (desire to increase friendship quality), social demonstration-approach (wanting to appear “cool” among friends), and social demonstration-avoid (fear of appearing socially inferior). Confirmatory factor analysis and Rasch analysis provide support for both the presence and valid measurement of social achievement goals among rural African American adolescents. }, number={6}, journal={Journal of Psychoeducational Assessment}, author={Jones, M.H. and Mueller, C.E. and Royal, K.D. and Shim, S.S. and Hart, C.O.}, year={2013}, pages={566–577} } @article{royal_2012, title={A suggestion for taking Rasch-based survey results even further}, volume={25}, number={4}, journal={Rasch Measurement Transactions}, author={Royal, K.D.}, year={2012}, pages={1341} } @article{royal_puffer_2012, title={Cheating: Its implications for American Board of Family Medicine examinees}, volume={25}, url={http://www.scopus.com/inward/record.url?eid=2-s2.0-84861849913&partnerID=MN8TOARS}, DOI={10.3122/jabfm.2012.03.120049}, abstractNote={Cheating is undesirable and unethical, but, unfortunately, sometimes it does occur. Recent events at 3 American Board of Medical Specialties specialty boards[1][1],[2][2] have illustrated that the medical certification industry is not immune from this phenomenon. Although there are numerous moral}, number={3}, journal={Journal of the American Board of Family Medicine}, author={Royal, KD and Puffer, JC}, year={2012}, pages={400–401} } @article{royal_puffer_2012, title={Cheating: its implications for ABFM examinees.}, volume={10}, url={http://europepmc.org/abstract/med/22585897}, DOI={10.1370/afm.1408}, abstractNote={Cheating is undesirable and unethical, but unfortunately, sometimes it does occur. Recent events at 2 ABMS specialty boards1,2 have illustrated the fact that the medical certification industry is not immune from this phenomenon either. Although there are numerous moral and professional implications involved with cheating, we wish to address the implications of cheating from a psychometric perspective. Our intent is to highlight some of the less obvious ways in which all ABFM diplomates could possibly be impacted should its diplomates and candidates resort to cheating on our examinations. So, what is cheating? Cizek3 defines it as “any action that violates the rules for administering a test, any behavior that gives an examinee an unfair advantage over other examinees, or any action on the part of an examinee or test administrator that decreases the accuracy of the intended inferences arising from the examinee’s test score or performance.” The ABFM goes to great length to ensure a fair test for all examinees. When examinees register for ABFM exams, they make a promise to adhere to both the ethical and legal standards associated with the administration of the examination. This compact between the ABFM and the candidates minimizes the risk of a compromised examination score(s). Unfortunately, however, when members of either party fail to adhere to the agreed upon standards problems can arise.}, number={3}, journal={Annals of Family Medicine}, author={Royal, KD and Puffer, JC}, year={2012}, pages={274–275} } @misc{mueller_hart_royal_jones_2012, title={Examining Achievement Goal Orientations in Rural African American High School Students: CFA Results}, DOI={10.1037/e641402012-001}, journal={PsycEXTRA Dataset}, publisher={American Psychological Association (APA)}, author={Mueller, Christian E. and Hart, Caroline O. and Royal, Kenneth D. and Jones, Martin H.}, year={2012} } @article{royal_2012, title={Measuring Liberal/Conservative Voting Tendencies among U.S. Senators}, volume={26}, number={2}, journal={Rasch Measurement Transactions}, author={Royal, K.D.}, year={2012}, pages={1366–1367} } @article{royal_thompson_2012, title={Measuring protestant Christians' willingness to seek professional psychological help for mental illness: A Rasch Measurement analysis}, volume={31}, number={3}, journal={Journal of Psychology and Christianity}, author={Royal, K.D. and Thompson, J.M.}, year={2012}, pages={195–204} } @article{royal_puffer_2012, title={The reliability of american board of family medicine examinations: Implications for test takers}, volume={25}, url={http://www.scopus.com/inward/record.url?eid=2-s2.0-84855643504&partnerID=MN8TOARS}, DOI={10.3122/jabfm.2012.01.110292}, abstractNote={A common theme among family physicians who have performed poorly repeatedly on the American Board of Family Medicine (ABFM) Maintenance of Certification (MC-FP) examination is the complaint that they received a score that was identical, or almost identical, to their score on a previous}, number={1}, journal={Journal of the American Board of Family Medicine}, author={Royal, KD and Puffer, JC}, year={2012}, pages={131–133} } @article{royal_puffer_family medicine_2012, title={Understanding the "sum of subtest to overall score discrepancy" on the MC-FP examination.}, volume={10}, url={http://europepmc.org/abstract/med/22230837}, DOI={10.1370/afm.1358}, abstractNote={When high-stakes examinations, such as the American Board of Family Medicine’s (ABFM) Maintenance of Certification-in Family Practice (MC-FP) examination are administered, candidates and diplomates are keenly interested in the accuracy of their test scores, especially when their scores are close to, but below, the pass/fail cut point. In some instances, candidates will attempt to reverse engineer their scores using the information provided on the score report in an effort to verify the “weighted sum of the subtest scores” is congruent with the overall test score. Any discrepancy might become alarming to the candidate, providing a seemingly legitimate reason to believe the overall score was inaccurate, thus prompting a phone call to the ABFM for further investigation and clarification. Historically, such a mistake in scoring has never been found; however, a statistical phenomenon that we will describe below could make it appear so. We would like to explain this phenomenon so that examinees who attempt to reverse engineer their score reports will better understand the “sum of subtest to overall score discrepancy” phenomenon.}, number={1}, journal={Annals of Family Medicine}, author={Royal, KD and Puffer, JC and Family Medicine, Society}, year={2012}, pages={81–82} } @article{royal_puffer_2012, title={Understanding the "sum of subtest to overall score discrepancy" on the maintenance of certification-family practice examination}, volume={25}, url={http://www.scopus.com/inward/record.url?eid=2-s2.0-84859318063&partnerID=MN8TOARS}, DOI={10.3122/jabfm.2012.02.120003}, abstractNote={When high-stakes examinations, such as the American Board of Family Medicine’s (ABFM’s) Maintenance of Certification-in Family Practice (MCFP) examination, are administered, candidates and diplomates are keenly interested in the accuracy of their test scores, especially when their scores are close to, but below, the pass/fail cutpoint. In some instances, candidates will attempt to reverse engineer their scores using the information provided on the score report in an effort to verify that the “weighted sum of the subtest scores” is congruent with the overall test score. Any discrepancy might become alarming to the candidate, providing a seemingly legitimate reason to believe the overall score was inaccurate, thus prompting a phone call to the ABFM for further investigation and clarification. Historically, such a mistake in scoring has never been found; however, a statistical phenomenon that we will describe below could make it appear so. We would like to explain this phenomenon so that examinees who attempt to reverse engineer their score reports will better understand the “sum of subtest to overall score discrepancy” phenomenon.}, number={2}, journal={Journal of the American Board of Family Medicine}, author={Royal, K.D. and Puffer, J.C.}, year={2012}, pages={260–261} } @article{royal_2012, title={Using Stylometric Techniques to Evaluate New Testament Authorship}, volume={8}, number={19}, journal={Journal of MultiDisciplinary Evaluation}, author={Royal, K.D.}, year={2012}, pages={1–7} } @article{eubanks_royal_2011, title={A survey of attitudes about methods of assessment}, volume={23}, number={3}, journal={Assessment Update}, author={Eubanks, D.A. and Royal, K.D.}, year={2011}, month={May}, pages={1–15} } @article{o neill_raddatz_royal_2011, title={Demonstrating the Construct Stability of a Translated Exam for Family Medicine Residents}, volume={6}, number={2}, journal={The International Journal of Educational and Psychological Assessment}, author={O Neill, T.R. and Raddatz, M. and Royal, K.D.}, year={2011}, pages={31–41} } @article{edkins_royal_2011, title={Evaluating the due process and crime control perspectives using Rasch measurement analysis}, volume={7}, number={16}, journal={Journal of MultiDisciplinary Evaluation}, author={Edkins, V.A. and Royal, K.D.}, year={2011}, pages={48–65} } @misc{hart_royal_mueller_2011, title={Examination of a Modified Achievement Goal Questionnaire for African American Middle School Students: A Rasch Analysis}, DOI={10.1037/e696352011-001}, journal={PsycEXTRA Dataset}, publisher={American Psychological Association (APA)}, author={Hart, Caroline O. and Royal, Kenneth D. and Mueller, Christian E.}, year={2011} } @article{royal_oneill_2011, title={Explaining discrepancies between the sum of subtest scale-scores to total scale-scores}, volume={25}, number={3}, journal={Rasch Measurement Transactions}, author={Royal, K.D. and ONeill, T.R.}, year={2011}, pages={1338} } @article{royal_parrent_clark_2011, title={Measuring education majors perceptions of academic misconduct: An Item Response Theory perspective}, volume={7}, DOI={10.21913/IJEI.v7i1.741}, abstractNote={The purpose of this study was to construct a psychometric ruler that illustrates university undergraduate education majors' perceptions of academic misconduct. A survey consisting of 38 items that pertain to issues of academic misconduct were administered to an undergraduate sample at a large state university. Utilising Rasch measurement analyses to construct objective measures, students' responses were modeled along a truly linear and equal interval continuum to produce a hierarchy of perceived academic offenses. Results and policy implications are discussed.}, number={1}, journal={International Journal for Education Integrity}, author={Royal, K.D. and Parrent, J. and Clark, R.}, year={2011}, pages={18–29} } @article{royal_2011, title={Media Review: Demonstrating Student Success: A Practical Guide to Outcomes-Based Assessment of Learning and Development in Student Affairs}, volume={48}, ISSN={1949-6591 1949-6605}, url={http://dx.doi.org/10.2202/1949-6605.6314}, DOI={10.2202/1949-6605.6314}, abstractNote={Demonstrating Student Success is intended to be a how-to resource for student affairs professionals conducting outcomes assessment. The authors begin their discussion by providing a well-crafted historical overview of higher education outcomes assessment. The second chapter is particularly important, and controversial, because the authors define outcomes assessment and what they believe it should be. They attempt to distinguish research from assessment and suggest that “assessmentmay rarely followstrict researchmethodology” (Bresciani,Gardner,&Hickmott, 2010, p. 19), which is certainly a contentious assertion, as most assessment professionals consider outcomes assessment a form of research. The remainder of the book is writtenwith the aforementioned theoretical perspective of outcomes assessment in mind, which can be a major strength if}, number={3}, journal={Journal of Student Affairs Research and Practice}, publisher={Informa UK Limited}, author={Royal, Kenneth D.}, year={2011}, month={Jul}, pages={381–383} } @article{o'neill_royal_puffer_2011, title={Performance on the American Board of Family Medicine (ABFM) certification examination: are superior test-taking skills alone sufficient to pass?}, volume={24}, url={http://europepmc.org/abstract/med/21383217}, DOI={10.3122/jabfm.2011.02.100162}, abstractNote={Introduction: Certification examinations used by American specialty boards have been the sine qua non for demonstrating the knowledge sufficient for attainment of board certification in the United States for more than 75 years. Some people contend that the examination is predominantly a test of superior test-taking skills rather than of family medicine decision-making ability. In an effort to explore the validity of this assertion, we administered the American Board of Family Medicine (ABFM) Certification to examinees who had demonstrated proficiency in taking standardized tests but had limited medical knowledge. Methods: Four nonphysician experts in the field of measurement and testing were administered one version of the 2009 ABFM certification examination. Scaled scores were calculated for each examinee, and psychometric analyses were performed on the examinees responses to examination items and compared with the performance of physicians who took the same examination. Results: The minimum passing threshold for the examination was a scaled score of 390, corresponding to 57.7% to 61.0% of questions answered correctly, depending on the version of the examination. The 4 nonphysician examinees performed poorly, with scaled scores that ranged from 20 to 160 (mean, 87.5; SD, 57.4). The number of questions answered correctly ranged from 24.0% to 35.1% (mean, 29.2%; SD, 0.05%). Rasch analyses of the examination items revealed that the nonphysician examinees were more likely to use guessing strategies in an effort to answer questions correctly. Distracter analysis suggest near-complete randomness in the nonphysician responses. Conclusions: Though all 4 nonphysician examinees performed better than would have been predicted by chance alone, none performed well enough to even fall within 8 SE below the passing thresholds; their performance was far below that of almost all physicians who completed the examination. Given that the nonphysicians relied heavily on the identifying cues in the phrasing of items and the manner in which response options were presented, the results affirm the notion that the ABFM certification examination is not primarily a measure of generic test-taking ability but measures information critical to the estimation of a family physician's knowledge sufficient for certification. Item analysis confirmed that items were well written, provided minimal cueing, and required medical knowledge to answer correctly.}, number={2}, journal={Journal of the American Board of Family Medicine}, author={O'Neill, TR and Royal, KD and Puffer, JC}, year={2011}, pages={175–180} } @article{royal_elahi_2011, title={Psychometric properties of the death anxiety scale (DAS) among terminally ill cancer patients}, volume={29}, url={http://www.scopus.com/inward/record.url?eid=2-s2.0-79961139751&partnerID=MN8TOARS}, DOI={10.1080/07347332.2011.582639}, abstractNote={Research conducted with the terminally ill population in relation to death anxiety is rare and mostly outdated. The purpose of this study was to evaluate the psychometric properties of the widely used Death Anxiety Scale (DAS) on a sample of terminal cancer patients. Additionally, validation studies of the DAS have exclusively used traditional statistical methods for analysis. The current study utilized an item response theory technique (IRT), namely the Rasch Rating Scale model for data analysis. The methodology employed may be useful for other researchers conducting validation studies from an IRT perspective.}, number={4}, journal={Journal of Psychosocial Oncology}, author={Royal, KD and Elahi, F}, year={2011}, pages={359–371} } @article{royal_puffer_2011, title={The reliability of ABFM examinations: implications for test-takers.}, volume={9}, url={http://europepmc.org/abstract/med/21911766}, DOI={10.1370/afm.1303}, abstractNote={A common theme among family physicians that have repeatedly performed poorly on the ABFM Maintenance of Certification (MC-FP) Examination is the complaint that they received a score that was identical, or almost identical to their score on a previous administration of the exam. From their perspective, it is a mystery as to why they received the exact same score (or a very similar score), despite additional study time and preparation. Often, physicians assume a mixup has occurred and ask if it is possible that results have erroneously been provided from their previous attempt. After a psychometric review, it is clear that there is no mistake at all. In fact, we anticipate many test-takers will receive a comparable score on future attempts at successfully taking the exam. We base this anticipation on the psychometric concept of reliability.}, number={5}, journal={Annals of Family Medicine}, author={Royal, KD and Puffer, JC}, year={2011}, pages={463–464} } @article{royal_2011, title={Understanding reliability in higher education student learning outcomes assessment}, volume={2}, number={2}, journal={Quality Assurance in Higher Education}, author={Royal, K.D.}, year={2011}, pages={8–15} } @article{royal_o'neill_2011, title={Using the CUTLO procedure to investigate guessing}, volume={25}, number={1}, journal={Rasch Measurement Transactions}, author={Royal, K.D. and O'Neill, T.R.}, year={2011}, pages={1319–1320} } @article{royal_2010, title={Evaluating faculty perceptions of student learning outcomes: A Rasch Measurement Analysis}, volume={6}, number={14}, journal={Journal of MultiDisciplinary Evaluation}, author={Royal, K.D.}, year={2010}, month={Aug}, pages={18–31} } @article{royal_eli_bradley_2010, title={Exploring community college faculty perceptions of student outcomes: Findings of a pilot study}, volume={34}, url={http://www.scopus.com/inward/record.url?eid=2-s2.0-79960092696&partnerID=MN8TOARS}, DOI={10.1080/10668920701827025}, abstractNote={This study explored the paradigmatic differences in perceptions of community college faculty employed at select Virginia and West Virginia community colleges collected via a web-based survey. The study is framed within the faculty self-classification along the “hard” and “social/behavior” science paradigm continuum. Given the paradigmatic continuum, faculty perceptions' of student outcomes were examined. Faculty respondents consistently reported the importance of intellectual growth; however, differences in relative importance of outcomes tied to emotional, cultural, and social growth exist. The potential implications of these perceptions on student experiences and outcomes are considered.}, number={7}, journal={Community College Journal of Research and Practice}, author={Royal, K.D. and Eli, J.A. and Bradley, K.D.}, year={2010}, pages={523–540} } @article{royal_2010, title={How skeptical are magicians?}, volume={24}, number={3}, journal={Rasch Measurement Transactions}, author={Royal, K.D.}, year={2010}, pages={1291} } @article{royal_2010, title={Making meaningful measurement in survey research: A demonstration of the utility of the Rasch model}, volume={28}, journal={IR Applications}, author={Royal, K.D.}, year={2010}, pages={1–16} } @article{royal_ensslen_ellis_homan_2010, title={Rating scale optimization in survey research: An application of the Rasch Rating Scale Model}, volume={5}, url={http://www.jaqm.ro/issues/volume-5,issue-4/7_royal_ellis_ensslen_homan.php.}, number={4}, journal={Journal of Applied Quantitative Methods}, author={Royal, K.D. and Ensslen, A. and Ellis, A. and Homan, A.}, year={2010}, pages={607–617} } @article{bradley_royal_bradley_2008, title={An investigation of honesty check items in higher education course evaluations}, volume={5}, DOI={10.19030/tlc.v5i8.1240}, abstractNote={The reliability and validity of course evaluations in higher education is often assumed. The typical Likert-type surveys utilized when students' evaluate the course and instructor often overlook measurement issues, or deal with them in an ineffective manner. Given the importance that is placed on higher education course evaluations, with results impacting such events as merit raises and promotion, the proper construction and use of evaluation tools is a critical issue. In an effort to assure 'honesty' in student responses, many institutions include items written positively and negatively, which are intended to measure the same construct. Using 537 course evaluations for a mathematics faculty member at a Midwest college, an item analysis is conducted with attention given to means and standard deviations, frequency counts, nonparametric correlations and tests of significant differences between questions that should, in theory, produce a similar measure or exactly opposite. A contention is made that the way the item is asked does matter, at least in some instances, and it should not be assumed that an item written in the positive and negative should directly correlate. The survey research community and institutions utilizing similar rating scale instruments will benefit from the results of this study, as well as the education community in general.}, number={8}, journal={Journal of College Teaching and Learning}, author={Bradley, K.D. and Royal, K.D. and Bradley, W.B.}, year={2008}, month={Aug}, pages={39–48} } @article{bradley_royal_cunningham_eli_weber_2008, title={What constitutes good educational research? A consideration of ethics, methods, and theory}, volume={21}, number={1}, journal={Mid-Western Educational Research Journal}, author={Bradley, K.D. and Royal, K.D. and Cunningham, J.C. and Eli, J.A. and Weber, J.}, year={2008}, pages={26–35} } @article{bradley_sampson_royal_2006, title={Applying the Rasch rating scale model to gain insights into students' conceptualisation of quality mathematics instruction}, volume={18}, url={http://www.scopus.com/inward/record.url?eid=2-s2.0-69949085145&partnerID=MN8TOARS}, DOI={10.1007/BF03217433}, number={2}, journal={Mathematics Education Research Journal}, author={Bradley, K.D. and Sampson, S.O. and Royal, K.D.}, year={2006}, pages={11–26} } @article{royal_bradley_lineberry_2005, title={Evaluating Interactive Television Courses: An Identification of Factors Associated with Student Satisfaction}, volume={8}, number={2}, journal={Online Journal of Distance Learning Administration}, author={Royal, K.D. and Bradley, K.D. and Lineberry, G.T.}, year={2005} } @article{royal_bradley_2005, title={Interactive television courses and students satisfaction}, volume={8}, number={4}, journal={Journal of Distance Learning Administration}, author={Royal, K.D. and Bradley, K.D.}, year={2005} } @misc{royal, title={On Algorithms, ‘Big Data’ and the Future of Psychometrics}, DOI={10.22541/au.149240152.20345892}, abstractNote={Kenneth D. Royal and Melanie Lybarger The topic of automation replacing human jobs has been receiving a great deal of media attention in recent months. In January, the McKinsey Global Institute (Manyika et al., 2017) published a report stating 51% of job tasks (not jobs) could be automated with current technologies. The topic of ‘big data’ and algorithms was also briefly discussed on the Rasch listserv last year and offered a great deal of food-for-thought regarding the future of psychometrics in particular. Several individuals noted a number of automated scoring procedures are being developed and fine-tuned, and each offer a great deal of promise. Multiple commenters noted the potential benefits of machine scoring using sophisticated algorithms, such as power, precision, and reliability. Some comments even predicted humans will become mostly obsolete in the future of psychometrics. Certainly, there is much to get excited about when thinking about the possibilities. However, there remain some issues that should encourage us to proceed with extreme caution. The Good For many years now algorithms have played a significant role in our everyday lives. For example, if you visit an online retailer’s website and click to view a product, you will likely be presented a number of recommendations for related products based on your presumed interests. In fact, years ago Amazon employed a number of individuals whose job was to critique books and provide recommendations to customers. Upon developing an algorithm that analyzed data about what customers had purchased, sales increased dramatically. Although some humans were (unfortunately) replaced with computers, the ‘good’ was that sales skyrocketed for both the immediate and foreseeable long-term future and the company was able to employ many more people. Similarly, many dating websites now use information about their subscribers to predict matches that are likely to be compatible. In some respects, this alleviates the need for friends and acquaintances to make what are often times awkward introductions between two parties, and feel guilty if the recommendation turns out to be a bad one. The ‘good’, in this case, is the ability to relieve people that have to maintain relationships with each party of the uncomfortable responsibility of playing matchmaker. While the aforementioned algorithms are generally innocuous, there are a number of examples that futurists predict will change most everything about our lives. For example, in recent years Google’s self-driving cars have gained considerable attention. Futurists imagine a world in which computerized cars will completely replace the need for humans to know how to drive. These cars will be better drivers than humans - they will have better reflexes, enjoy greater awareness of other vehicles, and will operate distraction-free (Marcus, 2012). Further, these cars will be able to drive closer together, at faster speeds, and will even be able to drop you off at work while they park themselves. Certainly, there is much to look forward to when things go as planned, but there is much to fear when things do not. The Bad Some examples of algorithmic failures are easy to measure in terms of costs. In 2010, the ‘flash crash’ occurred when an algorithmic failure from a firm in Kansas who ordered a single mass sell and triggered a series of events that led the Dow Jones Industrial Average into a tailspin. Within minutes, nearly $9 trillion in shareholder value was lost (Baumann, 2013). Although the stocks later rebounded that day, it was not without enormous anxiety, fear and confusion. Another example involving economics also incorporates psychosocial elements. Several years ago, individuals (from numerous countries) won lawsuits against Google when the autocomplete feature linked libelous and unflattering information to them when their names were entered into the Google search engine. Lawyers representing Google stated "We believe that Google should not be held liable for terms that appear in autocomplete as these are predicted by computer algorithms based on searches from previous users, not by Google itself." (Solomon, 2011). Courts, however, sided with the plaintiffs and required Google to manually change the search suggestions. Another example involves measures that are more abstract, and often undetectable for long periods of time. Consider ‘aggregator’ websites that collect content from other sources and reproduces it for further proliferation. News media sites are some of the most common examples of aggregators. The problem is media organizations have long been criticized with allegations of bias. Cass Sunstein, Director of the Harvard Law School's program on Behavioral Economics and Public Policy, has long discussed the problems of ‘echo chambers’, a phenomenon that occurs when people consume only the information that reinforces their views (2009). This typically results in extreme views, and when like-minded people get together, they tend to exhibit extreme behaviors. The present political landscapes in the United States (e.g., democrats vs. republicans) and Great Britain (e.g., “Brexit” - Britain leaving the European Union) highlight some of the consequences that result from echo chambers. Although algorithms may not be directly responsible for divisive political views throughout the U.S. (and beyond), their mass proliferation of biased information and perspectives certainly contributes to group polarization that may ultimately leave members of a society at odds with one another. Some might argue these costs are among the most significant of all. The Scary Gary Marcus, a professor of cognitive science at NYU, has published a number of pieces in The New Yorker discussing what the future may potentially hold if (and when) computers and robots reign supreme. In a 2012 article he presents the following scenario: Your car is speeding along a bridge at fifty miles per hour when an errant school bus carrying forty innocent children crosses its path. Should your car swerve, possibly risking the life of its owner (you), in order to save the children, or keep going, putting all forty kids at risk? If the decision must be made in milliseconds, the computer will have to make the call. Marcus’ example underscores a very serious problem regarding algorithms and computer judgments. That is, when we outsource our control we are also outsourcing our moral and ethical judgment. Let us consider another example. The Impermium corporation, which was acquired by Google in 2014, was essentially an anti-spam company whose software purported to automatically “identify not only spam and malicious links, but all kinds of harmful content—such as violence, racism, flagrant profanity, and hate speech—and allows site owners to act on it in real-time, before it reaches readers.” As Marcus (2015) points out, how does one “translate the concept of harm into the language of zeroes and ones?” Even if a technical operation was possible to do this, there remains the problem that morality and ethics is hardly a universally agreed upon set of ideals. Morality and ethics are, at best, a work-in-progress for humans, as cultural differences and a host of contextual circumstances presents an incredibly complex array of confounding variables. These types of programming decisions could have an enormous impact on the world. For example, algorithms that censor free speech in democratic countries could spark civil unrest among people already suspicious of their government; individuals flagged to be in violation of an offense could have his/her reputation irreparably damaged, be terminated by an employer, and/or charged with a crime(s). When we defer to computers and algorithms to make our decisions for us, we are entrusting that they have all the ‘right’ answers. This is a very scary proposition given the answers fed to machines come from data, which are often messy, out-of-date, subjective, and lacking in context. An additional concern involves the potential to program evil into code. While it is certainly possible that someone could program evil as part of an intentional, malicious act (e.g., terrorism), we are referring to evil in the sense of thoughtless actions that affect others. Melissa Orlie (1997), expanding on the idea of “ethical trespassing” as originally introduced by political theorist Hannah Arendt, discusses the notion of ‘ordinary evil’. Orlie argues that despite our best intentions, humans inevitably trespass on others by failing to predict every possible way in which our decisions might impact others. Thoughtless actions and unintended consequences must, therefore, be measured, included, and accounted for in our calculations and predictions. That said, the ability to do this perfectly in most contexts can never be achieved, so it would seem each day would present a new potential to open Pandora’s Box. Extensions to Psychometrics Some believe the ‘big data’ movement and advances in techniques designed to handle big data will, for the most part, make psychometricians obsolete. No one knows for sure what the future holds, but at present that seems to be a somewhat unlikely proposition. First, members of the psychometric community are notorious for being incredibly tedious with respect to not only the accuracy of information, but also the inferences made and the way in which results are used. Further, it is apparent that the greatest lessons learned from previous algorithmic failures pertains to the unintended consequences, albeit economically, socially, culturally, politically, and legally that may result (e.g., glitches that result in stock market plunges, legal liability for mistakes, increased divisions in political attitudes, etc.). Competing validity conceptualizations aside, earnest efforts to minimize unintended consequences is something most psychometricians take very seriously and already do. If anything, it seems a future in which algorithms are used exclusively could only be complemented by psychometricians who perform algorithmic audits (Morozov, 2013) and think meticulously about identifying various ‘ordinary evils’. Perhaps instead of debating whether robots are becoming more human or if humans are becoming more robotic, we would be better off simply appreciating and leveraging the strengths of both? References Baumann, N. (2013). Too fast to fail: How high-speed trading fuels Wall Street disasters. Mother Jones. Available at: http://www.motherjones.com/politics/2013/02/high-frequency-trading-danger-risk-wall-street Manyika, J., Chui, M., Miremadi, M., Bughin, J., George, K., Willmott, P., & Dewhurst, M. (2017). A future that works: Automation, employment, and productivity. The McKinsey Global Institute. Available at: http://www.mckinsey.com/global-themes/digital-disruption/harnessing-automation-for-a-future-that-works Marcus, G. (2012). Moral machines. The New Yorker. Available at: http://www.newyorker.com/news/news-desk/moral-machines Marcus, G. (2015). Teaching robots to be moral. The New Yorker. Available at: http://www.newyorker.com/tech/elements/teaching-robots-to-be-moral Morozov, E. To Save Everything, Click Here: The Folly of Technological Solutionism (2013). PublicAffairs Publishing, New York, NY. Orlie, M. (1997). Living ethically, acting politically. Cornell University Press, Ithaca, NY. Solomon, K. (2011). Google loses autocomplete lawsuit. Techradar. Available at: http://www.techradar.com/news/internet/google-loses-autocomplete-lawsuit-941498 Sunstein, C. R. (2009). Republic.com 2.0. Princeton University Press, Princeton, NJ.}, journal={Authorea}, author={Royal, Kenneth} }