Introduction

The Board of Pharmacy Specialties (BPS), an autonomous division of the American Pharmacists Association (APhA), seeks to improve patient care by promoting the recognition and value of specialized training, knowledge, and skills in pharmacy and board certification of pharmacist specialists. Board of Pharmacy Specialties specifies the requirements for earning and maintaining professional credentials (ie, board certifications) in recognized pharmacist practice specialties. The BPS Board Certified Psychiatric Pharmacist (BCPP) program is a credential for pharmacists who in their unique practice provide person-centered care that ensures the safe, appropriate, evidence-based, and cost-effective use of medications in the treatment of persons with mental illnesses.¹

Launched in 1994, the purpose of the BCPP program is to validate that the pharmacist has the advanced knowledge and experience to optimize outcomes and recovery for patients with mental illness by

• Designing, implementing, monitoring, and modifying treatment plans for patients.

• Educating patients, health care professionals, and other stakeholders.

• Providing leadership in the health system and public policy to improve the health of persons with mental illness.

The minimum requirements for the Psychiatric Pharmacy Certification include the demonstration of practice experience in 1 of 3 ways:²

• Four years of post-licensure/registration experience, including 50% or more of that time spent practicing in the domains delineated in the Psychiatric Pharmacy Content Outline.

• Completion of a PGY-1 residency plus 2 additional years of post-licensure/registration experience, including 50% or more of that time spent practicing in the domains delineated in the Psychiatric Pharmacy Content Outline.

• Completion of a specialty (PGY-2) residency in psychiatric pharmacy.

After meeting the eligibility criteria for the certification, the final step in earning the BCPP certification is passing a standardized examination. Since 2014, the BCPP Examination has been offered in biannual test administration cycles, in the spring (late April to mid-May) and in the fall (late September to mid-October). While the total score is the primary determinant for passing/failing status, performance is also evaluated on 3 subdomains. This 3-domain structure, along with associated examination weights (%), are the result of a Role Delineation Study, which is typically reevaluated on a 5-year cycle:

1. Person-Centered Care (55% of the examination, 96 questions)

2. Translation of Evidence into Practice and Education (30% of the examination, 53 questions)

3. Healthcare Policy, Advocacy, and Practice Management (15% of the examination, 26 questions)

Among the many indicators that are monitored for a high-stakes certification examination, the priority indicator is the passing rate or the proportion of candidates who pass the examination. Mean scores and pass rates should remain relatively stable from testing sample to testing sample; however, when there are wide swings in the observed sample estimates, the characteristics of the samples should be evaluated for explanatory factors.

Several factors can influence passing rates in examinations. Statistical sampling theory dictates that sample size has an inverse relationship to sampling error. The higher the sample size, the lower the sampling error and the less variation is seen in parameter estimates with repeated sampling. The BCPP program features comparatively smaller enrollments among BPS certifications. Due to sampling error, more fluctuation in passing rates tends to be seen in smaller testing programs. While some fluctuation is anticipated, recent trends in the BCPP passing rate feature variation that is greater than what would be expected due to sampling error. This finding leads to evaluation of candidate sample characteristics as the source of the fluctuation.

Several candidate characteristics can influence pass rates:

• Repeater status. Repeat test-takers represent candidates who have failed the examination on previous attempts. Thus, they reflect lower levels of demonstrated knowledge, which the test is designed to objectively measure. Contrary to widespread belief, the memory effects (potential score gains attributable to previously seeing examination content) owing to repeat administration tend not to influence the subsequent performance.³ A large preponderance of repeat test-takers will result in diminished overall performance on the exam and a reduction in the pass rate.

• International status. The recent analysis demonstrates that, for most BPS specialty examinations, international candidates tend to underperform candidates from the United States. This is likely because BPS examinations are based upon the US standards of practice and are made available internationally to candidates meeting eligibility criteria. They are not international examinations, per se. This information is disclosed to candidates in advance of testing through the BPS Candidate's Guide. Most international candidates who qualify for the BCPP Examination do so through the practice experience route. Roles for psychiatric pharmacists may vary based on an individual country's standard of practice, and this variation has relevance for how non-US psychiatric pharmacists perform on the examination, given that the assessment is based on standards and practice within the United States.

• Eligibility pathway. Performance among the 3 cohorts of candidates is statistically distinct, with PGY-2 residency candidates exhibiting higher scores than PGY-1 residency candidates, and PGY-1 candidates exhibiting higher scores than candidates qualifying by practice experience alone. These differences are observed across pharmacy specialties and were investigated in a recent publication.⁴

The pass rate for the BCPP Examination has fluctuated in recent test administration cycles. In particular, a consistent and progressive decrease in the pass rate since the Fall 2017 administration has caused concern for some Psychiatric Pharmacy stakeholders. This report presents the results of an observational, retrospective analysis of pass/fail outcomes for the BCPP program over recent years. Analyses presented hereafter include a trend analysis of pass rates, as well as an evaluation of factors contributing to fluctuation in the performance of the examination.

Methods

The primary focus of the present analyses is the passing rate (percentage of candidates passing) for the exam. To remove known confounders, pass rates were calculated using only data since the implementation of the last passing standard study, which was set in 2017, and removing repeater and international candidates from the analysis (ie, US and first-time-only candidates were included in the analysis).

It is hypothesized that a primary factor in the fluctuation of the pass rate is the relative prevalence of the respective eligibility pathways. Therefore, the pass-rate trend analysis also will include data points reflecting the proportion of each candidate cohort comprising the practice experience pathway.

In addition to the pass-rate trend analysis, we also evaluate the extent to which performance on the 3 domains of the BCPP Examination may influence performance in the 3 eligibility pathway groups. We investigate these effects through a 1-way ANOVA, inspecting differences in performance on each of the domains across the eligibility pathways. We surmise that the disparate domain-level performance may be exhibited among the eligibility groups, and that these differences in performance may shed light on whether certain topics assessed may influence overall performance. The ANOVA results are supplemented by a conservative post-hoc test (Tamhane,⁵ with equal variances not assumed) to evaluate significance of differences between each group, as well as effect sizes (Cohen η²) to ascertain on what domains we see larger effects.

Results

Pass-Rate Trends

Table 1 contains the first-time US passing rates for each administration cycle since Fall 2015.

TABLE 1 Pass rates for the Board Certified Psychiatric Pharmacist Examination, by testing window

View Fullscreen

As expected, the pass rate fluctuates considerably due to sampling error associated with sample sizes less than 100. There was a steady decline in the pass rate since the Spring 2017 window when the pass rate peaked at 87.9%. The lowest observed pass rate in recent years was for the Spring 2019 window, when the US first-time enrollment and the pass rate reached a minimum (N  = 48, 58.3%). However, the pass rate appears to have rebounded somewhat with the Fall 2019 examination window.

Figure 1 represents graphically the results of Table 1. The first-time US pass rate is indicated by the solid line at the top of the graph. The bottom dotted line represents the percentage of the candidate cohort comprising the practice experience eligibility pathway. For instance, the observed pass rate for the Spring 2019 testing window was 58.3%, and 42.0% of the Fall 2017 candidate group qualified for the exam by the practice experience route.

View Figure

• Download as Powerpoint
• Download Figure

Although the pattern is neither definitive nor invariably consistent, there is an inverse relationship between the pass rate and the preponderance of candidates who qualified by the practice experience pathway. As the prevalence of the 4-year experience cohort increases, the pass rate decreases, and vice versa. This pattern is particularly noticeable from Fall 2017 to Spring 2019; the percentage of the 4-year eligibility cohort steadily increases, while the pass rate decreases. Most notably, the rebound in the pass rate in Fall 2019 corresponds to a marked decrease in the influx of this eligibility group (79.2% pass rate, 24.4% of the sample qualified by the 4-year experience pathway).

Figure 2 demonstrates the more direct relationship between the pass rate and preponderance of PGY-2 graduates in the test-taking sample. This finding is seen most dramatically in Fall 2019, with a marked rise in the PGY-2 cohort and a corresponding increase in the pass rate.

View Figure

• Download as Powerpoint
• Download Figure

Domain Score ANOVA

Three different ANOVAs were conducted, 1 for each set of domain scores, and each evaluated difference in the 3 eligibility groups. Table 2 contains the results of the ANOVAs.

TABLE 2 ANOVA results including descriptive statistics, for sub-domain scores by eligibility route

View Fullscreen

Each ANOVA resulted in significant F tests, indicating significant group differences in each of the domain-level scores. There were significant group differences in each of the domain-level scores based on the ANOVA completed (Patient-Centered Care, F(2, 643) = 97.5, P < .05; Translation of Evidence into Practice and Education, F(2, 643) = 58.0, P < .05; Healthcare Policy, Advocacy, and Practice Management, F(2, 643) = 10.2, P < .05). A post-hoc Tamhane test revealed the following specific ordered group difference:

• Patient-Centered Care: All 3 eligibility groups were statistically distinct (the scores for the PGY-2 group were significantly higher than those of the PGY-1 groups, which were in turn significantly higher than those of the practice experience group).

• Translation of Evidence into Practice and Education: Two distinct performance groups were detected, with PGY-1 and PGY-2 candidates outperforming the practice experience cohort (ie, the PGY-1 and PGY-2 groups were not significantly different).

• Healthcare Policy, Advocacy, and Practice Management: The only statistically distinct groups were PGY-2 and practice experience groups.

Using Cohen η² as a measure of effect size, the disparate performance seen across eligibility groups is most pronounced on Domain 1 (effect size η² = 0.21), followed by Domain 2 (η² = 0.13), and Domain 3 (η² = 0.03).

Discussion

This study represents the first comprehensive study of examination performance trends for the BCPP Examination. It was conceived out of questions regarding the observed patterns in the pass rate for the BCPP Examination, particularly concerns regarding the decline in the pass rate. In addition to normal variation from window to window, and error variance owing to moderately low sample sizes, the changes in pass rate are in part the result of changes in relative proportions of candidates eligible based on PGY-2 and practice experience pathways. While more sophisticated statistical analysis would be required to evaluate the effect of eligibility makeup on each testing window's pass rate, it does appear from the trends that there is a relationship, and these observations are consistent with previously published analysis evaluating performance on BPS specialty examinations across eligibility groups.³ Additional analysis of the domain-level scores revealed patterns of performance among eligibility groups similar to that seen on the overall examination. The PGY-2 candidates outperform PGY-1 candidates, who in turn outperform candidates who qualify by practice experience. This finding should not be surprising, as the domains constitute positively correlated parts of the whole. The results of the ANOVA show that the differences observed across groups are statistically significant with a few exceptions: on Domain 2, PGY-1 candidates were indistinct from PGY-2 candidates; and on Domain 3, PGY-1 candidates were indistinct from both practice experience and PGY-2 candidates. What is perhaps more interesting in the domain-level scores is that the largest group effects were seen on scores for Domains 1 and 2 (large) followed by Domain 3 (small). The effect sizes are not a function of the scales or the ranges of scores (ie, the differing number of questions in each domain), as the effects persist when standardizing the scores to a z score (mean = 0, SD = 1). This result suggests that while performance across eligibility groups on Domain 3 is similar, Domains 1 and 2 distinguish candidate ability more effectively. Qualitatively speaking, Domain 1 (Person-Centered Care) tends to reflect the core clinical duties and therapeutics that distinguish the psychiatric pharmacist both from general pharmacists and from other specialties. Domain 3 (Healthcare Policy, Advocacy, and Practice Management) tends to represent duties and knowledge areas that are less patient-focused and are more generalizable and shared by other pharmacy specialties.

The most significant difference in performance across the eligibility groups was noted in Domain 1, which constitutes 55% of the examination questions. We speculate that there is a difference in the type, quality, and quantity of psychiatric pharmacy direct patient care experiences that occur between PGY-2 candidates, PGY-1 candidates, and practice experience candidates without postgraduate residency training. This experience likely reinforces the value and necessity of experiential learning that involves direct patient care, including both inpatient and outpatient psychiatric treatment modalities. Candidates with practice experience only may have clinical experiences that are limited to only 1 practice setting, and exposure to a wider variety of psychiatric disorders may also be limited, depending on their practice environment. This illuminates the importance of direct psychiatric pharmacy mentorship and teaching that a residency program director and/or clinical preceptors provide during a PGY-2 experience that may not occur for practice experience candidates. Further assessment of the value of mentorship and clinical teaching and its impact on building clinical knowledge and developing proficiency within psychiatric pharmacy is needed.

The eligibility-driven differences observed here indicate other closely related factors that could influence performance on the examination and warrant further investigation. Little is currently known about variation of certification success rates across other factors of interest. One is locality. For many psychiatric pharmacists, the balance between person-centered care and translation of evidence into practice varies based on scope of practice in different locales. As an illustration of this point, in some parts of the country (eg, portions of the Midwest and West), pharmacists working in psychiatric settings could be credentialed providers and document interventions in the patient's medical record, while in other parts of the country (eg, some areas of the Northeast, some Mid-Atlantic regional sites), pharmacists have more of a dispensing/order verification and curbside consult role in the health system. The variation in scopes of practice may be even more pronounced when accounting for international scopes of practice. This indicates the importance of obtaining more granular information about the qualifying psychiatric pharmacy practice experience. Investigating examination performance across region, role (research, management/administration, and direct patient care), practice setting (academic medical center versus private health system) would further elucidate this question and inform candidates on how these differences may factor into performance on this BCPP Examination. The differences observed across eligibility groups naturally leads to a consideration of the philosophy undergirding the eligibility requirements. The rationale for allowing board eligibility following completion of a PGY-2 in psychiatric pharmacy includes the following:

The experiential component is required to help assure practical application of components of the specialty knowledge being certified. There are multiple pathways to meet the practice experience requirement. The faster eligibility pathways recognize accredited residencies through the American Society of Health System Pharmacists (ASHP). The ASHP residency accreditation program identifies and grants public recognition to practice sites having pharmacy residency training programs that have been evaluated and found to meet the qualifications the ASHP's residency accreditations standards. Thus, accreditation of a pharmacy residency program provides a means of assurance to residency applicants that a program meets certain basic requirements and is, therefore, an acceptable site for postgraduate training in pharmacy practice in organized health care.⁶

The BPS recognizes that the pharmacy profession continues to grapple with the mismatch between the supply and demand of residency positions.⁷ Multiple eligibility pathways are available providing access for candidates interested in earning the BCPP credential, especially for those unable to secure a residency position. As part of the BPS ongoing quality assurance process, the board routinely evaluates examination results and reviews eligibility pathways. While 3 eligibility pathways exist, there are few data or analyses (other than examination results) to further elucidate the extent to which each of the 3 pathways are comparable or distinct. While examination results are convenient and reliable, other measures are needed to better understand the qualitative differences between the eligibility cohorts.

Other health disciplines, particularly medicine, lay out the process to achieve specialty certifications more deterministically than pharmacy. The American Board of Psychiatry and Neurology, for instance, defines the precertification training component strictly based on completion of an accredited specialty program (residency and/or fellowship).⁸ Completing an accredited specialty residency program as a component of eligibility is characteristic of physician certification in the medical specialties. In contrast to medical specialties, the BPS provision of non-residency eligibility routes creates a more heterogenous candidate population. That fact provides important context to the results documented in this study; namely, that heterogeneity in the candidate pool gives rise to the disparate performance among eligibility groups witnessed in this and other comparative studies.

Finally, we note that these findings have relevance to the future design of the Psychiatric Pharmacy Examination program, which is due to undergo a renewal of the Role Delineation in 2021, as well as a passing standard study in 2022.