- Home
- Kinesiology/Exercise and Sport Science
- Health Care in Exercise and Sport
- Coaching and Officiating
- Athletic Training and Therapy
- Evidence-Based Practice in Athletic Training
As one of the first texts of its kind, Evidence-Based Practice in Athletic Training contains essential information on the fundamentals of evidence-based practice (EBP) for students who are working toward certification in athletic training and athletic trainers who wish to stay up to date on best practices in the field. With EBP, all clinical decisions are based on available research studies, and these studies are selected and assessed according to specific criteria that yield evidence of benefit. EBP is a continuing education requirement for athletic trainers who are certified with the Board of Certification (BOC).
Grounded in solid science, Evidence-Based Practice in Athletic Training explains the basics of EBP and the research design methods that are so vital to its implementation. Starting in part I, the text introduces the various levels of evidence, well-built question development using the PICO technique (patient problem or population, intervention, comparison, and outcomes), the five steps of searching for evidence, and search techniques. Part II guides readers through researching specific questions and evaluating research studies, including how to incorporate the evidence they find into their clinical practice. Part III reviews the various research types, their uses and benefits, and research ethics as a critical part of the process of EBP. Through these step-by-step chapters, readers will be able to formulate clinical questions, perform research on current studies, analyze the available data, and apply the principles in their practice in order to provide the best and most accurate care possible.
In addition to in-depth information on the principles and application of EBP, Evidence-Based Practice in Athletic Training presents clinically based scenarios that allow students to apply their recently acquired knowledge to real-life situations, thus encouraging a deeper understanding of the topics presented throughout the text. These scenarios allow those who are learning EBP concepts for the first time to understand how EBP is incorporated clinically.
The most efficient, systematic, and thorough resource of its kind, Evidence-Based Practice in Athletic Training encourages students and current certified athletic trainers to ask meaningful questions, gain the knowledge they need for excelling in future practice, and rise to the top of their profession. For students who want a thorough skill base in EBP and for credentialed health care professionals who seek further knowledge in the area, Evidence-Based Practice in Athletic Training will help all current and future athletic trainers provide the best care for their athletes and clients.
Part I. Introduction
Chapter 1. Evidence-Based Practice Models
Definition of Evidence-Based Practice
Five Steps of Evidence-Based Practice
Levels of Evidence
Purpose of Evidence-Based Practice in Clinical Athletic Training
Summary
References
Chapter 2. Steps in Evidence-Based Practice Research
Step 1: Create a Clinically Relevant and Searchable Question Concerning Your Athlete
Step 2: Conduct a Database Search to Find the Best Evidence
Step 3: Critically Appraise the Articles or Evidence for Quality
Step 4: Critically Synthesize the Evidence to Choose and Implement a Treatment
Step 5: Assess the Outcomes by Monitoring the Athlete
Search Troubleshooting Tips
Summary
References
Chapter 3. Research Evaluation
Reliability
Validity
Prevalence Independent Test
Calculating Sensitivity and Specificity
Summary
References
Part II. Critical Appraisal of Evidence-Based Practice
Chapter 4. Diagnostic Research
Measurement
Methods
Summary
References
Chapter 5. Prognostic Research
Prognostic Designs
Prognostic Statistics
Appraisal Questions
Summary
References
Chapter 6. Systematic Reviews and Meta-Analyses
Review of the Evidence Pyramid
Definitions of and Differences Among Filtered Reviews
Strengths and Processes of Critically Appraised Topics
Strengths and Processes of Systematic Reviews
Strengths and Processes of Meta-Analyses
How to Find Critically Appraised Topics, Systematic Reviews, and Meta-Analyses
Practice EBP Searches
Summary
References
Chapter 7. EBP in Daily Clinical Practice
Your Practice Based on Evidence
Your Practice Based on Your Clinical Experience
Your Practice Based on Athlete Preferences and Values
Clinical Practice Guidelines
Practice Altruism
Summary
References
Chapter 8. Review Outcome Measures
Types of Outcome Measures
Psychometric Properties of Outcome Measures
Communicating Outcomes
Assessing Your Self-Practice
Adjusting Your Practice
Summary
References
Part III. Research Statistics and Design
Chapter 9. Quantitative Research
Types of Quantitative Data
Types of Quantitative Research Designs
Descriptive Statistics
Common Inferential Statistics
Summary
References
Chapter 10. Qualitative Studies
Types of Qualitative Research Studies
Gathering Data
Analyzing Data
Assessing Trustworthiness
Critical Appraisal of Qualitative Research Studies
Qualitative Research and Athletic Training
Summary
References
Chapter 11. Research Ethics
Ethical Roadblocks
Trustworthiness and Conflicts
Summary
References
Scot Raab, PhD, is an assistant professor in the athletic training education program at Northern Arizona University in Flagstaff. Raab earned his PhD in teaching and administration and has more than 20 years of experience in clinical practice, higher education instruction, and methodological review to contribute to EBP education. He teaches several research courses and mentors undergraduate and graduate students in research projects.
Debbie Craig, PhD, is the director of the athletic training education program in the department of physical therapy and athletic training and a professor at Northern Arizona University in Flagstaff. With more than a decade of clinical practice in athletic training and a PhD in educational leadership, Craig is an authority in EBP and research protocols. She teaches EBP to graduate students and is a member of the National Athletic Trainers’ Association.
"This is a much-needed book in the field of athletic training, as it helps both current and future athletic trainers focus on the importance of evidence-based practice and research. Furthermore, the emphasis on research helps support the validity and reliability of our practice guidelines and profession as a whole."
--Doody’s Book Review (5 Star Review)
What is Evidence-Based Practice Anyway?
Evidence-based practice (EBP) is a systematic method of reviewing the best evidence, combining it with the art of athletic training or your clinical expertise, and making informed choices to treat the athlete.
Definition of Evidence-Based Practice
Evidence-based practice (EBP) is a systematic method of reviewing the best evidence, combining it with the art of athletic training or your clinical expertise, and making informed choices to treat the athlete. Integrating EBP requires that you incorporate your clinical expertise, your athletes' values, and the best available clinical research into your practice. At its core, EBP is about focusing on athlete and clinical outcomes.
Clinical expertise is the culmination of your experience treating and providing care to athletes. It includes your personal values, preferences, experiences, and wisdom. Clinical expertise develops from hours of observation and trial and error and can result in a sense of pride and emotion. An AT who is vested in caring for an athlete can be uncomfortable being wrong about an assessment or a rehab and treatment program. Likewise, facing a challenging clinical case, deciding on a course of action, and having that action confirmed by other health care providers can be very rewarding. This is the reason many ATs are wary of incorporating new techniques that are unfamiliar to them. However, such a guarded approach can keep them from remembering that optimal athlete outcomes are the most important goal.
Patient values are the preconceived ideas and beliefs each athlete brings to the clinical setting. Athletes all have their own distinctive concerns that you must address. Often, the athlete arrives with a set feeling of trust in or distrust of ATs that has been cultivated from prior experience or social discussions. Furthermore, the athlete's values can be skewed by the appearance or functional use of the athletic training room and the amount of traffic. Some arrive lacking a clear understanding of the body and the healing process and may have set unrealistic time lines for their return to participation. This will influence their adherence to, or desire to complete, therapy. Treatment can be further confounded by comorbidities and the athlete's support network. Although the challenge is unique for each athlete, as an EBP clinician, you must consider all athletes' perspectives.
Clinical research is a scientific and systematic process that generates evidence through hypothesis testing and sound methodology. This is perhaps the most challenging portion of the EBP concept to incorporate. You may have limited experience analyzing peer-reviewed literature, and learning this process may thus appear daunting. Published studies should at a bare minimum introduce a research question, describe the methods used to assess subjects, outline the variables of interest, and explain the participant inclusion and exclusion criteria. The statistical analysis should align with the methods, and the authors should report only facts in the results section. The discussion or impression of the outcomes belongs in the conclusion or discussion section of an article. When assessing the quality of a clinical research article, ask the following questions:
- Were eligibility criteria specified?
- Were subjects randomly assigned to groups, and was assignment concealed, if appropriate?
- Were subjects similar at baseline?
- As appropriate, were subjects, clinicians, and assessors blinded to the treatment of participants?
- Was the completion rate of participants 85% or higher?
Eligibility criteria are a set of delimiters that detail who can be included in a study. These might include age limits, an association with a specific sport, a particular injury or medical condition, ethnicity, or sex. You need to know whether the study will be relevant to the athlete in your care.
Randomization is important in research because it helps to determine whether something truly made an impact, or whether it might have occurred by chance alone or as a result of a preconceived bias. An unethical researcher might place participants in a study group according to a notion that one group will respond more favorably to the treatment, with the purpose of showing support for that treatment. Likewise, whether participants were blinded (not aware that they were placed in a control or experimental group) is an important consideration when reviewing a study. To help the researcher or support a product or treatment they favor, participants who are not blinded may alter their efforts on tests or skew their reporting.
The similarity of participants is also crucial when determining a study's relevance. Participants should be similar in such factors as age, sport, and condition. It would be inappropriate to conclude that a new treatment works if those in the experimental group receiving the treatment were vastly different from those in the control group who did not receive a treatment. Furthermore, blinding the assessors to the participants' treatment limits the possibility of assessor bias.
Finally, researchers should report the participant completion rate to reveal the percentage of participants who dropped out of the study. A completion rate of less than 85% raises suspicion. Reviewers should question why so many participants dropped out. Was the study poorly organized? Was it painful? Was it too cumbersome or difficult?
With all of these aspects of research to consider, always keep the three components of EBP (clinical expertise, patient values, and clinical research) in mind. The bottom line is that EBP should always revolve around the athlete.
Five Steps of Evidence-Based Practice
To use evidence-based practice and retain an athlete focus, follow these five steps:
- Create aclinically relevant and searchable question concerning your athlete.
- Conduct a database search to find the best evidence.
- Critically appraise the articles or evidence for quality.
- Critically synthesize the evidence to choose and implement a treatment.
- Assess the outcomes by monitoring the athlete.
Let's take a closer look at each of these steps.
- Step 1: The first step is to start with the athlete, establish the clinical concern, and build a clinical question that centers on solving the issue or treating the condition. Further details on developing a clinical research question are presented in chapter 2. In the meantime, remember that the clinical question should be focused and searchable. A question that is too narrow in scope will return limited results. One that is too wide will return excessive information, limiting your ability to incorporate the content into a treatment plan. For example, if you are curious about the ability of ultrasound to facilitate increased tissue extensibility and are treating an athlete with ITB syndrome, searching for treat ITB with ultrasound may result in few or no results. It is too narrow and specific. If you search for ultrasound treatment, the results will most likely be too wide. A more appropriate approach might be to search for ultrasound AND connective tissue. This would return studies related to ultrasound and its effects on connective tissue. This will not be specific to ITB, but because ITB involves connective tissue, the treatment parameters and outcomes may be similar enough to draw upon for clinical use.
- Step 2: This step pertains to conducting the database search and looking for the best evidence related to the athlete's condition or injury. Chapter 2 explains how to conduct the search (including Boolean modifiers and search engines and databases), format the question, and conduct the first search. With practice, you will become adept at completing this in minimal time.
- Step 3: Appraise the articles for quality and for whether they are applicable to your athlete.1 Chapters 3 through 6 will help you learn how to do this. It is helpful to rate the quality and applicability of the studies you find on these two scales. A study may be of high quality but not applicable to your current athlete. For example, consider the effect of counterirritants. A research project that applied counterirritants to athletes with arthritis and found no improvement in range of motion at the fingers would not apply to an athlete's use of counterirritants to treat muscle spasms of the gastrocnemius. In this example, the study might be well designed but have no relevance to an athlete with a tight gastrocnemius. Conversely, a study of the effect of counterirritants on athletes with muscle spasms might be very applicable. However, if the article fails to report subjects' baseline measures, the inclusion or exclusion criteria, or the type of counterirritant used, this would indicate low quality. Without a certain amount of requisite data presented in the study, it would be impossible to devise a treatment plan.
- Step 4: This step involves critical processing and synthesizing. Integrating the evidence, your own clinical expertise and comfort in performing certain skills, and the values of the athlete will form the framework for your treatment plan.
- Step 5: This step returns to the athlete. You need to evaluate critically the progress and outcomes, reflecting on steps 1 through 4 and continually aiming to improve outcomes.
Learn more about Evidence-Based Practice in Athletic Training.
Is the Test Reliable?
The first term that is important for a consumer of research to understand is reliability. By strict definition, reliability refers to consistency in measures attained by people or instruments.
Reliability
The first term that is important for a consumer of research to understand is reliability. By strict definition, reliability refers to consistency in measures attained by people or instruments.1,2 Basically, does the tool you are using produce the same outcome each time you use it? An easy example is a pair of tape cutters. If every time you use the cutters to remove tape, you achieve a nice, clean cut, you would consider these cutters reliable. However, if sometimes they cut the tape and other times they do not, you would never know what outcome to expect; you would therefore consider these cutters unreliable. Reliability is important when you are trying to make a clinical decision. If you are performing an anterior drawer test on an ankle and one time you perform the test you feel laxity and the next time you fail to feel laxity, you will have difficulty making a decision. You may question whether the athlete changed, whether you applied different degrees of force, or whether the test itself is unreliable.
Reliability also relates to questionnaires. A well-structured item on a questionnaire should be easy to understand and not subject to misinterpretation. Let's look at an example of an item that uses a Likert scale of 1 to 4, with 1 being low or disagree and 4 being high or strongly agree. Here is the statement: Some of the most important tasks of an athletic trainer are administration duties and the bidding process. Although we can all agree that athletic trainers (ATs) often have some administrative duties and likely need to bid on purchases, this is a poor item that would probably be unreliable. One reader might believe that the most important duties of an AT relate to prevention and mark 1 for this item. Another reader might focus on the bidding portion of the statement and, believing that it's not that important, also mark 1. The researcher in this case has a value of 1 on two occasions but for very different reasons. However, an AT with administrative duties might focus on the administrative portion of the statement and rate this a 4. These varied scores based on individual interpretations of the statement would lead to low reliability and prohibit the researcher from drawing a valid conclusion.
Perhaps a better statement would look like this: Athletic trainers should have administrative skills. The scores for this may also vary, but they are less likely to vary because of misinterpretation. The variance in the scores would most likely be due to a variance in the opinions of the respondents. Thus, it is important to know that a study you are reviewing reports reliability values; without them, it's very difficult to trust the findings and make a judgment based on them.
You should be minimally familiar with several types of reliability measures as an evidence-based clinician. Presented here are four primary types: internal reliability, test - retest reliability, interrater reliability, and intrarater reliability.
Internal Reliability
Internal reliability is often an issue with surveys and questionnaires that employ a Likert scale system of values. The researcher wants to know whether the participant is responding to similar questions consistently. That is, did the person answer different questions about the same topic or construct in a similar manner?1 Let's assume that the researcher has a 30-item survey to find out athletes' opinions of the school's coaching staff. On item 4, which states that the school's coaching staff sincerely cares about athletes, an athlete might respond with a 1 (low or disagree). However, item 22 asks whether the athlete agrees that the coaches exhibit a caring nature; the athlete might score this item as a 4 (high or strongly agree). This is an example of two items asking about the same construct and receiving two very different responses, when the researcher would expect these answers to be similar. However, just because one person completes a survey in this fashion does not mean that the survey is unreliable. Only if a large numberof participants answer the survey in this fashion would it be considered low in internal reliability. This indicates that the sample population misunderstood the items, they deliberately answered in an odd fashion, or the items need to be dropped or reworded to increase consistency. This touches on one assumption of survey research, which is that participants answer honestly.
Test - Retest Reliability
The next form of reliability is test - retest reliability. As the name implies, it entails administering the test or assessment on more than one occasion. Test - retest reliability refers to the ability of an instrument or assessment to produce similar outcomes in the same group of people on multiple occasions in an amount of time that would not lead to a change in the measurement of interest.1,2 Let's assume that you use a dynamometer to assess the triceps extension strength in a group of athletes. The athletes are healthy college males between 18 and 24 years of age who are free of injury or disease and not participating in a strength training program. You would not expect their triceps strength to change in a week's time. It is safe to assume that the scores of the first test and those of the second test will be similar. In this case, large changes in strength measures are likely due to unreliable measures being detected by the dynamometer or test administrator error.
Not all test - retest measures are as easy to visualize or straightforward as the strength testing example. If you take a standardized test such as the SAT or GRE twice, a week apart, and you don't study between the tests, your scores will likely be similar. That's because taking a large standardized test covering topics you might not know won't teach you the topics. If a person scores low in algebra on the SAT, merely taking the SAT again won't improve his ability to apply algebra. This is not the case with certain types of cognitive assessment. A standard practice for the assessment of concussion often includes a computerized concussion assessment. These types of tests, however, result in athletes learning how to take the test, subsequently increasing their scores on follow-up trials. This is referred to as a practice effect.
Two of the most common cognitive-based tests in clinical athletic training practice are the ImPACT Concussion Management Software (ImPACT Applications, Pittsburgh, PA) and Concussion Sentinel (CogState Ltd, Victoria, Australia). These tests present with low to moderate test - retest reliability. This means that the scores of participants completing the assessments varied more than what is considered acceptable between tests, which diminishes their clinical relevance. Test - retest reliability is often reported in the literature as a value between 0 and 1, with value of 1 being more reliable or having fewer random effects that are unaccounted for in the results.3 The closer these reported values are to 1, the more reliable the assessments are, or the more trust you can put in them.
Interrater Reliability
Consensus between professionals on how to best treat an injury is important. It does, however, require that similar results or conditions be interpreted the same way. If you are an experienced AT reviewing injury reports upon returning to work after a few days off, you want to trust that other ATs completed certain orthopedic assessments the same way you would. More important, you hope that those tests would render the same outcome regardless of the person performing them. This is interrater reliability - that is, the agreement between clinicians who perform independent assessments of a condition.6 It is important to note that the consensus should be reached by both clinicians, but independently, without cognitive influence from each other. When the second AT is assessing an injury using the same assessment as the first, she should not know the outcome of the first.6 This ensures that the person performing the second assessment is not influenced by the outcome of the first assessment.
Also important when determining interrater reliability is that the athlete not recall the first assessment and respond differently to the second. This could be problematic when the assessment is on an injured body part and resulted in discomfort; the athlete may respond differently to the second assessment out of fear of re-creating the discomfort.
When reviewing articles and trying to determine the value of the results, remember that the higher the values for interrater reliability are, the more consistent the outcomes will be between assessors. Detailed examples are provided in subsequent chapters.
Intrarater Reliability
Intrarater reliability relates to the reproducibility of a clinical measure when performed by the same clinician on more than one occasion on the same athlete.2 The purpose of determining intrarater reliability is to assess whether the clinician is consistently using a reliable method of assessment. If the assessment method is not established as reliable, we would return to test - retest reliability to establish trust in the assessment or instrument (recall the tape cutters). An example would be assessing the range of motion of a knee. The bony landmarks used to measure knee flexion and the mechanics of the goniometer will not change from day 1 to day 2. However, range of motion assessments often vary when repeated by the same clinician. Imagine the difficulty in creating a treatment plan when on day 1 an athlete is assessed with 100° of flexion and on day 2 has only 90° of flexion. Variations in observed outcomes will confound the care provided to athletes.
Learn more about Evidence-Based Practice in Athletic Training.
Quickly and Easily Identify Low-Quality Articles Using CRAAP
Chapter 2 demonstrated how to perform a database search, and chapter 3 introduced ways to understand and evaluate the evidence. This chapter continues the discussion of evaluation methods by addressing how to appraise the diagnostic articles you have found.
Methods
Chapter 1 outlined the following five steps of evidence-based practice:
- Create aclinically relevant and searchable question concerning your athlete.
- Conduct a database search to find the best evidence.
- Critically appraise the articles or evidence for quality.
- Critically synthesize the evidence to choose and implement a treatment.
- Assess the outcomes by monitoring the athlete.
Chapter 2 demonstrated how to perform a database search, and chapter 3 introduced ways to understand and evaluate the evidence. This chapter continues the discussion of evaluation methods by addressing how to appraise the diagnostic articles you have found. Following are four steps to appraising diagnostic research:1
- Determine the applicability to your athlete.
- Assess the quality of the article.
- Review the results.
- Summarize the clinical bottom line.
As you gather your articles using a variety of search methods, you can apply a simple assessment to quickly and easily eliminate low-quality articles. Review any article based on the following five criteria (CRAAP):
- Currency. Is the information in need of being revised? The 2004 National Athletic Trainers' Association (NATA) position statement on the management of sport-related concussion allowed for same-day return to participation if symptoms resolved and the athlete passed a return-to-play test.3 The 2014 position statement update noted that our knowledge about how to treat concussions had significantly changed and ruled guidelines as recent as 10 years old obsolete.
- Relevance. Is the article written at a level appropriate for your needs, and do you understand it? If not, you may fail to reach a valid conclusion about the use of assessments in your situation. Has the information been peer reviewed, or is it an editorial or opinion piece? Editorials often have a bias and lack the scientific rigor required to make the information useful for sound clinical decisions.
- Authority. Is a sponsor involved that has a product to market? If so, read the information with great care. If an article is promoting an aquatics rehab program to aid in the recovery of knee surgery, but it was sponsored by a company that sells therapy pools, you need to suspect some potential bias. These types of conflicts of interest can be questionable.
- Accuracy. After completing your primary review, it is time to dig a little deeper and appraise the article. The Quality Assessment of Diagnostic Accuracy Studies (QUADAS) is a 14-question instrument to assist you in the appraisal (see the sidebar).1,4 The answer choices are yes, no, or uncertain; a greater number of yes answers is preferred.1 There is no set number of acceptable yes answers, but if any uncertain or no answers make you feel uncomfortable using the data to make a clinical decision, you may want to further research the topic and expand the number of articles you review.
- Purpose. Why is the information presented? Is it an advertisement? Is it funded by someone with a financial stake in selling something? If the answer to the preceding questions is yes, be sure to thoroughly review the academic content. The purpose of a quality article is to inform and provide outcomes, allowing readers to apply it as they see fit.
Each of the questions should be clearly addressed or answered in the article. If you are interested in learning whether a rehab program would be appropriate for high school athletes, the studies you review should have been conducted with people in the same age range. This is called generalizability - that is, can the results of a study be applied to another population based on similarities in the populations?
Authors should clearly tell you how they selected participants and why, as well as any reasons participants were excluded from participation. Did the author report issues with the data that could not be explained, and was the number of participants at the start and end of the study reported? If fewer completed the study than initiated it, was there an explanation for the dropout rate? If an author was researching the use of latex bandages to decrease swelling after ankle sprains and a volunteer participant had an allergy to latex, that person would need to leave the study. In this case, an author should report that 40 subjects volunteered to participate and were initially taking part in the study, and one was removed because of an exclusionary criterion. The author should not say that 39 people agreed to participate.
Similarly, if a new test (index test) is being assessed, it should be compared to an appropriate gold standard reference. If a researcher is testing a new axillary thermometer to establish whether it accurately assesses core body temperature, it should be compared to rectal thermometer measures. Comparing a new axillary thermometer to oral thermometers or others that are not scientifically accepted as the gold standard could introduce significant error, which would make the study results inappropriate to use in clinical practice.
Also important when appraising an article is assessing its methods section. Is it detailed enough that you could replicate it, if desired? You should be able to fully understand what was done during the study when reading the methods section.
The prevention of bias is also important during a study. If a participant was assessed using both an indexed and a referenced test, were the results interpreted separately? In other words, did the person completing the assessments know the results of the comparative assessment? A person who is aware of the other outcomes when completing an assessment creates a potential for biased reporting and a loss of objectivity, which is important in research.
Learn more about Evidence-Based Practice in Athletic Training.
What is Evidence-Based Practice Anyway?
Evidence-based practice (EBP) is a systematic method of reviewing the best evidence, combining it with the art of athletic training or your clinical expertise, and making informed choices to treat the athlete.
Definition of Evidence-Based Practice
Evidence-based practice (EBP) is a systematic method of reviewing the best evidence, combining it with the art of athletic training or your clinical expertise, and making informed choices to treat the athlete. Integrating EBP requires that you incorporate your clinical expertise, your athletes' values, and the best available clinical research into your practice. At its core, EBP is about focusing on athlete and clinical outcomes.
Clinical expertise is the culmination of your experience treating and providing care to athletes. It includes your personal values, preferences, experiences, and wisdom. Clinical expertise develops from hours of observation and trial and error and can result in a sense of pride and emotion. An AT who is vested in caring for an athlete can be uncomfortable being wrong about an assessment or a rehab and treatment program. Likewise, facing a challenging clinical case, deciding on a course of action, and having that action confirmed by other health care providers can be very rewarding. This is the reason many ATs are wary of incorporating new techniques that are unfamiliar to them. However, such a guarded approach can keep them from remembering that optimal athlete outcomes are the most important goal.
Patient values are the preconceived ideas and beliefs each athlete brings to the clinical setting. Athletes all have their own distinctive concerns that you must address. Often, the athlete arrives with a set feeling of trust in or distrust of ATs that has been cultivated from prior experience or social discussions. Furthermore, the athlete's values can be skewed by the appearance or functional use of the athletic training room and the amount of traffic. Some arrive lacking a clear understanding of the body and the healing process and may have set unrealistic time lines for their return to participation. This will influence their adherence to, or desire to complete, therapy. Treatment can be further confounded by comorbidities and the athlete's support network. Although the challenge is unique for each athlete, as an EBP clinician, you must consider all athletes' perspectives.
Clinical research is a scientific and systematic process that generates evidence through hypothesis testing and sound methodology. This is perhaps the most challenging portion of the EBP concept to incorporate. You may have limited experience analyzing peer-reviewed literature, and learning this process may thus appear daunting. Published studies should at a bare minimum introduce a research question, describe the methods used to assess subjects, outline the variables of interest, and explain the participant inclusion and exclusion criteria. The statistical analysis should align with the methods, and the authors should report only facts in the results section. The discussion or impression of the outcomes belongs in the conclusion or discussion section of an article. When assessing the quality of a clinical research article, ask the following questions:
- Were eligibility criteria specified?
- Were subjects randomly assigned to groups, and was assignment concealed, if appropriate?
- Were subjects similar at baseline?
- As appropriate, were subjects, clinicians, and assessors blinded to the treatment of participants?
- Was the completion rate of participants 85% or higher?
Eligibility criteria are a set of delimiters that detail who can be included in a study. These might include age limits, an association with a specific sport, a particular injury or medical condition, ethnicity, or sex. You need to know whether the study will be relevant to the athlete in your care.
Randomization is important in research because it helps to determine whether something truly made an impact, or whether it might have occurred by chance alone or as a result of a preconceived bias. An unethical researcher might place participants in a study group according to a notion that one group will respond more favorably to the treatment, with the purpose of showing support for that treatment. Likewise, whether participants were blinded (not aware that they were placed in a control or experimental group) is an important consideration when reviewing a study. To help the researcher or support a product or treatment they favor, participants who are not blinded may alter their efforts on tests or skew their reporting.
The similarity of participants is also crucial when determining a study's relevance. Participants should be similar in such factors as age, sport, and condition. It would be inappropriate to conclude that a new treatment works if those in the experimental group receiving the treatment were vastly different from those in the control group who did not receive a treatment. Furthermore, blinding the assessors to the participants' treatment limits the possibility of assessor bias.
Finally, researchers should report the participant completion rate to reveal the percentage of participants who dropped out of the study. A completion rate of less than 85% raises suspicion. Reviewers should question why so many participants dropped out. Was the study poorly organized? Was it painful? Was it too cumbersome or difficult?
With all of these aspects of research to consider, always keep the three components of EBP (clinical expertise, patient values, and clinical research) in mind. The bottom line is that EBP should always revolve around the athlete.
Five Steps of Evidence-Based Practice
To use evidence-based practice and retain an athlete focus, follow these five steps:
- Create aclinically relevant and searchable question concerning your athlete.
- Conduct a database search to find the best evidence.
- Critically appraise the articles or evidence for quality.
- Critically synthesize the evidence to choose and implement a treatment.
- Assess the outcomes by monitoring the athlete.
Let's take a closer look at each of these steps.
- Step 1: The first step is to start with the athlete, establish the clinical concern, and build a clinical question that centers on solving the issue or treating the condition. Further details on developing a clinical research question are presented in chapter 2. In the meantime, remember that the clinical question should be focused and searchable. A question that is too narrow in scope will return limited results. One that is too wide will return excessive information, limiting your ability to incorporate the content into a treatment plan. For example, if you are curious about the ability of ultrasound to facilitate increased tissue extensibility and are treating an athlete with ITB syndrome, searching for treat ITB with ultrasound may result in few or no results. It is too narrow and specific. If you search for ultrasound treatment, the results will most likely be too wide. A more appropriate approach might be to search for ultrasound AND connective tissue. This would return studies related to ultrasound and its effects on connective tissue. This will not be specific to ITB, but because ITB involves connective tissue, the treatment parameters and outcomes may be similar enough to draw upon for clinical use.
- Step 2: This step pertains to conducting the database search and looking for the best evidence related to the athlete's condition or injury. Chapter 2 explains how to conduct the search (including Boolean modifiers and search engines and databases), format the question, and conduct the first search. With practice, you will become adept at completing this in minimal time.
- Step 3: Appraise the articles for quality and for whether they are applicable to your athlete.1 Chapters 3 through 6 will help you learn how to do this. It is helpful to rate the quality and applicability of the studies you find on these two scales. A study may be of high quality but not applicable to your current athlete. For example, consider the effect of counterirritants. A research project that applied counterirritants to athletes with arthritis and found no improvement in range of motion at the fingers would not apply to an athlete's use of counterirritants to treat muscle spasms of the gastrocnemius. In this example, the study might be well designed but have no relevance to an athlete with a tight gastrocnemius. Conversely, a study of the effect of counterirritants on athletes with muscle spasms might be very applicable. However, if the article fails to report subjects' baseline measures, the inclusion or exclusion criteria, or the type of counterirritant used, this would indicate low quality. Without a certain amount of requisite data presented in the study, it would be impossible to devise a treatment plan.
- Step 4: This step involves critical processing and synthesizing. Integrating the evidence, your own clinical expertise and comfort in performing certain skills, and the values of the athlete will form the framework for your treatment plan.
- Step 5: This step returns to the athlete. You need to evaluate critically the progress and outcomes, reflecting on steps 1 through 4 and continually aiming to improve outcomes.
Learn more about Evidence-Based Practice in Athletic Training.
Is the Test Reliable?
The first term that is important for a consumer of research to understand is reliability. By strict definition, reliability refers to consistency in measures attained by people or instruments.
Reliability
The first term that is important for a consumer of research to understand is reliability. By strict definition, reliability refers to consistency in measures attained by people or instruments.1,2 Basically, does the tool you are using produce the same outcome each time you use it? An easy example is a pair of tape cutters. If every time you use the cutters to remove tape, you achieve a nice, clean cut, you would consider these cutters reliable. However, if sometimes they cut the tape and other times they do not, you would never know what outcome to expect; you would therefore consider these cutters unreliable. Reliability is important when you are trying to make a clinical decision. If you are performing an anterior drawer test on an ankle and one time you perform the test you feel laxity and the next time you fail to feel laxity, you will have difficulty making a decision. You may question whether the athlete changed, whether you applied different degrees of force, or whether the test itself is unreliable.
Reliability also relates to questionnaires. A well-structured item on a questionnaire should be easy to understand and not subject to misinterpretation. Let's look at an example of an item that uses a Likert scale of 1 to 4, with 1 being low or disagree and 4 being high or strongly agree. Here is the statement: Some of the most important tasks of an athletic trainer are administration duties and the bidding process. Although we can all agree that athletic trainers (ATs) often have some administrative duties and likely need to bid on purchases, this is a poor item that would probably be unreliable. One reader might believe that the most important duties of an AT relate to prevention and mark 1 for this item. Another reader might focus on the bidding portion of the statement and, believing that it's not that important, also mark 1. The researcher in this case has a value of 1 on two occasions but for very different reasons. However, an AT with administrative duties might focus on the administrative portion of the statement and rate this a 4. These varied scores based on individual interpretations of the statement would lead to low reliability and prohibit the researcher from drawing a valid conclusion.
Perhaps a better statement would look like this: Athletic trainers should have administrative skills. The scores for this may also vary, but they are less likely to vary because of misinterpretation. The variance in the scores would most likely be due to a variance in the opinions of the respondents. Thus, it is important to know that a study you are reviewing reports reliability values; without them, it's very difficult to trust the findings and make a judgment based on them.
You should be minimally familiar with several types of reliability measures as an evidence-based clinician. Presented here are four primary types: internal reliability, test - retest reliability, interrater reliability, and intrarater reliability.
Internal Reliability
Internal reliability is often an issue with surveys and questionnaires that employ a Likert scale system of values. The researcher wants to know whether the participant is responding to similar questions consistently. That is, did the person answer different questions about the same topic or construct in a similar manner?1 Let's assume that the researcher has a 30-item survey to find out athletes' opinions of the school's coaching staff. On item 4, which states that the school's coaching staff sincerely cares about athletes, an athlete might respond with a 1 (low or disagree). However, item 22 asks whether the athlete agrees that the coaches exhibit a caring nature; the athlete might score this item as a 4 (high or strongly agree). This is an example of two items asking about the same construct and receiving two very different responses, when the researcher would expect these answers to be similar. However, just because one person completes a survey in this fashion does not mean that the survey is unreliable. Only if a large numberof participants answer the survey in this fashion would it be considered low in internal reliability. This indicates that the sample population misunderstood the items, they deliberately answered in an odd fashion, or the items need to be dropped or reworded to increase consistency. This touches on one assumption of survey research, which is that participants answer honestly.
Test - Retest Reliability
The next form of reliability is test - retest reliability. As the name implies, it entails administering the test or assessment on more than one occasion. Test - retest reliability refers to the ability of an instrument or assessment to produce similar outcomes in the same group of people on multiple occasions in an amount of time that would not lead to a change in the measurement of interest.1,2 Let's assume that you use a dynamometer to assess the triceps extension strength in a group of athletes. The athletes are healthy college males between 18 and 24 years of age who are free of injury or disease and not participating in a strength training program. You would not expect their triceps strength to change in a week's time. It is safe to assume that the scores of the first test and those of the second test will be similar. In this case, large changes in strength measures are likely due to unreliable measures being detected by the dynamometer or test administrator error.
Not all test - retest measures are as easy to visualize or straightforward as the strength testing example. If you take a standardized test such as the SAT or GRE twice, a week apart, and you don't study between the tests, your scores will likely be similar. That's because taking a large standardized test covering topics you might not know won't teach you the topics. If a person scores low in algebra on the SAT, merely taking the SAT again won't improve his ability to apply algebra. This is not the case with certain types of cognitive assessment. A standard practice for the assessment of concussion often includes a computerized concussion assessment. These types of tests, however, result in athletes learning how to take the test, subsequently increasing their scores on follow-up trials. This is referred to as a practice effect.
Two of the most common cognitive-based tests in clinical athletic training practice are the ImPACT Concussion Management Software (ImPACT Applications, Pittsburgh, PA) and Concussion Sentinel (CogState Ltd, Victoria, Australia). These tests present with low to moderate test - retest reliability. This means that the scores of participants completing the assessments varied more than what is considered acceptable between tests, which diminishes their clinical relevance. Test - retest reliability is often reported in the literature as a value between 0 and 1, with value of 1 being more reliable or having fewer random effects that are unaccounted for in the results.3 The closer these reported values are to 1, the more reliable the assessments are, or the more trust you can put in them.
Interrater Reliability
Consensus between professionals on how to best treat an injury is important. It does, however, require that similar results or conditions be interpreted the same way. If you are an experienced AT reviewing injury reports upon returning to work after a few days off, you want to trust that other ATs completed certain orthopedic assessments the same way you would. More important, you hope that those tests would render the same outcome regardless of the person performing them. This is interrater reliability - that is, the agreement between clinicians who perform independent assessments of a condition.6 It is important to note that the consensus should be reached by both clinicians, but independently, without cognitive influence from each other. When the second AT is assessing an injury using the same assessment as the first, she should not know the outcome of the first.6 This ensures that the person performing the second assessment is not influenced by the outcome of the first assessment.
Also important when determining interrater reliability is that the athlete not recall the first assessment and respond differently to the second. This could be problematic when the assessment is on an injured body part and resulted in discomfort; the athlete may respond differently to the second assessment out of fear of re-creating the discomfort.
When reviewing articles and trying to determine the value of the results, remember that the higher the values for interrater reliability are, the more consistent the outcomes will be between assessors. Detailed examples are provided in subsequent chapters.
Intrarater Reliability
Intrarater reliability relates to the reproducibility of a clinical measure when performed by the same clinician on more than one occasion on the same athlete.2 The purpose of determining intrarater reliability is to assess whether the clinician is consistently using a reliable method of assessment. If the assessment method is not established as reliable, we would return to test - retest reliability to establish trust in the assessment or instrument (recall the tape cutters). An example would be assessing the range of motion of a knee. The bony landmarks used to measure knee flexion and the mechanics of the goniometer will not change from day 1 to day 2. However, range of motion assessments often vary when repeated by the same clinician. Imagine the difficulty in creating a treatment plan when on day 1 an athlete is assessed with 100° of flexion and on day 2 has only 90° of flexion. Variations in observed outcomes will confound the care provided to athletes.
Learn more about Evidence-Based Practice in Athletic Training.
Quickly and Easily Identify Low-Quality Articles Using CRAAP
Chapter 2 demonstrated how to perform a database search, and chapter 3 introduced ways to understand and evaluate the evidence. This chapter continues the discussion of evaluation methods by addressing how to appraise the diagnostic articles you have found.
Methods
Chapter 1 outlined the following five steps of evidence-based practice:
- Create aclinically relevant and searchable question concerning your athlete.
- Conduct a database search to find the best evidence.
- Critically appraise the articles or evidence for quality.
- Critically synthesize the evidence to choose and implement a treatment.
- Assess the outcomes by monitoring the athlete.
Chapter 2 demonstrated how to perform a database search, and chapter 3 introduced ways to understand and evaluate the evidence. This chapter continues the discussion of evaluation methods by addressing how to appraise the diagnostic articles you have found. Following are four steps to appraising diagnostic research:1
- Determine the applicability to your athlete.
- Assess the quality of the article.
- Review the results.
- Summarize the clinical bottom line.
As you gather your articles using a variety of search methods, you can apply a simple assessment to quickly and easily eliminate low-quality articles. Review any article based on the following five criteria (CRAAP):
- Currency. Is the information in need of being revised? The 2004 National Athletic Trainers' Association (NATA) position statement on the management of sport-related concussion allowed for same-day return to participation if symptoms resolved and the athlete passed a return-to-play test.3 The 2014 position statement update noted that our knowledge about how to treat concussions had significantly changed and ruled guidelines as recent as 10 years old obsolete.
- Relevance. Is the article written at a level appropriate for your needs, and do you understand it? If not, you may fail to reach a valid conclusion about the use of assessments in your situation. Has the information been peer reviewed, or is it an editorial or opinion piece? Editorials often have a bias and lack the scientific rigor required to make the information useful for sound clinical decisions.
- Authority. Is a sponsor involved that has a product to market? If so, read the information with great care. If an article is promoting an aquatics rehab program to aid in the recovery of knee surgery, but it was sponsored by a company that sells therapy pools, you need to suspect some potential bias. These types of conflicts of interest can be questionable.
- Accuracy. After completing your primary review, it is time to dig a little deeper and appraise the article. The Quality Assessment of Diagnostic Accuracy Studies (QUADAS) is a 14-question instrument to assist you in the appraisal (see the sidebar).1,4 The answer choices are yes, no, or uncertain; a greater number of yes answers is preferred.1 There is no set number of acceptable yes answers, but if any uncertain or no answers make you feel uncomfortable using the data to make a clinical decision, you may want to further research the topic and expand the number of articles you review.
- Purpose. Why is the information presented? Is it an advertisement? Is it funded by someone with a financial stake in selling something? If the answer to the preceding questions is yes, be sure to thoroughly review the academic content. The purpose of a quality article is to inform and provide outcomes, allowing readers to apply it as they see fit.
Each of the questions should be clearly addressed or answered in the article. If you are interested in learning whether a rehab program would be appropriate for high school athletes, the studies you review should have been conducted with people in the same age range. This is called generalizability - that is, can the results of a study be applied to another population based on similarities in the populations?
Authors should clearly tell you how they selected participants and why, as well as any reasons participants were excluded from participation. Did the author report issues with the data that could not be explained, and was the number of participants at the start and end of the study reported? If fewer completed the study than initiated it, was there an explanation for the dropout rate? If an author was researching the use of latex bandages to decrease swelling after ankle sprains and a volunteer participant had an allergy to latex, that person would need to leave the study. In this case, an author should report that 40 subjects volunteered to participate and were initially taking part in the study, and one was removed because of an exclusionary criterion. The author should not say that 39 people agreed to participate.
Similarly, if a new test (index test) is being assessed, it should be compared to an appropriate gold standard reference. If a researcher is testing a new axillary thermometer to establish whether it accurately assesses core body temperature, it should be compared to rectal thermometer measures. Comparing a new axillary thermometer to oral thermometers or others that are not scientifically accepted as the gold standard could introduce significant error, which would make the study results inappropriate to use in clinical practice.
Also important when appraising an article is assessing its methods section. Is it detailed enough that you could replicate it, if desired? You should be able to fully understand what was done during the study when reading the methods section.
The prevention of bias is also important during a study. If a participant was assessed using both an indexed and a referenced test, were the results interpreted separately? In other words, did the person completing the assessments know the results of the comparative assessment? A person who is aware of the other outcomes when completing an assessment creates a potential for biased reporting and a loss of objectivity, which is important in research.
Learn more about Evidence-Based Practice in Athletic Training.
What is Evidence-Based Practice Anyway?
Evidence-based practice (EBP) is a systematic method of reviewing the best evidence, combining it with the art of athletic training or your clinical expertise, and making informed choices to treat the athlete.
Definition of Evidence-Based Practice
Evidence-based practice (EBP) is a systematic method of reviewing the best evidence, combining it with the art of athletic training or your clinical expertise, and making informed choices to treat the athlete. Integrating EBP requires that you incorporate your clinical expertise, your athletes' values, and the best available clinical research into your practice. At its core, EBP is about focusing on athlete and clinical outcomes.
Clinical expertise is the culmination of your experience treating and providing care to athletes. It includes your personal values, preferences, experiences, and wisdom. Clinical expertise develops from hours of observation and trial and error and can result in a sense of pride and emotion. An AT who is vested in caring for an athlete can be uncomfortable being wrong about an assessment or a rehab and treatment program. Likewise, facing a challenging clinical case, deciding on a course of action, and having that action confirmed by other health care providers can be very rewarding. This is the reason many ATs are wary of incorporating new techniques that are unfamiliar to them. However, such a guarded approach can keep them from remembering that optimal athlete outcomes are the most important goal.
Patient values are the preconceived ideas and beliefs each athlete brings to the clinical setting. Athletes all have their own distinctive concerns that you must address. Often, the athlete arrives with a set feeling of trust in or distrust of ATs that has been cultivated from prior experience or social discussions. Furthermore, the athlete's values can be skewed by the appearance or functional use of the athletic training room and the amount of traffic. Some arrive lacking a clear understanding of the body and the healing process and may have set unrealistic time lines for their return to participation. This will influence their adherence to, or desire to complete, therapy. Treatment can be further confounded by comorbidities and the athlete's support network. Although the challenge is unique for each athlete, as an EBP clinician, you must consider all athletes' perspectives.
Clinical research is a scientific and systematic process that generates evidence through hypothesis testing and sound methodology. This is perhaps the most challenging portion of the EBP concept to incorporate. You may have limited experience analyzing peer-reviewed literature, and learning this process may thus appear daunting. Published studies should at a bare minimum introduce a research question, describe the methods used to assess subjects, outline the variables of interest, and explain the participant inclusion and exclusion criteria. The statistical analysis should align with the methods, and the authors should report only facts in the results section. The discussion or impression of the outcomes belongs in the conclusion or discussion section of an article. When assessing the quality of a clinical research article, ask the following questions:
- Were eligibility criteria specified?
- Were subjects randomly assigned to groups, and was assignment concealed, if appropriate?
- Were subjects similar at baseline?
- As appropriate, were subjects, clinicians, and assessors blinded to the treatment of participants?
- Was the completion rate of participants 85% or higher?
Eligibility criteria are a set of delimiters that detail who can be included in a study. These might include age limits, an association with a specific sport, a particular injury or medical condition, ethnicity, or sex. You need to know whether the study will be relevant to the athlete in your care.
Randomization is important in research because it helps to determine whether something truly made an impact, or whether it might have occurred by chance alone or as a result of a preconceived bias. An unethical researcher might place participants in a study group according to a notion that one group will respond more favorably to the treatment, with the purpose of showing support for that treatment. Likewise, whether participants were blinded (not aware that they were placed in a control or experimental group) is an important consideration when reviewing a study. To help the researcher or support a product or treatment they favor, participants who are not blinded may alter their efforts on tests or skew their reporting.
The similarity of participants is also crucial when determining a study's relevance. Participants should be similar in such factors as age, sport, and condition. It would be inappropriate to conclude that a new treatment works if those in the experimental group receiving the treatment were vastly different from those in the control group who did not receive a treatment. Furthermore, blinding the assessors to the participants' treatment limits the possibility of assessor bias.
Finally, researchers should report the participant completion rate to reveal the percentage of participants who dropped out of the study. A completion rate of less than 85% raises suspicion. Reviewers should question why so many participants dropped out. Was the study poorly organized? Was it painful? Was it too cumbersome or difficult?
With all of these aspects of research to consider, always keep the three components of EBP (clinical expertise, patient values, and clinical research) in mind. The bottom line is that EBP should always revolve around the athlete.
Five Steps of Evidence-Based Practice
To use evidence-based practice and retain an athlete focus, follow these five steps:
- Create aclinically relevant and searchable question concerning your athlete.
- Conduct a database search to find the best evidence.
- Critically appraise the articles or evidence for quality.
- Critically synthesize the evidence to choose and implement a treatment.
- Assess the outcomes by monitoring the athlete.
Let's take a closer look at each of these steps.
- Step 1: The first step is to start with the athlete, establish the clinical concern, and build a clinical question that centers on solving the issue or treating the condition. Further details on developing a clinical research question are presented in chapter 2. In the meantime, remember that the clinical question should be focused and searchable. A question that is too narrow in scope will return limited results. One that is too wide will return excessive information, limiting your ability to incorporate the content into a treatment plan. For example, if you are curious about the ability of ultrasound to facilitate increased tissue extensibility and are treating an athlete with ITB syndrome, searching for treat ITB with ultrasound may result in few or no results. It is too narrow and specific. If you search for ultrasound treatment, the results will most likely be too wide. A more appropriate approach might be to search for ultrasound AND connective tissue. This would return studies related to ultrasound and its effects on connective tissue. This will not be specific to ITB, but because ITB involves connective tissue, the treatment parameters and outcomes may be similar enough to draw upon for clinical use.
- Step 2: This step pertains to conducting the database search and looking for the best evidence related to the athlete's condition or injury. Chapter 2 explains how to conduct the search (including Boolean modifiers and search engines and databases), format the question, and conduct the first search. With practice, you will become adept at completing this in minimal time.
- Step 3: Appraise the articles for quality and for whether they are applicable to your athlete.1 Chapters 3 through 6 will help you learn how to do this. It is helpful to rate the quality and applicability of the studies you find on these two scales. A study may be of high quality but not applicable to your current athlete. For example, consider the effect of counterirritants. A research project that applied counterirritants to athletes with arthritis and found no improvement in range of motion at the fingers would not apply to an athlete's use of counterirritants to treat muscle spasms of the gastrocnemius. In this example, the study might be well designed but have no relevance to an athlete with a tight gastrocnemius. Conversely, a study of the effect of counterirritants on athletes with muscle spasms might be very applicable. However, if the article fails to report subjects' baseline measures, the inclusion or exclusion criteria, or the type of counterirritant used, this would indicate low quality. Without a certain amount of requisite data presented in the study, it would be impossible to devise a treatment plan.
- Step 4: This step involves critical processing and synthesizing. Integrating the evidence, your own clinical expertise and comfort in performing certain skills, and the values of the athlete will form the framework for your treatment plan.
- Step 5: This step returns to the athlete. You need to evaluate critically the progress and outcomes, reflecting on steps 1 through 4 and continually aiming to improve outcomes.
Learn more about Evidence-Based Practice in Athletic Training.
Is the Test Reliable?
The first term that is important for a consumer of research to understand is reliability. By strict definition, reliability refers to consistency in measures attained by people or instruments.
Reliability
The first term that is important for a consumer of research to understand is reliability. By strict definition, reliability refers to consistency in measures attained by people or instruments.1,2 Basically, does the tool you are using produce the same outcome each time you use it? An easy example is a pair of tape cutters. If every time you use the cutters to remove tape, you achieve a nice, clean cut, you would consider these cutters reliable. However, if sometimes they cut the tape and other times they do not, you would never know what outcome to expect; you would therefore consider these cutters unreliable. Reliability is important when you are trying to make a clinical decision. If you are performing an anterior drawer test on an ankle and one time you perform the test you feel laxity and the next time you fail to feel laxity, you will have difficulty making a decision. You may question whether the athlete changed, whether you applied different degrees of force, or whether the test itself is unreliable.
Reliability also relates to questionnaires. A well-structured item on a questionnaire should be easy to understand and not subject to misinterpretation. Let's look at an example of an item that uses a Likert scale of 1 to 4, with 1 being low or disagree and 4 being high or strongly agree. Here is the statement: Some of the most important tasks of an athletic trainer are administration duties and the bidding process. Although we can all agree that athletic trainers (ATs) often have some administrative duties and likely need to bid on purchases, this is a poor item that would probably be unreliable. One reader might believe that the most important duties of an AT relate to prevention and mark 1 for this item. Another reader might focus on the bidding portion of the statement and, believing that it's not that important, also mark 1. The researcher in this case has a value of 1 on two occasions but for very different reasons. However, an AT with administrative duties might focus on the administrative portion of the statement and rate this a 4. These varied scores based on individual interpretations of the statement would lead to low reliability and prohibit the researcher from drawing a valid conclusion.
Perhaps a better statement would look like this: Athletic trainers should have administrative skills. The scores for this may also vary, but they are less likely to vary because of misinterpretation. The variance in the scores would most likely be due to a variance in the opinions of the respondents. Thus, it is important to know that a study you are reviewing reports reliability values; without them, it's very difficult to trust the findings and make a judgment based on them.
You should be minimally familiar with several types of reliability measures as an evidence-based clinician. Presented here are four primary types: internal reliability, test - retest reliability, interrater reliability, and intrarater reliability.
Internal Reliability
Internal reliability is often an issue with surveys and questionnaires that employ a Likert scale system of values. The researcher wants to know whether the participant is responding to similar questions consistently. That is, did the person answer different questions about the same topic or construct in a similar manner?1 Let's assume that the researcher has a 30-item survey to find out athletes' opinions of the school's coaching staff. On item 4, which states that the school's coaching staff sincerely cares about athletes, an athlete might respond with a 1 (low or disagree). However, item 22 asks whether the athlete agrees that the coaches exhibit a caring nature; the athlete might score this item as a 4 (high or strongly agree). This is an example of two items asking about the same construct and receiving two very different responses, when the researcher would expect these answers to be similar. However, just because one person completes a survey in this fashion does not mean that the survey is unreliable. Only if a large numberof participants answer the survey in this fashion would it be considered low in internal reliability. This indicates that the sample population misunderstood the items, they deliberately answered in an odd fashion, or the items need to be dropped or reworded to increase consistency. This touches on one assumption of survey research, which is that participants answer honestly.
Test - Retest Reliability
The next form of reliability is test - retest reliability. As the name implies, it entails administering the test or assessment on more than one occasion. Test - retest reliability refers to the ability of an instrument or assessment to produce similar outcomes in the same group of people on multiple occasions in an amount of time that would not lead to a change in the measurement of interest.1,2 Let's assume that you use a dynamometer to assess the triceps extension strength in a group of athletes. The athletes are healthy college males between 18 and 24 years of age who are free of injury or disease and not participating in a strength training program. You would not expect their triceps strength to change in a week's time. It is safe to assume that the scores of the first test and those of the second test will be similar. In this case, large changes in strength measures are likely due to unreliable measures being detected by the dynamometer or test administrator error.
Not all test - retest measures are as easy to visualize or straightforward as the strength testing example. If you take a standardized test such as the SAT or GRE twice, a week apart, and you don't study between the tests, your scores will likely be similar. That's because taking a large standardized test covering topics you might not know won't teach you the topics. If a person scores low in algebra on the SAT, merely taking the SAT again won't improve his ability to apply algebra. This is not the case with certain types of cognitive assessment. A standard practice for the assessment of concussion often includes a computerized concussion assessment. These types of tests, however, result in athletes learning how to take the test, subsequently increasing their scores on follow-up trials. This is referred to as a practice effect.
Two of the most common cognitive-based tests in clinical athletic training practice are the ImPACT Concussion Management Software (ImPACT Applications, Pittsburgh, PA) and Concussion Sentinel (CogState Ltd, Victoria, Australia). These tests present with low to moderate test - retest reliability. This means that the scores of participants completing the assessments varied more than what is considered acceptable between tests, which diminishes their clinical relevance. Test - retest reliability is often reported in the literature as a value between 0 and 1, with value of 1 being more reliable or having fewer random effects that are unaccounted for in the results.3 The closer these reported values are to 1, the more reliable the assessments are, or the more trust you can put in them.
Interrater Reliability
Consensus between professionals on how to best treat an injury is important. It does, however, require that similar results or conditions be interpreted the same way. If you are an experienced AT reviewing injury reports upon returning to work after a few days off, you want to trust that other ATs completed certain orthopedic assessments the same way you would. More important, you hope that those tests would render the same outcome regardless of the person performing them. This is interrater reliability - that is, the agreement between clinicians who perform independent assessments of a condition.6 It is important to note that the consensus should be reached by both clinicians, but independently, without cognitive influence from each other. When the second AT is assessing an injury using the same assessment as the first, she should not know the outcome of the first.6 This ensures that the person performing the second assessment is not influenced by the outcome of the first assessment.
Also important when determining interrater reliability is that the athlete not recall the first assessment and respond differently to the second. This could be problematic when the assessment is on an injured body part and resulted in discomfort; the athlete may respond differently to the second assessment out of fear of re-creating the discomfort.
When reviewing articles and trying to determine the value of the results, remember that the higher the values for interrater reliability are, the more consistent the outcomes will be between assessors. Detailed examples are provided in subsequent chapters.
Intrarater Reliability
Intrarater reliability relates to the reproducibility of a clinical measure when performed by the same clinician on more than one occasion on the same athlete.2 The purpose of determining intrarater reliability is to assess whether the clinician is consistently using a reliable method of assessment. If the assessment method is not established as reliable, we would return to test - retest reliability to establish trust in the assessment or instrument (recall the tape cutters). An example would be assessing the range of motion of a knee. The bony landmarks used to measure knee flexion and the mechanics of the goniometer will not change from day 1 to day 2. However, range of motion assessments often vary when repeated by the same clinician. Imagine the difficulty in creating a treatment plan when on day 1 an athlete is assessed with 100° of flexion and on day 2 has only 90° of flexion. Variations in observed outcomes will confound the care provided to athletes.
Learn more about Evidence-Based Practice in Athletic Training.
Quickly and Easily Identify Low-Quality Articles Using CRAAP
Chapter 2 demonstrated how to perform a database search, and chapter 3 introduced ways to understand and evaluate the evidence. This chapter continues the discussion of evaluation methods by addressing how to appraise the diagnostic articles you have found.
Methods
Chapter 1 outlined the following five steps of evidence-based practice:
- Create aclinically relevant and searchable question concerning your athlete.
- Conduct a database search to find the best evidence.
- Critically appraise the articles or evidence for quality.
- Critically synthesize the evidence to choose and implement a treatment.
- Assess the outcomes by monitoring the athlete.
Chapter 2 demonstrated how to perform a database search, and chapter 3 introduced ways to understand and evaluate the evidence. This chapter continues the discussion of evaluation methods by addressing how to appraise the diagnostic articles you have found. Following are four steps to appraising diagnostic research:1
- Determine the applicability to your athlete.
- Assess the quality of the article.
- Review the results.
- Summarize the clinical bottom line.
As you gather your articles using a variety of search methods, you can apply a simple assessment to quickly and easily eliminate low-quality articles. Review any article based on the following five criteria (CRAAP):
- Currency. Is the information in need of being revised? The 2004 National Athletic Trainers' Association (NATA) position statement on the management of sport-related concussion allowed for same-day return to participation if symptoms resolved and the athlete passed a return-to-play test.3 The 2014 position statement update noted that our knowledge about how to treat concussions had significantly changed and ruled guidelines as recent as 10 years old obsolete.
- Relevance. Is the article written at a level appropriate for your needs, and do you understand it? If not, you may fail to reach a valid conclusion about the use of assessments in your situation. Has the information been peer reviewed, or is it an editorial or opinion piece? Editorials often have a bias and lack the scientific rigor required to make the information useful for sound clinical decisions.
- Authority. Is a sponsor involved that has a product to market? If so, read the information with great care. If an article is promoting an aquatics rehab program to aid in the recovery of knee surgery, but it was sponsored by a company that sells therapy pools, you need to suspect some potential bias. These types of conflicts of interest can be questionable.
- Accuracy. After completing your primary review, it is time to dig a little deeper and appraise the article. The Quality Assessment of Diagnostic Accuracy Studies (QUADAS) is a 14-question instrument to assist you in the appraisal (see the sidebar).1,4 The answer choices are yes, no, or uncertain; a greater number of yes answers is preferred.1 There is no set number of acceptable yes answers, but if any uncertain or no answers make you feel uncomfortable using the data to make a clinical decision, you may want to further research the topic and expand the number of articles you review.
- Purpose. Why is the information presented? Is it an advertisement? Is it funded by someone with a financial stake in selling something? If the answer to the preceding questions is yes, be sure to thoroughly review the academic content. The purpose of a quality article is to inform and provide outcomes, allowing readers to apply it as they see fit.
Each of the questions should be clearly addressed or answered in the article. If you are interested in learning whether a rehab program would be appropriate for high school athletes, the studies you review should have been conducted with people in the same age range. This is called generalizability - that is, can the results of a study be applied to another population based on similarities in the populations?
Authors should clearly tell you how they selected participants and why, as well as any reasons participants were excluded from participation. Did the author report issues with the data that could not be explained, and was the number of participants at the start and end of the study reported? If fewer completed the study than initiated it, was there an explanation for the dropout rate? If an author was researching the use of latex bandages to decrease swelling after ankle sprains and a volunteer participant had an allergy to latex, that person would need to leave the study. In this case, an author should report that 40 subjects volunteered to participate and were initially taking part in the study, and one was removed because of an exclusionary criterion. The author should not say that 39 people agreed to participate.
Similarly, if a new test (index test) is being assessed, it should be compared to an appropriate gold standard reference. If a researcher is testing a new axillary thermometer to establish whether it accurately assesses core body temperature, it should be compared to rectal thermometer measures. Comparing a new axillary thermometer to oral thermometers or others that are not scientifically accepted as the gold standard could introduce significant error, which would make the study results inappropriate to use in clinical practice.
Also important when appraising an article is assessing its methods section. Is it detailed enough that you could replicate it, if desired? You should be able to fully understand what was done during the study when reading the methods section.
The prevention of bias is also important during a study. If a participant was assessed using both an indexed and a referenced test, were the results interpreted separately? In other words, did the person completing the assessments know the results of the comparative assessment? A person who is aware of the other outcomes when completing an assessment creates a potential for biased reporting and a loss of objectivity, which is important in research.
Learn more about Evidence-Based Practice in Athletic Training.
What is Evidence-Based Practice Anyway?
Evidence-based practice (EBP) is a systematic method of reviewing the best evidence, combining it with the art of athletic training or your clinical expertise, and making informed choices to treat the athlete.
Definition of Evidence-Based Practice
Evidence-based practice (EBP) is a systematic method of reviewing the best evidence, combining it with the art of athletic training or your clinical expertise, and making informed choices to treat the athlete. Integrating EBP requires that you incorporate your clinical expertise, your athletes' values, and the best available clinical research into your practice. At its core, EBP is about focusing on athlete and clinical outcomes.
Clinical expertise is the culmination of your experience treating and providing care to athletes. It includes your personal values, preferences, experiences, and wisdom. Clinical expertise develops from hours of observation and trial and error and can result in a sense of pride and emotion. An AT who is vested in caring for an athlete can be uncomfortable being wrong about an assessment or a rehab and treatment program. Likewise, facing a challenging clinical case, deciding on a course of action, and having that action confirmed by other health care providers can be very rewarding. This is the reason many ATs are wary of incorporating new techniques that are unfamiliar to them. However, such a guarded approach can keep them from remembering that optimal athlete outcomes are the most important goal.
Patient values are the preconceived ideas and beliefs each athlete brings to the clinical setting. Athletes all have their own distinctive concerns that you must address. Often, the athlete arrives with a set feeling of trust in or distrust of ATs that has been cultivated from prior experience or social discussions. Furthermore, the athlete's values can be skewed by the appearance or functional use of the athletic training room and the amount of traffic. Some arrive lacking a clear understanding of the body and the healing process and may have set unrealistic time lines for their return to participation. This will influence their adherence to, or desire to complete, therapy. Treatment can be further confounded by comorbidities and the athlete's support network. Although the challenge is unique for each athlete, as an EBP clinician, you must consider all athletes' perspectives.
Clinical research is a scientific and systematic process that generates evidence through hypothesis testing and sound methodology. This is perhaps the most challenging portion of the EBP concept to incorporate. You may have limited experience analyzing peer-reviewed literature, and learning this process may thus appear daunting. Published studies should at a bare minimum introduce a research question, describe the methods used to assess subjects, outline the variables of interest, and explain the participant inclusion and exclusion criteria. The statistical analysis should align with the methods, and the authors should report only facts in the results section. The discussion or impression of the outcomes belongs in the conclusion or discussion section of an article. When assessing the quality of a clinical research article, ask the following questions:
- Were eligibility criteria specified?
- Were subjects randomly assigned to groups, and was assignment concealed, if appropriate?
- Were subjects similar at baseline?
- As appropriate, were subjects, clinicians, and assessors blinded to the treatment of participants?
- Was the completion rate of participants 85% or higher?
Eligibility criteria are a set of delimiters that detail who can be included in a study. These might include age limits, an association with a specific sport, a particular injury or medical condition, ethnicity, or sex. You need to know whether the study will be relevant to the athlete in your care.
Randomization is important in research because it helps to determine whether something truly made an impact, or whether it might have occurred by chance alone or as a result of a preconceived bias. An unethical researcher might place participants in a study group according to a notion that one group will respond more favorably to the treatment, with the purpose of showing support for that treatment. Likewise, whether participants were blinded (not aware that they were placed in a control or experimental group) is an important consideration when reviewing a study. To help the researcher or support a product or treatment they favor, participants who are not blinded may alter their efforts on tests or skew their reporting.
The similarity of participants is also crucial when determining a study's relevance. Participants should be similar in such factors as age, sport, and condition. It would be inappropriate to conclude that a new treatment works if those in the experimental group receiving the treatment were vastly different from those in the control group who did not receive a treatment. Furthermore, blinding the assessors to the participants' treatment limits the possibility of assessor bias.
Finally, researchers should report the participant completion rate to reveal the percentage of participants who dropped out of the study. A completion rate of less than 85% raises suspicion. Reviewers should question why so many participants dropped out. Was the study poorly organized? Was it painful? Was it too cumbersome or difficult?
With all of these aspects of research to consider, always keep the three components of EBP (clinical expertise, patient values, and clinical research) in mind. The bottom line is that EBP should always revolve around the athlete.
Five Steps of Evidence-Based Practice
To use evidence-based practice and retain an athlete focus, follow these five steps:
- Create aclinically relevant and searchable question concerning your athlete.
- Conduct a database search to find the best evidence.
- Critically appraise the articles or evidence for quality.
- Critically synthesize the evidence to choose and implement a treatment.
- Assess the outcomes by monitoring the athlete.
Let's take a closer look at each of these steps.
- Step 1: The first step is to start with the athlete, establish the clinical concern, and build a clinical question that centers on solving the issue or treating the condition. Further details on developing a clinical research question are presented in chapter 2. In the meantime, remember that the clinical question should be focused and searchable. A question that is too narrow in scope will return limited results. One that is too wide will return excessive information, limiting your ability to incorporate the content into a treatment plan. For example, if you are curious about the ability of ultrasound to facilitate increased tissue extensibility and are treating an athlete with ITB syndrome, searching for treat ITB with ultrasound may result in few or no results. It is too narrow and specific. If you search for ultrasound treatment, the results will most likely be too wide. A more appropriate approach might be to search for ultrasound AND connective tissue. This would return studies related to ultrasound and its effects on connective tissue. This will not be specific to ITB, but because ITB involves connective tissue, the treatment parameters and outcomes may be similar enough to draw upon for clinical use.
- Step 2: This step pertains to conducting the database search and looking for the best evidence related to the athlete's condition or injury. Chapter 2 explains how to conduct the search (including Boolean modifiers and search engines and databases), format the question, and conduct the first search. With practice, you will become adept at completing this in minimal time.
- Step 3: Appraise the articles for quality and for whether they are applicable to your athlete.1 Chapters 3 through 6 will help you learn how to do this. It is helpful to rate the quality and applicability of the studies you find on these two scales. A study may be of high quality but not applicable to your current athlete. For example, consider the effect of counterirritants. A research project that applied counterirritants to athletes with arthritis and found no improvement in range of motion at the fingers would not apply to an athlete's use of counterirritants to treat muscle spasms of the gastrocnemius. In this example, the study might be well designed but have no relevance to an athlete with a tight gastrocnemius. Conversely, a study of the effect of counterirritants on athletes with muscle spasms might be very applicable. However, if the article fails to report subjects' baseline measures, the inclusion or exclusion criteria, or the type of counterirritant used, this would indicate low quality. Without a certain amount of requisite data presented in the study, it would be impossible to devise a treatment plan.
- Step 4: This step involves critical processing and synthesizing. Integrating the evidence, your own clinical expertise and comfort in performing certain skills, and the values of the athlete will form the framework for your treatment plan.
- Step 5: This step returns to the athlete. You need to evaluate critically the progress and outcomes, reflecting on steps 1 through 4 and continually aiming to improve outcomes.
Learn more about Evidence-Based Practice in Athletic Training.
Is the Test Reliable?
The first term that is important for a consumer of research to understand is reliability. By strict definition, reliability refers to consistency in measures attained by people or instruments.
Reliability
The first term that is important for a consumer of research to understand is reliability. By strict definition, reliability refers to consistency in measures attained by people or instruments.1,2 Basically, does the tool you are using produce the same outcome each time you use it? An easy example is a pair of tape cutters. If every time you use the cutters to remove tape, you achieve a nice, clean cut, you would consider these cutters reliable. However, if sometimes they cut the tape and other times they do not, you would never know what outcome to expect; you would therefore consider these cutters unreliable. Reliability is important when you are trying to make a clinical decision. If you are performing an anterior drawer test on an ankle and one time you perform the test you feel laxity and the next time you fail to feel laxity, you will have difficulty making a decision. You may question whether the athlete changed, whether you applied different degrees of force, or whether the test itself is unreliable.
Reliability also relates to questionnaires. A well-structured item on a questionnaire should be easy to understand and not subject to misinterpretation. Let's look at an example of an item that uses a Likert scale of 1 to 4, with 1 being low or disagree and 4 being high or strongly agree. Here is the statement: Some of the most important tasks of an athletic trainer are administration duties and the bidding process. Although we can all agree that athletic trainers (ATs) often have some administrative duties and likely need to bid on purchases, this is a poor item that would probably be unreliable. One reader might believe that the most important duties of an AT relate to prevention and mark 1 for this item. Another reader might focus on the bidding portion of the statement and, believing that it's not that important, also mark 1. The researcher in this case has a value of 1 on two occasions but for very different reasons. However, an AT with administrative duties might focus on the administrative portion of the statement and rate this a 4. These varied scores based on individual interpretations of the statement would lead to low reliability and prohibit the researcher from drawing a valid conclusion.
Perhaps a better statement would look like this: Athletic trainers should have administrative skills. The scores for this may also vary, but they are less likely to vary because of misinterpretation. The variance in the scores would most likely be due to a variance in the opinions of the respondents. Thus, it is important to know that a study you are reviewing reports reliability values; without them, it's very difficult to trust the findings and make a judgment based on them.
You should be minimally familiar with several types of reliability measures as an evidence-based clinician. Presented here are four primary types: internal reliability, test - retest reliability, interrater reliability, and intrarater reliability.
Internal Reliability
Internal reliability is often an issue with surveys and questionnaires that employ a Likert scale system of values. The researcher wants to know whether the participant is responding to similar questions consistently. That is, did the person answer different questions about the same topic or construct in a similar manner?1 Let's assume that the researcher has a 30-item survey to find out athletes' opinions of the school's coaching staff. On item 4, which states that the school's coaching staff sincerely cares about athletes, an athlete might respond with a 1 (low or disagree). However, item 22 asks whether the athlete agrees that the coaches exhibit a caring nature; the athlete might score this item as a 4 (high or strongly agree). This is an example of two items asking about the same construct and receiving two very different responses, when the researcher would expect these answers to be similar. However, just because one person completes a survey in this fashion does not mean that the survey is unreliable. Only if a large numberof participants answer the survey in this fashion would it be considered low in internal reliability. This indicates that the sample population misunderstood the items, they deliberately answered in an odd fashion, or the items need to be dropped or reworded to increase consistency. This touches on one assumption of survey research, which is that participants answer honestly.
Test - Retest Reliability
The next form of reliability is test - retest reliability. As the name implies, it entails administering the test or assessment on more than one occasion. Test - retest reliability refers to the ability of an instrument or assessment to produce similar outcomes in the same group of people on multiple occasions in an amount of time that would not lead to a change in the measurement of interest.1,2 Let's assume that you use a dynamometer to assess the triceps extension strength in a group of athletes. The athletes are healthy college males between 18 and 24 years of age who are free of injury or disease and not participating in a strength training program. You would not expect their triceps strength to change in a week's time. It is safe to assume that the scores of the first test and those of the second test will be similar. In this case, large changes in strength measures are likely due to unreliable measures being detected by the dynamometer or test administrator error.
Not all test - retest measures are as easy to visualize or straightforward as the strength testing example. If you take a standardized test such as the SAT or GRE twice, a week apart, and you don't study between the tests, your scores will likely be similar. That's because taking a large standardized test covering topics you might not know won't teach you the topics. If a person scores low in algebra on the SAT, merely taking the SAT again won't improve his ability to apply algebra. This is not the case with certain types of cognitive assessment. A standard practice for the assessment of concussion often includes a computerized concussion assessment. These types of tests, however, result in athletes learning how to take the test, subsequently increasing their scores on follow-up trials. This is referred to as a practice effect.
Two of the most common cognitive-based tests in clinical athletic training practice are the ImPACT Concussion Management Software (ImPACT Applications, Pittsburgh, PA) and Concussion Sentinel (CogState Ltd, Victoria, Australia). These tests present with low to moderate test - retest reliability. This means that the scores of participants completing the assessments varied more than what is considered acceptable between tests, which diminishes their clinical relevance. Test - retest reliability is often reported in the literature as a value between 0 and 1, with value of 1 being more reliable or having fewer random effects that are unaccounted for in the results.3 The closer these reported values are to 1, the more reliable the assessments are, or the more trust you can put in them.
Interrater Reliability
Consensus between professionals on how to best treat an injury is important. It does, however, require that similar results or conditions be interpreted the same way. If you are an experienced AT reviewing injury reports upon returning to work after a few days off, you want to trust that other ATs completed certain orthopedic assessments the same way you would. More important, you hope that those tests would render the same outcome regardless of the person performing them. This is interrater reliability - that is, the agreement between clinicians who perform independent assessments of a condition.6 It is important to note that the consensus should be reached by both clinicians, but independently, without cognitive influence from each other. When the second AT is assessing an injury using the same assessment as the first, she should not know the outcome of the first.6 This ensures that the person performing the second assessment is not influenced by the outcome of the first assessment.
Also important when determining interrater reliability is that the athlete not recall the first assessment and respond differently to the second. This could be problematic when the assessment is on an injured body part and resulted in discomfort; the athlete may respond differently to the second assessment out of fear of re-creating the discomfort.
When reviewing articles and trying to determine the value of the results, remember that the higher the values for interrater reliability are, the more consistent the outcomes will be between assessors. Detailed examples are provided in subsequent chapters.
Intrarater Reliability
Intrarater reliability relates to the reproducibility of a clinical measure when performed by the same clinician on more than one occasion on the same athlete.2 The purpose of determining intrarater reliability is to assess whether the clinician is consistently using a reliable method of assessment. If the assessment method is not established as reliable, we would return to test - retest reliability to establish trust in the assessment or instrument (recall the tape cutters). An example would be assessing the range of motion of a knee. The bony landmarks used to measure knee flexion and the mechanics of the goniometer will not change from day 1 to day 2. However, range of motion assessments often vary when repeated by the same clinician. Imagine the difficulty in creating a treatment plan when on day 1 an athlete is assessed with 100° of flexion and on day 2 has only 90° of flexion. Variations in observed outcomes will confound the care provided to athletes.
Learn more about Evidence-Based Practice in Athletic Training.
Quickly and Easily Identify Low-Quality Articles Using CRAAP
Chapter 2 demonstrated how to perform a database search, and chapter 3 introduced ways to understand and evaluate the evidence. This chapter continues the discussion of evaluation methods by addressing how to appraise the diagnostic articles you have found.
Methods
Chapter 1 outlined the following five steps of evidence-based practice:
- Create aclinically relevant and searchable question concerning your athlete.
- Conduct a database search to find the best evidence.
- Critically appraise the articles or evidence for quality.
- Critically synthesize the evidence to choose and implement a treatment.
- Assess the outcomes by monitoring the athlete.
Chapter 2 demonstrated how to perform a database search, and chapter 3 introduced ways to understand and evaluate the evidence. This chapter continues the discussion of evaluation methods by addressing how to appraise the diagnostic articles you have found. Following are four steps to appraising diagnostic research:1
- Determine the applicability to your athlete.
- Assess the quality of the article.
- Review the results.
- Summarize the clinical bottom line.
As you gather your articles using a variety of search methods, you can apply a simple assessment to quickly and easily eliminate low-quality articles. Review any article based on the following five criteria (CRAAP):
- Currency. Is the information in need of being revised? The 2004 National Athletic Trainers' Association (NATA) position statement on the management of sport-related concussion allowed for same-day return to participation if symptoms resolved and the athlete passed a return-to-play test.3 The 2014 position statement update noted that our knowledge about how to treat concussions had significantly changed and ruled guidelines as recent as 10 years old obsolete.
- Relevance. Is the article written at a level appropriate for your needs, and do you understand it? If not, you may fail to reach a valid conclusion about the use of assessments in your situation. Has the information been peer reviewed, or is it an editorial or opinion piece? Editorials often have a bias and lack the scientific rigor required to make the information useful for sound clinical decisions.
- Authority. Is a sponsor involved that has a product to market? If so, read the information with great care. If an article is promoting an aquatics rehab program to aid in the recovery of knee surgery, but it was sponsored by a company that sells therapy pools, you need to suspect some potential bias. These types of conflicts of interest can be questionable.
- Accuracy. After completing your primary review, it is time to dig a little deeper and appraise the article. The Quality Assessment of Diagnostic Accuracy Studies (QUADAS) is a 14-question instrument to assist you in the appraisal (see the sidebar).1,4 The answer choices are yes, no, or uncertain; a greater number of yes answers is preferred.1 There is no set number of acceptable yes answers, but if any uncertain or no answers make you feel uncomfortable using the data to make a clinical decision, you may want to further research the topic and expand the number of articles you review.
- Purpose. Why is the information presented? Is it an advertisement? Is it funded by someone with a financial stake in selling something? If the answer to the preceding questions is yes, be sure to thoroughly review the academic content. The purpose of a quality article is to inform and provide outcomes, allowing readers to apply it as they see fit.
Each of the questions should be clearly addressed or answered in the article. If you are interested in learning whether a rehab program would be appropriate for high school athletes, the studies you review should have been conducted with people in the same age range. This is called generalizability - that is, can the results of a study be applied to another population based on similarities in the populations?
Authors should clearly tell you how they selected participants and why, as well as any reasons participants were excluded from participation. Did the author report issues with the data that could not be explained, and was the number of participants at the start and end of the study reported? If fewer completed the study than initiated it, was there an explanation for the dropout rate? If an author was researching the use of latex bandages to decrease swelling after ankle sprains and a volunteer participant had an allergy to latex, that person would need to leave the study. In this case, an author should report that 40 subjects volunteered to participate and were initially taking part in the study, and one was removed because of an exclusionary criterion. The author should not say that 39 people agreed to participate.
Similarly, if a new test (index test) is being assessed, it should be compared to an appropriate gold standard reference. If a researcher is testing a new axillary thermometer to establish whether it accurately assesses core body temperature, it should be compared to rectal thermometer measures. Comparing a new axillary thermometer to oral thermometers or others that are not scientifically accepted as the gold standard could introduce significant error, which would make the study results inappropriate to use in clinical practice.
Also important when appraising an article is assessing its methods section. Is it detailed enough that you could replicate it, if desired? You should be able to fully understand what was done during the study when reading the methods section.
The prevention of bias is also important during a study. If a participant was assessed using both an indexed and a referenced test, were the results interpreted separately? In other words, did the person completing the assessments know the results of the comparative assessment? A person who is aware of the other outcomes when completing an assessment creates a potential for biased reporting and a loss of objectivity, which is important in research.
Learn more about Evidence-Based Practice in Athletic Training.
What is Evidence-Based Practice Anyway?
Evidence-based practice (EBP) is a systematic method of reviewing the best evidence, combining it with the art of athletic training or your clinical expertise, and making informed choices to treat the athlete.
Definition of Evidence-Based Practice
Evidence-based practice (EBP) is a systematic method of reviewing the best evidence, combining it with the art of athletic training or your clinical expertise, and making informed choices to treat the athlete. Integrating EBP requires that you incorporate your clinical expertise, your athletes' values, and the best available clinical research into your practice. At its core, EBP is about focusing on athlete and clinical outcomes.
Clinical expertise is the culmination of your experience treating and providing care to athletes. It includes your personal values, preferences, experiences, and wisdom. Clinical expertise develops from hours of observation and trial and error and can result in a sense of pride and emotion. An AT who is vested in caring for an athlete can be uncomfortable being wrong about an assessment or a rehab and treatment program. Likewise, facing a challenging clinical case, deciding on a course of action, and having that action confirmed by other health care providers can be very rewarding. This is the reason many ATs are wary of incorporating new techniques that are unfamiliar to them. However, such a guarded approach can keep them from remembering that optimal athlete outcomes are the most important goal.
Patient values are the preconceived ideas and beliefs each athlete brings to the clinical setting. Athletes all have their own distinctive concerns that you must address. Often, the athlete arrives with a set feeling of trust in or distrust of ATs that has been cultivated from prior experience or social discussions. Furthermore, the athlete's values can be skewed by the appearance or functional use of the athletic training room and the amount of traffic. Some arrive lacking a clear understanding of the body and the healing process and may have set unrealistic time lines for their return to participation. This will influence their adherence to, or desire to complete, therapy. Treatment can be further confounded by comorbidities and the athlete's support network. Although the challenge is unique for each athlete, as an EBP clinician, you must consider all athletes' perspectives.
Clinical research is a scientific and systematic process that generates evidence through hypothesis testing and sound methodology. This is perhaps the most challenging portion of the EBP concept to incorporate. You may have limited experience analyzing peer-reviewed literature, and learning this process may thus appear daunting. Published studies should at a bare minimum introduce a research question, describe the methods used to assess subjects, outline the variables of interest, and explain the participant inclusion and exclusion criteria. The statistical analysis should align with the methods, and the authors should report only facts in the results section. The discussion or impression of the outcomes belongs in the conclusion or discussion section of an article. When assessing the quality of a clinical research article, ask the following questions:
- Were eligibility criteria specified?
- Were subjects randomly assigned to groups, and was assignment concealed, if appropriate?
- Were subjects similar at baseline?
- As appropriate, were subjects, clinicians, and assessors blinded to the treatment of participants?
- Was the completion rate of participants 85% or higher?
Eligibility criteria are a set of delimiters that detail who can be included in a study. These might include age limits, an association with a specific sport, a particular injury or medical condition, ethnicity, or sex. You need to know whether the study will be relevant to the athlete in your care.
Randomization is important in research because it helps to determine whether something truly made an impact, or whether it might have occurred by chance alone or as a result of a preconceived bias. An unethical researcher might place participants in a study group according to a notion that one group will respond more favorably to the treatment, with the purpose of showing support for that treatment. Likewise, whether participants were blinded (not aware that they were placed in a control or experimental group) is an important consideration when reviewing a study. To help the researcher or support a product or treatment they favor, participants who are not blinded may alter their efforts on tests or skew their reporting.
The similarity of participants is also crucial when determining a study's relevance. Participants should be similar in such factors as age, sport, and condition. It would be inappropriate to conclude that a new treatment works if those in the experimental group receiving the treatment were vastly different from those in the control group who did not receive a treatment. Furthermore, blinding the assessors to the participants' treatment limits the possibility of assessor bias.
Finally, researchers should report the participant completion rate to reveal the percentage of participants who dropped out of the study. A completion rate of less than 85% raises suspicion. Reviewers should question why so many participants dropped out. Was the study poorly organized? Was it painful? Was it too cumbersome or difficult?
With all of these aspects of research to consider, always keep the three components of EBP (clinical expertise, patient values, and clinical research) in mind. The bottom line is that EBP should always revolve around the athlete.
Five Steps of Evidence-Based Practice
To use evidence-based practice and retain an athlete focus, follow these five steps:
- Create aclinically relevant and searchable question concerning your athlete.
- Conduct a database search to find the best evidence.
- Critically appraise the articles or evidence for quality.
- Critically synthesize the evidence to choose and implement a treatment.
- Assess the outcomes by monitoring the athlete.
Let's take a closer look at each of these steps.
- Step 1: The first step is to start with the athlete, establish the clinical concern, and build a clinical question that centers on solving the issue or treating the condition. Further details on developing a clinical research question are presented in chapter 2. In the meantime, remember that the clinical question should be focused and searchable. A question that is too narrow in scope will return limited results. One that is too wide will return excessive information, limiting your ability to incorporate the content into a treatment plan. For example, if you are curious about the ability of ultrasound to facilitate increased tissue extensibility and are treating an athlete with ITB syndrome, searching for treat ITB with ultrasound may result in few or no results. It is too narrow and specific. If you search for ultrasound treatment, the results will most likely be too wide. A more appropriate approach might be to search for ultrasound AND connective tissue. This would return studies related to ultrasound and its effects on connective tissue. This will not be specific to ITB, but because ITB involves connective tissue, the treatment parameters and outcomes may be similar enough to draw upon for clinical use.
- Step 2: This step pertains to conducting the database search and looking for the best evidence related to the athlete's condition or injury. Chapter 2 explains how to conduct the search (including Boolean modifiers and search engines and databases), format the question, and conduct the first search. With practice, you will become adept at completing this in minimal time.
- Step 3: Appraise the articles for quality and for whether they are applicable to your athlete.1 Chapters 3 through 6 will help you learn how to do this. It is helpful to rate the quality and applicability of the studies you find on these two scales. A study may be of high quality but not applicable to your current athlete. For example, consider the effect of counterirritants. A research project that applied counterirritants to athletes with arthritis and found no improvement in range of motion at the fingers would not apply to an athlete's use of counterirritants to treat muscle spasms of the gastrocnemius. In this example, the study might be well designed but have no relevance to an athlete with a tight gastrocnemius. Conversely, a study of the effect of counterirritants on athletes with muscle spasms might be very applicable. However, if the article fails to report subjects' baseline measures, the inclusion or exclusion criteria, or the type of counterirritant used, this would indicate low quality. Without a certain amount of requisite data presented in the study, it would be impossible to devise a treatment plan.
- Step 4: This step involves critical processing and synthesizing. Integrating the evidence, your own clinical expertise and comfort in performing certain skills, and the values of the athlete will form the framework for your treatment plan.
- Step 5: This step returns to the athlete. You need to evaluate critically the progress and outcomes, reflecting on steps 1 through 4 and continually aiming to improve outcomes.
Learn more about Evidence-Based Practice in Athletic Training.
Is the Test Reliable?
The first term that is important for a consumer of research to understand is reliability. By strict definition, reliability refers to consistency in measures attained by people or instruments.
Reliability
The first term that is important for a consumer of research to understand is reliability. By strict definition, reliability refers to consistency in measures attained by people or instruments.1,2 Basically, does the tool you are using produce the same outcome each time you use it? An easy example is a pair of tape cutters. If every time you use the cutters to remove tape, you achieve a nice, clean cut, you would consider these cutters reliable. However, if sometimes they cut the tape and other times they do not, you would never know what outcome to expect; you would therefore consider these cutters unreliable. Reliability is important when you are trying to make a clinical decision. If you are performing an anterior drawer test on an ankle and one time you perform the test you feel laxity and the next time you fail to feel laxity, you will have difficulty making a decision. You may question whether the athlete changed, whether you applied different degrees of force, or whether the test itself is unreliable.
Reliability also relates to questionnaires. A well-structured item on a questionnaire should be easy to understand and not subject to misinterpretation. Let's look at an example of an item that uses a Likert scale of 1 to 4, with 1 being low or disagree and 4 being high or strongly agree. Here is the statement: Some of the most important tasks of an athletic trainer are administration duties and the bidding process. Although we can all agree that athletic trainers (ATs) often have some administrative duties and likely need to bid on purchases, this is a poor item that would probably be unreliable. One reader might believe that the most important duties of an AT relate to prevention and mark 1 for this item. Another reader might focus on the bidding portion of the statement and, believing that it's not that important, also mark 1. The researcher in this case has a value of 1 on two occasions but for very different reasons. However, an AT with administrative duties might focus on the administrative portion of the statement and rate this a 4. These varied scores based on individual interpretations of the statement would lead to low reliability and prohibit the researcher from drawing a valid conclusion.
Perhaps a better statement would look like this: Athletic trainers should have administrative skills. The scores for this may also vary, but they are less likely to vary because of misinterpretation. The variance in the scores would most likely be due to a variance in the opinions of the respondents. Thus, it is important to know that a study you are reviewing reports reliability values; without them, it's very difficult to trust the findings and make a judgment based on them.
You should be minimally familiar with several types of reliability measures as an evidence-based clinician. Presented here are four primary types: internal reliability, test - retest reliability, interrater reliability, and intrarater reliability.
Internal Reliability
Internal reliability is often an issue with surveys and questionnaires that employ a Likert scale system of values. The researcher wants to know whether the participant is responding to similar questions consistently. That is, did the person answer different questions about the same topic or construct in a similar manner?1 Let's assume that the researcher has a 30-item survey to find out athletes' opinions of the school's coaching staff. On item 4, which states that the school's coaching staff sincerely cares about athletes, an athlete might respond with a 1 (low or disagree). However, item 22 asks whether the athlete agrees that the coaches exhibit a caring nature; the athlete might score this item as a 4 (high or strongly agree). This is an example of two items asking about the same construct and receiving two very different responses, when the researcher would expect these answers to be similar. However, just because one person completes a survey in this fashion does not mean that the survey is unreliable. Only if a large numberof participants answer the survey in this fashion would it be considered low in internal reliability. This indicates that the sample population misunderstood the items, they deliberately answered in an odd fashion, or the items need to be dropped or reworded to increase consistency. This touches on one assumption of survey research, which is that participants answer honestly.
Test - Retest Reliability
The next form of reliability is test - retest reliability. As the name implies, it entails administering the test or assessment on more than one occasion. Test - retest reliability refers to the ability of an instrument or assessment to produce similar outcomes in the same group of people on multiple occasions in an amount of time that would not lead to a change in the measurement of interest.1,2 Let's assume that you use a dynamometer to assess the triceps extension strength in a group of athletes. The athletes are healthy college males between 18 and 24 years of age who are free of injury or disease and not participating in a strength training program. You would not expect their triceps strength to change in a week's time. It is safe to assume that the scores of the first test and those of the second test will be similar. In this case, large changes in strength measures are likely due to unreliable measures being detected by the dynamometer or test administrator error.
Not all test - retest measures are as easy to visualize or straightforward as the strength testing example. If you take a standardized test such as the SAT or GRE twice, a week apart, and you don't study between the tests, your scores will likely be similar. That's because taking a large standardized test covering topics you might not know won't teach you the topics. If a person scores low in algebra on the SAT, merely taking the SAT again won't improve his ability to apply algebra. This is not the case with certain types of cognitive assessment. A standard practice for the assessment of concussion often includes a computerized concussion assessment. These types of tests, however, result in athletes learning how to take the test, subsequently increasing their scores on follow-up trials. This is referred to as a practice effect.
Two of the most common cognitive-based tests in clinical athletic training practice are the ImPACT Concussion Management Software (ImPACT Applications, Pittsburgh, PA) and Concussion Sentinel (CogState Ltd, Victoria, Australia). These tests present with low to moderate test - retest reliability. This means that the scores of participants completing the assessments varied more than what is considered acceptable between tests, which diminishes their clinical relevance. Test - retest reliability is often reported in the literature as a value between 0 and 1, with value of 1 being more reliable or having fewer random effects that are unaccounted for in the results.3 The closer these reported values are to 1, the more reliable the assessments are, or the more trust you can put in them.
Interrater Reliability
Consensus between professionals on how to best treat an injury is important. It does, however, require that similar results or conditions be interpreted the same way. If you are an experienced AT reviewing injury reports upon returning to work after a few days off, you want to trust that other ATs completed certain orthopedic assessments the same way you would. More important, you hope that those tests would render the same outcome regardless of the person performing them. This is interrater reliability - that is, the agreement between clinicians who perform independent assessments of a condition.6 It is important to note that the consensus should be reached by both clinicians, but independently, without cognitive influence from each other. When the second AT is assessing an injury using the same assessment as the first, she should not know the outcome of the first.6 This ensures that the person performing the second assessment is not influenced by the outcome of the first assessment.
Also important when determining interrater reliability is that the athlete not recall the first assessment and respond differently to the second. This could be problematic when the assessment is on an injured body part and resulted in discomfort; the athlete may respond differently to the second assessment out of fear of re-creating the discomfort.
When reviewing articles and trying to determine the value of the results, remember that the higher the values for interrater reliability are, the more consistent the outcomes will be between assessors. Detailed examples are provided in subsequent chapters.
Intrarater Reliability
Intrarater reliability relates to the reproducibility of a clinical measure when performed by the same clinician on more than one occasion on the same athlete.2 The purpose of determining intrarater reliability is to assess whether the clinician is consistently using a reliable method of assessment. If the assessment method is not established as reliable, we would return to test - retest reliability to establish trust in the assessment or instrument (recall the tape cutters). An example would be assessing the range of motion of a knee. The bony landmarks used to measure knee flexion and the mechanics of the goniometer will not change from day 1 to day 2. However, range of motion assessments often vary when repeated by the same clinician. Imagine the difficulty in creating a treatment plan when on day 1 an athlete is assessed with 100° of flexion and on day 2 has only 90° of flexion. Variations in observed outcomes will confound the care provided to athletes.
Learn more about Evidence-Based Practice in Athletic Training.
Quickly and Easily Identify Low-Quality Articles Using CRAAP
Chapter 2 demonstrated how to perform a database search, and chapter 3 introduced ways to understand and evaluate the evidence. This chapter continues the discussion of evaluation methods by addressing how to appraise the diagnostic articles you have found.
Methods
Chapter 1 outlined the following five steps of evidence-based practice:
- Create aclinically relevant and searchable question concerning your athlete.
- Conduct a database search to find the best evidence.
- Critically appraise the articles or evidence for quality.
- Critically synthesize the evidence to choose and implement a treatment.
- Assess the outcomes by monitoring the athlete.
Chapter 2 demonstrated how to perform a database search, and chapter 3 introduced ways to understand and evaluate the evidence. This chapter continues the discussion of evaluation methods by addressing how to appraise the diagnostic articles you have found. Following are four steps to appraising diagnostic research:1
- Determine the applicability to your athlete.
- Assess the quality of the article.
- Review the results.
- Summarize the clinical bottom line.
As you gather your articles using a variety of search methods, you can apply a simple assessment to quickly and easily eliminate low-quality articles. Review any article based on the following five criteria (CRAAP):
- Currency. Is the information in need of being revised? The 2004 National Athletic Trainers' Association (NATA) position statement on the management of sport-related concussion allowed for same-day return to participation if symptoms resolved and the athlete passed a return-to-play test.3 The 2014 position statement update noted that our knowledge about how to treat concussions had significantly changed and ruled guidelines as recent as 10 years old obsolete.
- Relevance. Is the article written at a level appropriate for your needs, and do you understand it? If not, you may fail to reach a valid conclusion about the use of assessments in your situation. Has the information been peer reviewed, or is it an editorial or opinion piece? Editorials often have a bias and lack the scientific rigor required to make the information useful for sound clinical decisions.
- Authority. Is a sponsor involved that has a product to market? If so, read the information with great care. If an article is promoting an aquatics rehab program to aid in the recovery of knee surgery, but it was sponsored by a company that sells therapy pools, you need to suspect some potential bias. These types of conflicts of interest can be questionable.
- Accuracy. After completing your primary review, it is time to dig a little deeper and appraise the article. The Quality Assessment of Diagnostic Accuracy Studies (QUADAS) is a 14-question instrument to assist you in the appraisal (see the sidebar).1,4 The answer choices are yes, no, or uncertain; a greater number of yes answers is preferred.1 There is no set number of acceptable yes answers, but if any uncertain or no answers make you feel uncomfortable using the data to make a clinical decision, you may want to further research the topic and expand the number of articles you review.
- Purpose. Why is the information presented? Is it an advertisement? Is it funded by someone with a financial stake in selling something? If the answer to the preceding questions is yes, be sure to thoroughly review the academic content. The purpose of a quality article is to inform and provide outcomes, allowing readers to apply it as they see fit.
Each of the questions should be clearly addressed or answered in the article. If you are interested in learning whether a rehab program would be appropriate for high school athletes, the studies you review should have been conducted with people in the same age range. This is called generalizability - that is, can the results of a study be applied to another population based on similarities in the populations?
Authors should clearly tell you how they selected participants and why, as well as any reasons participants were excluded from participation. Did the author report issues with the data that could not be explained, and was the number of participants at the start and end of the study reported? If fewer completed the study than initiated it, was there an explanation for the dropout rate? If an author was researching the use of latex bandages to decrease swelling after ankle sprains and a volunteer participant had an allergy to latex, that person would need to leave the study. In this case, an author should report that 40 subjects volunteered to participate and were initially taking part in the study, and one was removed because of an exclusionary criterion. The author should not say that 39 people agreed to participate.
Similarly, if a new test (index test) is being assessed, it should be compared to an appropriate gold standard reference. If a researcher is testing a new axillary thermometer to establish whether it accurately assesses core body temperature, it should be compared to rectal thermometer measures. Comparing a new axillary thermometer to oral thermometers or others that are not scientifically accepted as the gold standard could introduce significant error, which would make the study results inappropriate to use in clinical practice.
Also important when appraising an article is assessing its methods section. Is it detailed enough that you could replicate it, if desired? You should be able to fully understand what was done during the study when reading the methods section.
The prevention of bias is also important during a study. If a participant was assessed using both an indexed and a referenced test, were the results interpreted separately? In other words, did the person completing the assessments know the results of the comparative assessment? A person who is aware of the other outcomes when completing an assessment creates a potential for biased reporting and a loss of objectivity, which is important in research.
Learn more about Evidence-Based Practice in Athletic Training.
What is Evidence-Based Practice Anyway?
Evidence-based practice (EBP) is a systematic method of reviewing the best evidence, combining it with the art of athletic training or your clinical expertise, and making informed choices to treat the athlete.
Definition of Evidence-Based Practice
Evidence-based practice (EBP) is a systematic method of reviewing the best evidence, combining it with the art of athletic training or your clinical expertise, and making informed choices to treat the athlete. Integrating EBP requires that you incorporate your clinical expertise, your athletes' values, and the best available clinical research into your practice. At its core, EBP is about focusing on athlete and clinical outcomes.
Clinical expertise is the culmination of your experience treating and providing care to athletes. It includes your personal values, preferences, experiences, and wisdom. Clinical expertise develops from hours of observation and trial and error and can result in a sense of pride and emotion. An AT who is vested in caring for an athlete can be uncomfortable being wrong about an assessment or a rehab and treatment program. Likewise, facing a challenging clinical case, deciding on a course of action, and having that action confirmed by other health care providers can be very rewarding. This is the reason many ATs are wary of incorporating new techniques that are unfamiliar to them. However, such a guarded approach can keep them from remembering that optimal athlete outcomes are the most important goal.
Patient values are the preconceived ideas and beliefs each athlete brings to the clinical setting. Athletes all have their own distinctive concerns that you must address. Often, the athlete arrives with a set feeling of trust in or distrust of ATs that has been cultivated from prior experience or social discussions. Furthermore, the athlete's values can be skewed by the appearance or functional use of the athletic training room and the amount of traffic. Some arrive lacking a clear understanding of the body and the healing process and may have set unrealistic time lines for their return to participation. This will influence their adherence to, or desire to complete, therapy. Treatment can be further confounded by comorbidities and the athlete's support network. Although the challenge is unique for each athlete, as an EBP clinician, you must consider all athletes' perspectives.
Clinical research is a scientific and systematic process that generates evidence through hypothesis testing and sound methodology. This is perhaps the most challenging portion of the EBP concept to incorporate. You may have limited experience analyzing peer-reviewed literature, and learning this process may thus appear daunting. Published studies should at a bare minimum introduce a research question, describe the methods used to assess subjects, outline the variables of interest, and explain the participant inclusion and exclusion criteria. The statistical analysis should align with the methods, and the authors should report only facts in the results section. The discussion or impression of the outcomes belongs in the conclusion or discussion section of an article. When assessing the quality of a clinical research article, ask the following questions:
- Were eligibility criteria specified?
- Were subjects randomly assigned to groups, and was assignment concealed, if appropriate?
- Were subjects similar at baseline?
- As appropriate, were subjects, clinicians, and assessors blinded to the treatment of participants?
- Was the completion rate of participants 85% or higher?
Eligibility criteria are a set of delimiters that detail who can be included in a study. These might include age limits, an association with a specific sport, a particular injury or medical condition, ethnicity, or sex. You need to know whether the study will be relevant to the athlete in your care.
Randomization is important in research because it helps to determine whether something truly made an impact, or whether it might have occurred by chance alone or as a result of a preconceived bias. An unethical researcher might place participants in a study group according to a notion that one group will respond more favorably to the treatment, with the purpose of showing support for that treatment. Likewise, whether participants were blinded (not aware that they were placed in a control or experimental group) is an important consideration when reviewing a study. To help the researcher or support a product or treatment they favor, participants who are not blinded may alter their efforts on tests or skew their reporting.
The similarity of participants is also crucial when determining a study's relevance. Participants should be similar in such factors as age, sport, and condition. It would be inappropriate to conclude that a new treatment works if those in the experimental group receiving the treatment were vastly different from those in the control group who did not receive a treatment. Furthermore, blinding the assessors to the participants' treatment limits the possibility of assessor bias.
Finally, researchers should report the participant completion rate to reveal the percentage of participants who dropped out of the study. A completion rate of less than 85% raises suspicion. Reviewers should question why so many participants dropped out. Was the study poorly organized? Was it painful? Was it too cumbersome or difficult?
With all of these aspects of research to consider, always keep the three components of EBP (clinical expertise, patient values, and clinical research) in mind. The bottom line is that EBP should always revolve around the athlete.
Five Steps of Evidence-Based Practice
To use evidence-based practice and retain an athlete focus, follow these five steps:
- Create aclinically relevant and searchable question concerning your athlete.
- Conduct a database search to find the best evidence.
- Critically appraise the articles or evidence for quality.
- Critically synthesize the evidence to choose and implement a treatment.
- Assess the outcomes by monitoring the athlete.
Let's take a closer look at each of these steps.
- Step 1: The first step is to start with the athlete, establish the clinical concern, and build a clinical question that centers on solving the issue or treating the condition. Further details on developing a clinical research question are presented in chapter 2. In the meantime, remember that the clinical question should be focused and searchable. A question that is too narrow in scope will return limited results. One that is too wide will return excessive information, limiting your ability to incorporate the content into a treatment plan. For example, if you are curious about the ability of ultrasound to facilitate increased tissue extensibility and are treating an athlete with ITB syndrome, searching for treat ITB with ultrasound may result in few or no results. It is too narrow and specific. If you search for ultrasound treatment, the results will most likely be too wide. A more appropriate approach might be to search for ultrasound AND connective tissue. This would return studies related to ultrasound and its effects on connective tissue. This will not be specific to ITB, but because ITB involves connective tissue, the treatment parameters and outcomes may be similar enough to draw upon for clinical use.
- Step 2: This step pertains to conducting the database search and looking for the best evidence related to the athlete's condition or injury. Chapter 2 explains how to conduct the search (including Boolean modifiers and search engines and databases), format the question, and conduct the first search. With practice, you will become adept at completing this in minimal time.
- Step 3: Appraise the articles for quality and for whether they are applicable to your athlete.1 Chapters 3 through 6 will help you learn how to do this. It is helpful to rate the quality and applicability of the studies you find on these two scales. A study may be of high quality but not applicable to your current athlete. For example, consider the effect of counterirritants. A research project that applied counterirritants to athletes with arthritis and found no improvement in range of motion at the fingers would not apply to an athlete's use of counterirritants to treat muscle spasms of the gastrocnemius. In this example, the study might be well designed but have no relevance to an athlete with a tight gastrocnemius. Conversely, a study of the effect of counterirritants on athletes with muscle spasms might be very applicable. However, if the article fails to report subjects' baseline measures, the inclusion or exclusion criteria, or the type of counterirritant used, this would indicate low quality. Without a certain amount of requisite data presented in the study, it would be impossible to devise a treatment plan.
- Step 4: This step involves critical processing and synthesizing. Integrating the evidence, your own clinical expertise and comfort in performing certain skills, and the values of the athlete will form the framework for your treatment plan.
- Step 5: This step returns to the athlete. You need to evaluate critically the progress and outcomes, reflecting on steps 1 through 4 and continually aiming to improve outcomes.
Learn more about Evidence-Based Practice in Athletic Training.
Is the Test Reliable?
The first term that is important for a consumer of research to understand is reliability. By strict definition, reliability refers to consistency in measures attained by people or instruments.
Reliability
The first term that is important for a consumer of research to understand is reliability. By strict definition, reliability refers to consistency in measures attained by people or instruments.1,2 Basically, does the tool you are using produce the same outcome each time you use it? An easy example is a pair of tape cutters. If every time you use the cutters to remove tape, you achieve a nice, clean cut, you would consider these cutters reliable. However, if sometimes they cut the tape and other times they do not, you would never know what outcome to expect; you would therefore consider these cutters unreliable. Reliability is important when you are trying to make a clinical decision. If you are performing an anterior drawer test on an ankle and one time you perform the test you feel laxity and the next time you fail to feel laxity, you will have difficulty making a decision. You may question whether the athlete changed, whether you applied different degrees of force, or whether the test itself is unreliable.
Reliability also relates to questionnaires. A well-structured item on a questionnaire should be easy to understand and not subject to misinterpretation. Let's look at an example of an item that uses a Likert scale of 1 to 4, with 1 being low or disagree and 4 being high or strongly agree. Here is the statement: Some of the most important tasks of an athletic trainer are administration duties and the bidding process. Although we can all agree that athletic trainers (ATs) often have some administrative duties and likely need to bid on purchases, this is a poor item that would probably be unreliable. One reader might believe that the most important duties of an AT relate to prevention and mark 1 for this item. Another reader might focus on the bidding portion of the statement and, believing that it's not that important, also mark 1. The researcher in this case has a value of 1 on two occasions but for very different reasons. However, an AT with administrative duties might focus on the administrative portion of the statement and rate this a 4. These varied scores based on individual interpretations of the statement would lead to low reliability and prohibit the researcher from drawing a valid conclusion.
Perhaps a better statement would look like this: Athletic trainers should have administrative skills. The scores for this may also vary, but they are less likely to vary because of misinterpretation. The variance in the scores would most likely be due to a variance in the opinions of the respondents. Thus, it is important to know that a study you are reviewing reports reliability values; without them, it's very difficult to trust the findings and make a judgment based on them.
You should be minimally familiar with several types of reliability measures as an evidence-based clinician. Presented here are four primary types: internal reliability, test - retest reliability, interrater reliability, and intrarater reliability.
Internal Reliability
Internal reliability is often an issue with surveys and questionnaires that employ a Likert scale system of values. The researcher wants to know whether the participant is responding to similar questions consistently. That is, did the person answer different questions about the same topic or construct in a similar manner?1 Let's assume that the researcher has a 30-item survey to find out athletes' opinions of the school's coaching staff. On item 4, which states that the school's coaching staff sincerely cares about athletes, an athlete might respond with a 1 (low or disagree). However, item 22 asks whether the athlete agrees that the coaches exhibit a caring nature; the athlete might score this item as a 4 (high or strongly agree). This is an example of two items asking about the same construct and receiving two very different responses, when the researcher would expect these answers to be similar. However, just because one person completes a survey in this fashion does not mean that the survey is unreliable. Only if a large numberof participants answer the survey in this fashion would it be considered low in internal reliability. This indicates that the sample population misunderstood the items, they deliberately answered in an odd fashion, or the items need to be dropped or reworded to increase consistency. This touches on one assumption of survey research, which is that participants answer honestly.
Test - Retest Reliability
The next form of reliability is test - retest reliability. As the name implies, it entails administering the test or assessment on more than one occasion. Test - retest reliability refers to the ability of an instrument or assessment to produce similar outcomes in the same group of people on multiple occasions in an amount of time that would not lead to a change in the measurement of interest.1,2 Let's assume that you use a dynamometer to assess the triceps extension strength in a group of athletes. The athletes are healthy college males between 18 and 24 years of age who are free of injury or disease and not participating in a strength training program. You would not expect their triceps strength to change in a week's time. It is safe to assume that the scores of the first test and those of the second test will be similar. In this case, large changes in strength measures are likely due to unreliable measures being detected by the dynamometer or test administrator error.
Not all test - retest measures are as easy to visualize or straightforward as the strength testing example. If you take a standardized test such as the SAT or GRE twice, a week apart, and you don't study between the tests, your scores will likely be similar. That's because taking a large standardized test covering topics you might not know won't teach you the topics. If a person scores low in algebra on the SAT, merely taking the SAT again won't improve his ability to apply algebra. This is not the case with certain types of cognitive assessment. A standard practice for the assessment of concussion often includes a computerized concussion assessment. These types of tests, however, result in athletes learning how to take the test, subsequently increasing their scores on follow-up trials. This is referred to as a practice effect.
Two of the most common cognitive-based tests in clinical athletic training practice are the ImPACT Concussion Management Software (ImPACT Applications, Pittsburgh, PA) and Concussion Sentinel (CogState Ltd, Victoria, Australia). These tests present with low to moderate test - retest reliability. This means that the scores of participants completing the assessments varied more than what is considered acceptable between tests, which diminishes their clinical relevance. Test - retest reliability is often reported in the literature as a value between 0 and 1, with value of 1 being more reliable or having fewer random effects that are unaccounted for in the results.3 The closer these reported values are to 1, the more reliable the assessments are, or the more trust you can put in them.
Interrater Reliability
Consensus between professionals on how to best treat an injury is important. It does, however, require that similar results or conditions be interpreted the same way. If you are an experienced AT reviewing injury reports upon returning to work after a few days off, you want to trust that other ATs completed certain orthopedic assessments the same way you would. More important, you hope that those tests would render the same outcome regardless of the person performing them. This is interrater reliability - that is, the agreement between clinicians who perform independent assessments of a condition.6 It is important to note that the consensus should be reached by both clinicians, but independently, without cognitive influence from each other. When the second AT is assessing an injury using the same assessment as the first, she should not know the outcome of the first.6 This ensures that the person performing the second assessment is not influenced by the outcome of the first assessment.
Also important when determining interrater reliability is that the athlete not recall the first assessment and respond differently to the second. This could be problematic when the assessment is on an injured body part and resulted in discomfort; the athlete may respond differently to the second assessment out of fear of re-creating the discomfort.
When reviewing articles and trying to determine the value of the results, remember that the higher the values for interrater reliability are, the more consistent the outcomes will be between assessors. Detailed examples are provided in subsequent chapters.
Intrarater Reliability
Intrarater reliability relates to the reproducibility of a clinical measure when performed by the same clinician on more than one occasion on the same athlete.2 The purpose of determining intrarater reliability is to assess whether the clinician is consistently using a reliable method of assessment. If the assessment method is not established as reliable, we would return to test - retest reliability to establish trust in the assessment or instrument (recall the tape cutters). An example would be assessing the range of motion of a knee. The bony landmarks used to measure knee flexion and the mechanics of the goniometer will not change from day 1 to day 2. However, range of motion assessments often vary when repeated by the same clinician. Imagine the difficulty in creating a treatment plan when on day 1 an athlete is assessed with 100° of flexion and on day 2 has only 90° of flexion. Variations in observed outcomes will confound the care provided to athletes.
Learn more about Evidence-Based Practice in Athletic Training.
Quickly and Easily Identify Low-Quality Articles Using CRAAP
Chapter 2 demonstrated how to perform a database search, and chapter 3 introduced ways to understand and evaluate the evidence. This chapter continues the discussion of evaluation methods by addressing how to appraise the diagnostic articles you have found.
Methods
Chapter 1 outlined the following five steps of evidence-based practice:
- Create aclinically relevant and searchable question concerning your athlete.
- Conduct a database search to find the best evidence.
- Critically appraise the articles or evidence for quality.
- Critically synthesize the evidence to choose and implement a treatment.
- Assess the outcomes by monitoring the athlete.
Chapter 2 demonstrated how to perform a database search, and chapter 3 introduced ways to understand and evaluate the evidence. This chapter continues the discussion of evaluation methods by addressing how to appraise the diagnostic articles you have found. Following are four steps to appraising diagnostic research:1
- Determine the applicability to your athlete.
- Assess the quality of the article.
- Review the results.
- Summarize the clinical bottom line.
As you gather your articles using a variety of search methods, you can apply a simple assessment to quickly and easily eliminate low-quality articles. Review any article based on the following five criteria (CRAAP):
- Currency. Is the information in need of being revised? The 2004 National Athletic Trainers' Association (NATA) position statement on the management of sport-related concussion allowed for same-day return to participation if symptoms resolved and the athlete passed a return-to-play test.3 The 2014 position statement update noted that our knowledge about how to treat concussions had significantly changed and ruled guidelines as recent as 10 years old obsolete.
- Relevance. Is the article written at a level appropriate for your needs, and do you understand it? If not, you may fail to reach a valid conclusion about the use of assessments in your situation. Has the information been peer reviewed, or is it an editorial or opinion piece? Editorials often have a bias and lack the scientific rigor required to make the information useful for sound clinical decisions.
- Authority. Is a sponsor involved that has a product to market? If so, read the information with great care. If an article is promoting an aquatics rehab program to aid in the recovery of knee surgery, but it was sponsored by a company that sells therapy pools, you need to suspect some potential bias. These types of conflicts of interest can be questionable.
- Accuracy. After completing your primary review, it is time to dig a little deeper and appraise the article. The Quality Assessment of Diagnostic Accuracy Studies (QUADAS) is a 14-question instrument to assist you in the appraisal (see the sidebar).1,4 The answer choices are yes, no, or uncertain; a greater number of yes answers is preferred.1 There is no set number of acceptable yes answers, but if any uncertain or no answers make you feel uncomfortable using the data to make a clinical decision, you may want to further research the topic and expand the number of articles you review.
- Purpose. Why is the information presented? Is it an advertisement? Is it funded by someone with a financial stake in selling something? If the answer to the preceding questions is yes, be sure to thoroughly review the academic content. The purpose of a quality article is to inform and provide outcomes, allowing readers to apply it as they see fit.
Each of the questions should be clearly addressed or answered in the article. If you are interested in learning whether a rehab program would be appropriate for high school athletes, the studies you review should have been conducted with people in the same age range. This is called generalizability - that is, can the results of a study be applied to another population based on similarities in the populations?
Authors should clearly tell you how they selected participants and why, as well as any reasons participants were excluded from participation. Did the author report issues with the data that could not be explained, and was the number of participants at the start and end of the study reported? If fewer completed the study than initiated it, was there an explanation for the dropout rate? If an author was researching the use of latex bandages to decrease swelling after ankle sprains and a volunteer participant had an allergy to latex, that person would need to leave the study. In this case, an author should report that 40 subjects volunteered to participate and were initially taking part in the study, and one was removed because of an exclusionary criterion. The author should not say that 39 people agreed to participate.
Similarly, if a new test (index test) is being assessed, it should be compared to an appropriate gold standard reference. If a researcher is testing a new axillary thermometer to establish whether it accurately assesses core body temperature, it should be compared to rectal thermometer measures. Comparing a new axillary thermometer to oral thermometers or others that are not scientifically accepted as the gold standard could introduce significant error, which would make the study results inappropriate to use in clinical practice.
Also important when appraising an article is assessing its methods section. Is it detailed enough that you could replicate it, if desired? You should be able to fully understand what was done during the study when reading the methods section.
The prevention of bias is also important during a study. If a participant was assessed using both an indexed and a referenced test, were the results interpreted separately? In other words, did the person completing the assessments know the results of the comparative assessment? A person who is aware of the other outcomes when completing an assessment creates a potential for biased reporting and a loss of objectivity, which is important in research.
Learn more about Evidence-Based Practice in Athletic Training.
What is Evidence-Based Practice Anyway?
Evidence-based practice (EBP) is a systematic method of reviewing the best evidence, combining it with the art of athletic training or your clinical expertise, and making informed choices to treat the athlete.
Definition of Evidence-Based Practice
Evidence-based practice (EBP) is a systematic method of reviewing the best evidence, combining it with the art of athletic training or your clinical expertise, and making informed choices to treat the athlete. Integrating EBP requires that you incorporate your clinical expertise, your athletes' values, and the best available clinical research into your practice. At its core, EBP is about focusing on athlete and clinical outcomes.
Clinical expertise is the culmination of your experience treating and providing care to athletes. It includes your personal values, preferences, experiences, and wisdom. Clinical expertise develops from hours of observation and trial and error and can result in a sense of pride and emotion. An AT who is vested in caring for an athlete can be uncomfortable being wrong about an assessment or a rehab and treatment program. Likewise, facing a challenging clinical case, deciding on a course of action, and having that action confirmed by other health care providers can be very rewarding. This is the reason many ATs are wary of incorporating new techniques that are unfamiliar to them. However, such a guarded approach can keep them from remembering that optimal athlete outcomes are the most important goal.
Patient values are the preconceived ideas and beliefs each athlete brings to the clinical setting. Athletes all have their own distinctive concerns that you must address. Often, the athlete arrives with a set feeling of trust in or distrust of ATs that has been cultivated from prior experience or social discussions. Furthermore, the athlete's values can be skewed by the appearance or functional use of the athletic training room and the amount of traffic. Some arrive lacking a clear understanding of the body and the healing process and may have set unrealistic time lines for their return to participation. This will influence their adherence to, or desire to complete, therapy. Treatment can be further confounded by comorbidities and the athlete's support network. Although the challenge is unique for each athlete, as an EBP clinician, you must consider all athletes' perspectives.
Clinical research is a scientific and systematic process that generates evidence through hypothesis testing and sound methodology. This is perhaps the most challenging portion of the EBP concept to incorporate. You may have limited experience analyzing peer-reviewed literature, and learning this process may thus appear daunting. Published studies should at a bare minimum introduce a research question, describe the methods used to assess subjects, outline the variables of interest, and explain the participant inclusion and exclusion criteria. The statistical analysis should align with the methods, and the authors should report only facts in the results section. The discussion or impression of the outcomes belongs in the conclusion or discussion section of an article. When assessing the quality of a clinical research article, ask the following questions:
- Were eligibility criteria specified?
- Were subjects randomly assigned to groups, and was assignment concealed, if appropriate?
- Were subjects similar at baseline?
- As appropriate, were subjects, clinicians, and assessors blinded to the treatment of participants?
- Was the completion rate of participants 85% or higher?
Eligibility criteria are a set of delimiters that detail who can be included in a study. These might include age limits, an association with a specific sport, a particular injury or medical condition, ethnicity, or sex. You need to know whether the study will be relevant to the athlete in your care.
Randomization is important in research because it helps to determine whether something truly made an impact, or whether it might have occurred by chance alone or as a result of a preconceived bias. An unethical researcher might place participants in a study group according to a notion that one group will respond more favorably to the treatment, with the purpose of showing support for that treatment. Likewise, whether participants were blinded (not aware that they were placed in a control or experimental group) is an important consideration when reviewing a study. To help the researcher or support a product or treatment they favor, participants who are not blinded may alter their efforts on tests or skew their reporting.
The similarity of participants is also crucial when determining a study's relevance. Participants should be similar in such factors as age, sport, and condition. It would be inappropriate to conclude that a new treatment works if those in the experimental group receiving the treatment were vastly different from those in the control group who did not receive a treatment. Furthermore, blinding the assessors to the participants' treatment limits the possibility of assessor bias.
Finally, researchers should report the participant completion rate to reveal the percentage of participants who dropped out of the study. A completion rate of less than 85% raises suspicion. Reviewers should question why so many participants dropped out. Was the study poorly organized? Was it painful? Was it too cumbersome or difficult?
With all of these aspects of research to consider, always keep the three components of EBP (clinical expertise, patient values, and clinical research) in mind. The bottom line is that EBP should always revolve around the athlete.
Five Steps of Evidence-Based Practice
To use evidence-based practice and retain an athlete focus, follow these five steps:
- Create aclinically relevant and searchable question concerning your athlete.
- Conduct a database search to find the best evidence.
- Critically appraise the articles or evidence for quality.
- Critically synthesize the evidence to choose and implement a treatment.
- Assess the outcomes by monitoring the athlete.
Let's take a closer look at each of these steps.
- Step 1: The first step is to start with the athlete, establish the clinical concern, and build a clinical question that centers on solving the issue or treating the condition. Further details on developing a clinical research question are presented in chapter 2. In the meantime, remember that the clinical question should be focused and searchable. A question that is too narrow in scope will return limited results. One that is too wide will return excessive information, limiting your ability to incorporate the content into a treatment plan. For example, if you are curious about the ability of ultrasound to facilitate increased tissue extensibility and are treating an athlete with ITB syndrome, searching for treat ITB with ultrasound may result in few or no results. It is too narrow and specific. If you search for ultrasound treatment, the results will most likely be too wide. A more appropriate approach might be to search for ultrasound AND connective tissue. This would return studies related to ultrasound and its effects on connective tissue. This will not be specific to ITB, but because ITB involves connective tissue, the treatment parameters and outcomes may be similar enough to draw upon for clinical use.
- Step 2: This step pertains to conducting the database search and looking for the best evidence related to the athlete's condition or injury. Chapter 2 explains how to conduct the search (including Boolean modifiers and search engines and databases), format the question, and conduct the first search. With practice, you will become adept at completing this in minimal time.
- Step 3: Appraise the articles for quality and for whether they are applicable to your athlete.1 Chapters 3 through 6 will help you learn how to do this. It is helpful to rate the quality and applicability of the studies you find on these two scales. A study may be of high quality but not applicable to your current athlete. For example, consider the effect of counterirritants. A research project that applied counterirritants to athletes with arthritis and found no improvement in range of motion at the fingers would not apply to an athlete's use of counterirritants to treat muscle spasms of the gastrocnemius. In this example, the study might be well designed but have no relevance to an athlete with a tight gastrocnemius. Conversely, a study of the effect of counterirritants on athletes with muscle spasms might be very applicable. However, if the article fails to report subjects' baseline measures, the inclusion or exclusion criteria, or the type of counterirritant used, this would indicate low quality. Without a certain amount of requisite data presented in the study, it would be impossible to devise a treatment plan.
- Step 4: This step involves critical processing and synthesizing. Integrating the evidence, your own clinical expertise and comfort in performing certain skills, and the values of the athlete will form the framework for your treatment plan.
- Step 5: This step returns to the athlete. You need to evaluate critically the progress and outcomes, reflecting on steps 1 through 4 and continually aiming to improve outcomes.
Learn more about Evidence-Based Practice in Athletic Training.
Is the Test Reliable?
The first term that is important for a consumer of research to understand is reliability. By strict definition, reliability refers to consistency in measures attained by people or instruments.
Reliability
The first term that is important for a consumer of research to understand is reliability. By strict definition, reliability refers to consistency in measures attained by people or instruments.1,2 Basically, does the tool you are using produce the same outcome each time you use it? An easy example is a pair of tape cutters. If every time you use the cutters to remove tape, you achieve a nice, clean cut, you would consider these cutters reliable. However, if sometimes they cut the tape and other times they do not, you would never know what outcome to expect; you would therefore consider these cutters unreliable. Reliability is important when you are trying to make a clinical decision. If you are performing an anterior drawer test on an ankle and one time you perform the test you feel laxity and the next time you fail to feel laxity, you will have difficulty making a decision. You may question whether the athlete changed, whether you applied different degrees of force, or whether the test itself is unreliable.
Reliability also relates to questionnaires. A well-structured item on a questionnaire should be easy to understand and not subject to misinterpretation. Let's look at an example of an item that uses a Likert scale of 1 to 4, with 1 being low or disagree and 4 being high or strongly agree. Here is the statement: Some of the most important tasks of an athletic trainer are administration duties and the bidding process. Although we can all agree that athletic trainers (ATs) often have some administrative duties and likely need to bid on purchases, this is a poor item that would probably be unreliable. One reader might believe that the most important duties of an AT relate to prevention and mark 1 for this item. Another reader might focus on the bidding portion of the statement and, believing that it's not that important, also mark 1. The researcher in this case has a value of 1 on two occasions but for very different reasons. However, an AT with administrative duties might focus on the administrative portion of the statement and rate this a 4. These varied scores based on individual interpretations of the statement would lead to low reliability and prohibit the researcher from drawing a valid conclusion.
Perhaps a better statement would look like this: Athletic trainers should have administrative skills. The scores for this may also vary, but they are less likely to vary because of misinterpretation. The variance in the scores would most likely be due to a variance in the opinions of the respondents. Thus, it is important to know that a study you are reviewing reports reliability values; without them, it's very difficult to trust the findings and make a judgment based on them.
You should be minimally familiar with several types of reliability measures as an evidence-based clinician. Presented here are four primary types: internal reliability, test - retest reliability, interrater reliability, and intrarater reliability.
Internal Reliability
Internal reliability is often an issue with surveys and questionnaires that employ a Likert scale system of values. The researcher wants to know whether the participant is responding to similar questions consistently. That is, did the person answer different questions about the same topic or construct in a similar manner?1 Let's assume that the researcher has a 30-item survey to find out athletes' opinions of the school's coaching staff. On item 4, which states that the school's coaching staff sincerely cares about athletes, an athlete might respond with a 1 (low or disagree). However, item 22 asks whether the athlete agrees that the coaches exhibit a caring nature; the athlete might score this item as a 4 (high or strongly agree). This is an example of two items asking about the same construct and receiving two very different responses, when the researcher would expect these answers to be similar. However, just because one person completes a survey in this fashion does not mean that the survey is unreliable. Only if a large numberof participants answer the survey in this fashion would it be considered low in internal reliability. This indicates that the sample population misunderstood the items, they deliberately answered in an odd fashion, or the items need to be dropped or reworded to increase consistency. This touches on one assumption of survey research, which is that participants answer honestly.
Test - Retest Reliability
The next form of reliability is test - retest reliability. As the name implies, it entails administering the test or assessment on more than one occasion. Test - retest reliability refers to the ability of an instrument or assessment to produce similar outcomes in the same group of people on multiple occasions in an amount of time that would not lead to a change in the measurement of interest.1,2 Let's assume that you use a dynamometer to assess the triceps extension strength in a group of athletes. The athletes are healthy college males between 18 and 24 years of age who are free of injury or disease and not participating in a strength training program. You would not expect their triceps strength to change in a week's time. It is safe to assume that the scores of the first test and those of the second test will be similar. In this case, large changes in strength measures are likely due to unreliable measures being detected by the dynamometer or test administrator error.
Not all test - retest measures are as easy to visualize or straightforward as the strength testing example. If you take a standardized test such as the SAT or GRE twice, a week apart, and you don't study between the tests, your scores will likely be similar. That's because taking a large standardized test covering topics you might not know won't teach you the topics. If a person scores low in algebra on the SAT, merely taking the SAT again won't improve his ability to apply algebra. This is not the case with certain types of cognitive assessment. A standard practice for the assessment of concussion often includes a computerized concussion assessment. These types of tests, however, result in athletes learning how to take the test, subsequently increasing their scores on follow-up trials. This is referred to as a practice effect.
Two of the most common cognitive-based tests in clinical athletic training practice are the ImPACT Concussion Management Software (ImPACT Applications, Pittsburgh, PA) and Concussion Sentinel (CogState Ltd, Victoria, Australia). These tests present with low to moderate test - retest reliability. This means that the scores of participants completing the assessments varied more than what is considered acceptable between tests, which diminishes their clinical relevance. Test - retest reliability is often reported in the literature as a value between 0 and 1, with value of 1 being more reliable or having fewer random effects that are unaccounted for in the results.3 The closer these reported values are to 1, the more reliable the assessments are, or the more trust you can put in them.
Interrater Reliability
Consensus between professionals on how to best treat an injury is important. It does, however, require that similar results or conditions be interpreted the same way. If you are an experienced AT reviewing injury reports upon returning to work after a few days off, you want to trust that other ATs completed certain orthopedic assessments the same way you would. More important, you hope that those tests would render the same outcome regardless of the person performing them. This is interrater reliability - that is, the agreement between clinicians who perform independent assessments of a condition.6 It is important to note that the consensus should be reached by both clinicians, but independently, without cognitive influence from each other. When the second AT is assessing an injury using the same assessment as the first, she should not know the outcome of the first.6 This ensures that the person performing the second assessment is not influenced by the outcome of the first assessment.
Also important when determining interrater reliability is that the athlete not recall the first assessment and respond differently to the second. This could be problematic when the assessment is on an injured body part and resulted in discomfort; the athlete may respond differently to the second assessment out of fear of re-creating the discomfort.
When reviewing articles and trying to determine the value of the results, remember that the higher the values for interrater reliability are, the more consistent the outcomes will be between assessors. Detailed examples are provided in subsequent chapters.
Intrarater Reliability
Intrarater reliability relates to the reproducibility of a clinical measure when performed by the same clinician on more than one occasion on the same athlete.2 The purpose of determining intrarater reliability is to assess whether the clinician is consistently using a reliable method of assessment. If the assessment method is not established as reliable, we would return to test - retest reliability to establish trust in the assessment or instrument (recall the tape cutters). An example would be assessing the range of motion of a knee. The bony landmarks used to measure knee flexion and the mechanics of the goniometer will not change from day 1 to day 2. However, range of motion assessments often vary when repeated by the same clinician. Imagine the difficulty in creating a treatment plan when on day 1 an athlete is assessed with 100° of flexion and on day 2 has only 90° of flexion. Variations in observed outcomes will confound the care provided to athletes.
Learn more about Evidence-Based Practice in Athletic Training.
Quickly and Easily Identify Low-Quality Articles Using CRAAP
Chapter 2 demonstrated how to perform a database search, and chapter 3 introduced ways to understand and evaluate the evidence. This chapter continues the discussion of evaluation methods by addressing how to appraise the diagnostic articles you have found.
Methods
Chapter 1 outlined the following five steps of evidence-based practice:
- Create aclinically relevant and searchable question concerning your athlete.
- Conduct a database search to find the best evidence.
- Critically appraise the articles or evidence for quality.
- Critically synthesize the evidence to choose and implement a treatment.
- Assess the outcomes by monitoring the athlete.
Chapter 2 demonstrated how to perform a database search, and chapter 3 introduced ways to understand and evaluate the evidence. This chapter continues the discussion of evaluation methods by addressing how to appraise the diagnostic articles you have found. Following are four steps to appraising diagnostic research:1
- Determine the applicability to your athlete.
- Assess the quality of the article.
- Review the results.
- Summarize the clinical bottom line.
As you gather your articles using a variety of search methods, you can apply a simple assessment to quickly and easily eliminate low-quality articles. Review any article based on the following five criteria (CRAAP):
- Currency. Is the information in need of being revised? The 2004 National Athletic Trainers' Association (NATA) position statement on the management of sport-related concussion allowed for same-day return to participation if symptoms resolved and the athlete passed a return-to-play test.3 The 2014 position statement update noted that our knowledge about how to treat concussions had significantly changed and ruled guidelines as recent as 10 years old obsolete.
- Relevance. Is the article written at a level appropriate for your needs, and do you understand it? If not, you may fail to reach a valid conclusion about the use of assessments in your situation. Has the information been peer reviewed, or is it an editorial or opinion piece? Editorials often have a bias and lack the scientific rigor required to make the information useful for sound clinical decisions.
- Authority. Is a sponsor involved that has a product to market? If so, read the information with great care. If an article is promoting an aquatics rehab program to aid in the recovery of knee surgery, but it was sponsored by a company that sells therapy pools, you need to suspect some potential bias. These types of conflicts of interest can be questionable.
- Accuracy. After completing your primary review, it is time to dig a little deeper and appraise the article. The Quality Assessment of Diagnostic Accuracy Studies (QUADAS) is a 14-question instrument to assist you in the appraisal (see the sidebar).1,4 The answer choices are yes, no, or uncertain; a greater number of yes answers is preferred.1 There is no set number of acceptable yes answers, but if any uncertain or no answers make you feel uncomfortable using the data to make a clinical decision, you may want to further research the topic and expand the number of articles you review.
- Purpose. Why is the information presented? Is it an advertisement? Is it funded by someone with a financial stake in selling something? If the answer to the preceding questions is yes, be sure to thoroughly review the academic content. The purpose of a quality article is to inform and provide outcomes, allowing readers to apply it as they see fit.
Each of the questions should be clearly addressed or answered in the article. If you are interested in learning whether a rehab program would be appropriate for high school athletes, the studies you review should have been conducted with people in the same age range. This is called generalizability - that is, can the results of a study be applied to another population based on similarities in the populations?
Authors should clearly tell you how they selected participants and why, as well as any reasons participants were excluded from participation. Did the author report issues with the data that could not be explained, and was the number of participants at the start and end of the study reported? If fewer completed the study than initiated it, was there an explanation for the dropout rate? If an author was researching the use of latex bandages to decrease swelling after ankle sprains and a volunteer participant had an allergy to latex, that person would need to leave the study. In this case, an author should report that 40 subjects volunteered to participate and were initially taking part in the study, and one was removed because of an exclusionary criterion. The author should not say that 39 people agreed to participate.
Similarly, if a new test (index test) is being assessed, it should be compared to an appropriate gold standard reference. If a researcher is testing a new axillary thermometer to establish whether it accurately assesses core body temperature, it should be compared to rectal thermometer measures. Comparing a new axillary thermometer to oral thermometers or others that are not scientifically accepted as the gold standard could introduce significant error, which would make the study results inappropriate to use in clinical practice.
Also important when appraising an article is assessing its methods section. Is it detailed enough that you could replicate it, if desired? You should be able to fully understand what was done during the study when reading the methods section.
The prevention of bias is also important during a study. If a participant was assessed using both an indexed and a referenced test, were the results interpreted separately? In other words, did the person completing the assessments know the results of the comparative assessment? A person who is aware of the other outcomes when completing an assessment creates a potential for biased reporting and a loss of objectivity, which is important in research.
Learn more about Evidence-Based Practice in Athletic Training.
What is Evidence-Based Practice Anyway?
Evidence-based practice (EBP) is a systematic method of reviewing the best evidence, combining it with the art of athletic training or your clinical expertise, and making informed choices to treat the athlete.
Definition of Evidence-Based Practice
Evidence-based practice (EBP) is a systematic method of reviewing the best evidence, combining it with the art of athletic training or your clinical expertise, and making informed choices to treat the athlete. Integrating EBP requires that you incorporate your clinical expertise, your athletes' values, and the best available clinical research into your practice. At its core, EBP is about focusing on athlete and clinical outcomes.
Clinical expertise is the culmination of your experience treating and providing care to athletes. It includes your personal values, preferences, experiences, and wisdom. Clinical expertise develops from hours of observation and trial and error and can result in a sense of pride and emotion. An AT who is vested in caring for an athlete can be uncomfortable being wrong about an assessment or a rehab and treatment program. Likewise, facing a challenging clinical case, deciding on a course of action, and having that action confirmed by other health care providers can be very rewarding. This is the reason many ATs are wary of incorporating new techniques that are unfamiliar to them. However, such a guarded approach can keep them from remembering that optimal athlete outcomes are the most important goal.
Patient values are the preconceived ideas and beliefs each athlete brings to the clinical setting. Athletes all have their own distinctive concerns that you must address. Often, the athlete arrives with a set feeling of trust in or distrust of ATs that has been cultivated from prior experience or social discussions. Furthermore, the athlete's values can be skewed by the appearance or functional use of the athletic training room and the amount of traffic. Some arrive lacking a clear understanding of the body and the healing process and may have set unrealistic time lines for their return to participation. This will influence their adherence to, or desire to complete, therapy. Treatment can be further confounded by comorbidities and the athlete's support network. Although the challenge is unique for each athlete, as an EBP clinician, you must consider all athletes' perspectives.
Clinical research is a scientific and systematic process that generates evidence through hypothesis testing and sound methodology. This is perhaps the most challenging portion of the EBP concept to incorporate. You may have limited experience analyzing peer-reviewed literature, and learning this process may thus appear daunting. Published studies should at a bare minimum introduce a research question, describe the methods used to assess subjects, outline the variables of interest, and explain the participant inclusion and exclusion criteria. The statistical analysis should align with the methods, and the authors should report only facts in the results section. The discussion or impression of the outcomes belongs in the conclusion or discussion section of an article. When assessing the quality of a clinical research article, ask the following questions:
- Were eligibility criteria specified?
- Were subjects randomly assigned to groups, and was assignment concealed, if appropriate?
- Were subjects similar at baseline?
- As appropriate, were subjects, clinicians, and assessors blinded to the treatment of participants?
- Was the completion rate of participants 85% or higher?
Eligibility criteria are a set of delimiters that detail who can be included in a study. These might include age limits, an association with a specific sport, a particular injury or medical condition, ethnicity, or sex. You need to know whether the study will be relevant to the athlete in your care.
Randomization is important in research because it helps to determine whether something truly made an impact, or whether it might have occurred by chance alone or as a result of a preconceived bias. An unethical researcher might place participants in a study group according to a notion that one group will respond more favorably to the treatment, with the purpose of showing support for that treatment. Likewise, whether participants were blinded (not aware that they were placed in a control or experimental group) is an important consideration when reviewing a study. To help the researcher or support a product or treatment they favor, participants who are not blinded may alter their efforts on tests or skew their reporting.
The similarity of participants is also crucial when determining a study's relevance. Participants should be similar in such factors as age, sport, and condition. It would be inappropriate to conclude that a new treatment works if those in the experimental group receiving the treatment were vastly different from those in the control group who did not receive a treatment. Furthermore, blinding the assessors to the participants' treatment limits the possibility of assessor bias.
Finally, researchers should report the participant completion rate to reveal the percentage of participants who dropped out of the study. A completion rate of less than 85% raises suspicion. Reviewers should question why so many participants dropped out. Was the study poorly organized? Was it painful? Was it too cumbersome or difficult?
With all of these aspects of research to consider, always keep the three components of EBP (clinical expertise, patient values, and clinical research) in mind. The bottom line is that EBP should always revolve around the athlete.
Five Steps of Evidence-Based Practice
To use evidence-based practice and retain an athlete focus, follow these five steps:
- Create aclinically relevant and searchable question concerning your athlete.
- Conduct a database search to find the best evidence.
- Critically appraise the articles or evidence for quality.
- Critically synthesize the evidence to choose and implement a treatment.
- Assess the outcomes by monitoring the athlete.
Let's take a closer look at each of these steps.
- Step 1: The first step is to start with the athlete, establish the clinical concern, and build a clinical question that centers on solving the issue or treating the condition. Further details on developing a clinical research question are presented in chapter 2. In the meantime, remember that the clinical question should be focused and searchable. A question that is too narrow in scope will return limited results. One that is too wide will return excessive information, limiting your ability to incorporate the content into a treatment plan. For example, if you are curious about the ability of ultrasound to facilitate increased tissue extensibility and are treating an athlete with ITB syndrome, searching for treat ITB with ultrasound may result in few or no results. It is too narrow and specific. If you search for ultrasound treatment, the results will most likely be too wide. A more appropriate approach might be to search for ultrasound AND connective tissue. This would return studies related to ultrasound and its effects on connective tissue. This will not be specific to ITB, but because ITB involves connective tissue, the treatment parameters and outcomes may be similar enough to draw upon for clinical use.
- Step 2: This step pertains to conducting the database search and looking for the best evidence related to the athlete's condition or injury. Chapter 2 explains how to conduct the search (including Boolean modifiers and search engines and databases), format the question, and conduct the first search. With practice, you will become adept at completing this in minimal time.
- Step 3: Appraise the articles for quality and for whether they are applicable to your athlete.1 Chapters 3 through 6 will help you learn how to do this. It is helpful to rate the quality and applicability of the studies you find on these two scales. A study may be of high quality but not applicable to your current athlete. For example, consider the effect of counterirritants. A research project that applied counterirritants to athletes with arthritis and found no improvement in range of motion at the fingers would not apply to an athlete's use of counterirritants to treat muscle spasms of the gastrocnemius. In this example, the study might be well designed but have no relevance to an athlete with a tight gastrocnemius. Conversely, a study of the effect of counterirritants on athletes with muscle spasms might be very applicable. However, if the article fails to report subjects' baseline measures, the inclusion or exclusion criteria, or the type of counterirritant used, this would indicate low quality. Without a certain amount of requisite data presented in the study, it would be impossible to devise a treatment plan.
- Step 4: This step involves critical processing and synthesizing. Integrating the evidence, your own clinical expertise and comfort in performing certain skills, and the values of the athlete will form the framework for your treatment plan.
- Step 5: This step returns to the athlete. You need to evaluate critically the progress and outcomes, reflecting on steps 1 through 4 and continually aiming to improve outcomes.
Learn more about Evidence-Based Practice in Athletic Training.
Is the Test Reliable?
The first term that is important for a consumer of research to understand is reliability. By strict definition, reliability refers to consistency in measures attained by people or instruments.
Reliability
The first term that is important for a consumer of research to understand is reliability. By strict definition, reliability refers to consistency in measures attained by people or instruments.1,2 Basically, does the tool you are using produce the same outcome each time you use it? An easy example is a pair of tape cutters. If every time you use the cutters to remove tape, you achieve a nice, clean cut, you would consider these cutters reliable. However, if sometimes they cut the tape and other times they do not, you would never know what outcome to expect; you would therefore consider these cutters unreliable. Reliability is important when you are trying to make a clinical decision. If you are performing an anterior drawer test on an ankle and one time you perform the test you feel laxity and the next time you fail to feel laxity, you will have difficulty making a decision. You may question whether the athlete changed, whether you applied different degrees of force, or whether the test itself is unreliable.
Reliability also relates to questionnaires. A well-structured item on a questionnaire should be easy to understand and not subject to misinterpretation. Let's look at an example of an item that uses a Likert scale of 1 to 4, with 1 being low or disagree and 4 being high or strongly agree. Here is the statement: Some of the most important tasks of an athletic trainer are administration duties and the bidding process. Although we can all agree that athletic trainers (ATs) often have some administrative duties and likely need to bid on purchases, this is a poor item that would probably be unreliable. One reader might believe that the most important duties of an AT relate to prevention and mark 1 for this item. Another reader might focus on the bidding portion of the statement and, believing that it's not that important, also mark 1. The researcher in this case has a value of 1 on two occasions but for very different reasons. However, an AT with administrative duties might focus on the administrative portion of the statement and rate this a 4. These varied scores based on individual interpretations of the statement would lead to low reliability and prohibit the researcher from drawing a valid conclusion.
Perhaps a better statement would look like this: Athletic trainers should have administrative skills. The scores for this may also vary, but they are less likely to vary because of misinterpretation. The variance in the scores would most likely be due to a variance in the opinions of the respondents. Thus, it is important to know that a study you are reviewing reports reliability values; without them, it's very difficult to trust the findings and make a judgment based on them.
You should be minimally familiar with several types of reliability measures as an evidence-based clinician. Presented here are four primary types: internal reliability, test - retest reliability, interrater reliability, and intrarater reliability.
Internal Reliability
Internal reliability is often an issue with surveys and questionnaires that employ a Likert scale system of values. The researcher wants to know whether the participant is responding to similar questions consistently. That is, did the person answer different questions about the same topic or construct in a similar manner?1 Let's assume that the researcher has a 30-item survey to find out athletes' opinions of the school's coaching staff. On item 4, which states that the school's coaching staff sincerely cares about athletes, an athlete might respond with a 1 (low or disagree). However, item 22 asks whether the athlete agrees that the coaches exhibit a caring nature; the athlete might score this item as a 4 (high or strongly agree). This is an example of two items asking about the same construct and receiving two very different responses, when the researcher would expect these answers to be similar. However, just because one person completes a survey in this fashion does not mean that the survey is unreliable. Only if a large numberof participants answer the survey in this fashion would it be considered low in internal reliability. This indicates that the sample population misunderstood the items, they deliberately answered in an odd fashion, or the items need to be dropped or reworded to increase consistency. This touches on one assumption of survey research, which is that participants answer honestly.
Test - Retest Reliability
The next form of reliability is test - retest reliability. As the name implies, it entails administering the test or assessment on more than one occasion. Test - retest reliability refers to the ability of an instrument or assessment to produce similar outcomes in the same group of people on multiple occasions in an amount of time that would not lead to a change in the measurement of interest.1,2 Let's assume that you use a dynamometer to assess the triceps extension strength in a group of athletes. The athletes are healthy college males between 18 and 24 years of age who are free of injury or disease and not participating in a strength training program. You would not expect their triceps strength to change in a week's time. It is safe to assume that the scores of the first test and those of the second test will be similar. In this case, large changes in strength measures are likely due to unreliable measures being detected by the dynamometer or test administrator error.
Not all test - retest measures are as easy to visualize or straightforward as the strength testing example. If you take a standardized test such as the SAT or GRE twice, a week apart, and you don't study between the tests, your scores will likely be similar. That's because taking a large standardized test covering topics you might not know won't teach you the topics. If a person scores low in algebra on the SAT, merely taking the SAT again won't improve his ability to apply algebra. This is not the case with certain types of cognitive assessment. A standard practice for the assessment of concussion often includes a computerized concussion assessment. These types of tests, however, result in athletes learning how to take the test, subsequently increasing their scores on follow-up trials. This is referred to as a practice effect.
Two of the most common cognitive-based tests in clinical athletic training practice are the ImPACT Concussion Management Software (ImPACT Applications, Pittsburgh, PA) and Concussion Sentinel (CogState Ltd, Victoria, Australia). These tests present with low to moderate test - retest reliability. This means that the scores of participants completing the assessments varied more than what is considered acceptable between tests, which diminishes their clinical relevance. Test - retest reliability is often reported in the literature as a value between 0 and 1, with value of 1 being more reliable or having fewer random effects that are unaccounted for in the results.3 The closer these reported values are to 1, the more reliable the assessments are, or the more trust you can put in them.
Interrater Reliability
Consensus between professionals on how to best treat an injury is important. It does, however, require that similar results or conditions be interpreted the same way. If you are an experienced AT reviewing injury reports upon returning to work after a few days off, you want to trust that other ATs completed certain orthopedic assessments the same way you would. More important, you hope that those tests would render the same outcome regardless of the person performing them. This is interrater reliability - that is, the agreement between clinicians who perform independent assessments of a condition.6 It is important to note that the consensus should be reached by both clinicians, but independently, without cognitive influence from each other. When the second AT is assessing an injury using the same assessment as the first, she should not know the outcome of the first.6 This ensures that the person performing the second assessment is not influenced by the outcome of the first assessment.
Also important when determining interrater reliability is that the athlete not recall the first assessment and respond differently to the second. This could be problematic when the assessment is on an injured body part and resulted in discomfort; the athlete may respond differently to the second assessment out of fear of re-creating the discomfort.
When reviewing articles and trying to determine the value of the results, remember that the higher the values for interrater reliability are, the more consistent the outcomes will be between assessors. Detailed examples are provided in subsequent chapters.
Intrarater Reliability
Intrarater reliability relates to the reproducibility of a clinical measure when performed by the same clinician on more than one occasion on the same athlete.2 The purpose of determining intrarater reliability is to assess whether the clinician is consistently using a reliable method of assessment. If the assessment method is not established as reliable, we would return to test - retest reliability to establish trust in the assessment or instrument (recall the tape cutters). An example would be assessing the range of motion of a knee. The bony landmarks used to measure knee flexion and the mechanics of the goniometer will not change from day 1 to day 2. However, range of motion assessments often vary when repeated by the same clinician. Imagine the difficulty in creating a treatment plan when on day 1 an athlete is assessed with 100° of flexion and on day 2 has only 90° of flexion. Variations in observed outcomes will confound the care provided to athletes.
Learn more about Evidence-Based Practice in Athletic Training.
Quickly and Easily Identify Low-Quality Articles Using CRAAP
Chapter 2 demonstrated how to perform a database search, and chapter 3 introduced ways to understand and evaluate the evidence. This chapter continues the discussion of evaluation methods by addressing how to appraise the diagnostic articles you have found.
Methods
Chapter 1 outlined the following five steps of evidence-based practice:
- Create aclinically relevant and searchable question concerning your athlete.
- Conduct a database search to find the best evidence.
- Critically appraise the articles or evidence for quality.
- Critically synthesize the evidence to choose and implement a treatment.
- Assess the outcomes by monitoring the athlete.
Chapter 2 demonstrated how to perform a database search, and chapter 3 introduced ways to understand and evaluate the evidence. This chapter continues the discussion of evaluation methods by addressing how to appraise the diagnostic articles you have found. Following are four steps to appraising diagnostic research:1
- Determine the applicability to your athlete.
- Assess the quality of the article.
- Review the results.
- Summarize the clinical bottom line.
As you gather your articles using a variety of search methods, you can apply a simple assessment to quickly and easily eliminate low-quality articles. Review any article based on the following five criteria (CRAAP):
- Currency. Is the information in need of being revised? The 2004 National Athletic Trainers' Association (NATA) position statement on the management of sport-related concussion allowed for same-day return to participation if symptoms resolved and the athlete passed a return-to-play test.3 The 2014 position statement update noted that our knowledge about how to treat concussions had significantly changed and ruled guidelines as recent as 10 years old obsolete.
- Relevance. Is the article written at a level appropriate for your needs, and do you understand it? If not, you may fail to reach a valid conclusion about the use of assessments in your situation. Has the information been peer reviewed, or is it an editorial or opinion piece? Editorials often have a bias and lack the scientific rigor required to make the information useful for sound clinical decisions.
- Authority. Is a sponsor involved that has a product to market? If so, read the information with great care. If an article is promoting an aquatics rehab program to aid in the recovery of knee surgery, but it was sponsored by a company that sells therapy pools, you need to suspect some potential bias. These types of conflicts of interest can be questionable.
- Accuracy. After completing your primary review, it is time to dig a little deeper and appraise the article. The Quality Assessment of Diagnostic Accuracy Studies (QUADAS) is a 14-question instrument to assist you in the appraisal (see the sidebar).1,4 The answer choices are yes, no, or uncertain; a greater number of yes answers is preferred.1 There is no set number of acceptable yes answers, but if any uncertain or no answers make you feel uncomfortable using the data to make a clinical decision, you may want to further research the topic and expand the number of articles you review.
- Purpose. Why is the information presented? Is it an advertisement? Is it funded by someone with a financial stake in selling something? If the answer to the preceding questions is yes, be sure to thoroughly review the academic content. The purpose of a quality article is to inform and provide outcomes, allowing readers to apply it as they see fit.
Each of the questions should be clearly addressed or answered in the article. If you are interested in learning whether a rehab program would be appropriate for high school athletes, the studies you review should have been conducted with people in the same age range. This is called generalizability - that is, can the results of a study be applied to another population based on similarities in the populations?
Authors should clearly tell you how they selected participants and why, as well as any reasons participants were excluded from participation. Did the author report issues with the data that could not be explained, and was the number of participants at the start and end of the study reported? If fewer completed the study than initiated it, was there an explanation for the dropout rate? If an author was researching the use of latex bandages to decrease swelling after ankle sprains and a volunteer participant had an allergy to latex, that person would need to leave the study. In this case, an author should report that 40 subjects volunteered to participate and were initially taking part in the study, and one was removed because of an exclusionary criterion. The author should not say that 39 people agreed to participate.
Similarly, if a new test (index test) is being assessed, it should be compared to an appropriate gold standard reference. If a researcher is testing a new axillary thermometer to establish whether it accurately assesses core body temperature, it should be compared to rectal thermometer measures. Comparing a new axillary thermometer to oral thermometers or others that are not scientifically accepted as the gold standard could introduce significant error, which would make the study results inappropriate to use in clinical practice.
Also important when appraising an article is assessing its methods section. Is it detailed enough that you could replicate it, if desired? You should be able to fully understand what was done during the study when reading the methods section.
The prevention of bias is also important during a study. If a participant was assessed using both an indexed and a referenced test, were the results interpreted separately? In other words, did the person completing the assessments know the results of the comparative assessment? A person who is aware of the other outcomes when completing an assessment creates a potential for biased reporting and a loss of objectivity, which is important in research.
Learn more about Evidence-Based Practice in Athletic Training.
What is Evidence-Based Practice Anyway?
Evidence-based practice (EBP) is a systematic method of reviewing the best evidence, combining it with the art of athletic training or your clinical expertise, and making informed choices to treat the athlete.
Definition of Evidence-Based Practice
Evidence-based practice (EBP) is a systematic method of reviewing the best evidence, combining it with the art of athletic training or your clinical expertise, and making informed choices to treat the athlete. Integrating EBP requires that you incorporate your clinical expertise, your athletes' values, and the best available clinical research into your practice. At its core, EBP is about focusing on athlete and clinical outcomes.
Clinical expertise is the culmination of your experience treating and providing care to athletes. It includes your personal values, preferences, experiences, and wisdom. Clinical expertise develops from hours of observation and trial and error and can result in a sense of pride and emotion. An AT who is vested in caring for an athlete can be uncomfortable being wrong about an assessment or a rehab and treatment program. Likewise, facing a challenging clinical case, deciding on a course of action, and having that action confirmed by other health care providers can be very rewarding. This is the reason many ATs are wary of incorporating new techniques that are unfamiliar to them. However, such a guarded approach can keep them from remembering that optimal athlete outcomes are the most important goal.
Patient values are the preconceived ideas and beliefs each athlete brings to the clinical setting. Athletes all have their own distinctive concerns that you must address. Often, the athlete arrives with a set feeling of trust in or distrust of ATs that has been cultivated from prior experience or social discussions. Furthermore, the athlete's values can be skewed by the appearance or functional use of the athletic training room and the amount of traffic. Some arrive lacking a clear understanding of the body and the healing process and may have set unrealistic time lines for their return to participation. This will influence their adherence to, or desire to complete, therapy. Treatment can be further confounded by comorbidities and the athlete's support network. Although the challenge is unique for each athlete, as an EBP clinician, you must consider all athletes' perspectives.
Clinical research is a scientific and systematic process that generates evidence through hypothesis testing and sound methodology. This is perhaps the most challenging portion of the EBP concept to incorporate. You may have limited experience analyzing peer-reviewed literature, and learning this process may thus appear daunting. Published studies should at a bare minimum introduce a research question, describe the methods used to assess subjects, outline the variables of interest, and explain the participant inclusion and exclusion criteria. The statistical analysis should align with the methods, and the authors should report only facts in the results section. The discussion or impression of the outcomes belongs in the conclusion or discussion section of an article. When assessing the quality of a clinical research article, ask the following questions:
- Were eligibility criteria specified?
- Were subjects randomly assigned to groups, and was assignment concealed, if appropriate?
- Were subjects similar at baseline?
- As appropriate, were subjects, clinicians, and assessors blinded to the treatment of participants?
- Was the completion rate of participants 85% or higher?
Eligibility criteria are a set of delimiters that detail who can be included in a study. These might include age limits, an association with a specific sport, a particular injury or medical condition, ethnicity, or sex. You need to know whether the study will be relevant to the athlete in your care.
Randomization is important in research because it helps to determine whether something truly made an impact, or whether it might have occurred by chance alone or as a result of a preconceived bias. An unethical researcher might place participants in a study group according to a notion that one group will respond more favorably to the treatment, with the purpose of showing support for that treatment. Likewise, whether participants were blinded (not aware that they were placed in a control or experimental group) is an important consideration when reviewing a study. To help the researcher or support a product or treatment they favor, participants who are not blinded may alter their efforts on tests or skew their reporting.
The similarity of participants is also crucial when determining a study's relevance. Participants should be similar in such factors as age, sport, and condition. It would be inappropriate to conclude that a new treatment works if those in the experimental group receiving the treatment were vastly different from those in the control group who did not receive a treatment. Furthermore, blinding the assessors to the participants' treatment limits the possibility of assessor bias.
Finally, researchers should report the participant completion rate to reveal the percentage of participants who dropped out of the study. A completion rate of less than 85% raises suspicion. Reviewers should question why so many participants dropped out. Was the study poorly organized? Was it painful? Was it too cumbersome or difficult?
With all of these aspects of research to consider, always keep the three components of EBP (clinical expertise, patient values, and clinical research) in mind. The bottom line is that EBP should always revolve around the athlete.
Five Steps of Evidence-Based Practice
To use evidence-based practice and retain an athlete focus, follow these five steps:
- Create aclinically relevant and searchable question concerning your athlete.
- Conduct a database search to find the best evidence.
- Critically appraise the articles or evidence for quality.
- Critically synthesize the evidence to choose and implement a treatment.
- Assess the outcomes by monitoring the athlete.
Let's take a closer look at each of these steps.
- Step 1: The first step is to start with the athlete, establish the clinical concern, and build a clinical question that centers on solving the issue or treating the condition. Further details on developing a clinical research question are presented in chapter 2. In the meantime, remember that the clinical question should be focused and searchable. A question that is too narrow in scope will return limited results. One that is too wide will return excessive information, limiting your ability to incorporate the content into a treatment plan. For example, if you are curious about the ability of ultrasound to facilitate increased tissue extensibility and are treating an athlete with ITB syndrome, searching for treat ITB with ultrasound may result in few or no results. It is too narrow and specific. If you search for ultrasound treatment, the results will most likely be too wide. A more appropriate approach might be to search for ultrasound AND connective tissue. This would return studies related to ultrasound and its effects on connective tissue. This will not be specific to ITB, but because ITB involves connective tissue, the treatment parameters and outcomes may be similar enough to draw upon for clinical use.
- Step 2: This step pertains to conducting the database search and looking for the best evidence related to the athlete's condition or injury. Chapter 2 explains how to conduct the search (including Boolean modifiers and search engines and databases), format the question, and conduct the first search. With practice, you will become adept at completing this in minimal time.
- Step 3: Appraise the articles for quality and for whether they are applicable to your athlete.1 Chapters 3 through 6 will help you learn how to do this. It is helpful to rate the quality and applicability of the studies you find on these two scales. A study may be of high quality but not applicable to your current athlete. For example, consider the effect of counterirritants. A research project that applied counterirritants to athletes with arthritis and found no improvement in range of motion at the fingers would not apply to an athlete's use of counterirritants to treat muscle spasms of the gastrocnemius. In this example, the study might be well designed but have no relevance to an athlete with a tight gastrocnemius. Conversely, a study of the effect of counterirritants on athletes with muscle spasms might be very applicable. However, if the article fails to report subjects' baseline measures, the inclusion or exclusion criteria, or the type of counterirritant used, this would indicate low quality. Without a certain amount of requisite data presented in the study, it would be impossible to devise a treatment plan.
- Step 4: This step involves critical processing and synthesizing. Integrating the evidence, your own clinical expertise and comfort in performing certain skills, and the values of the athlete will form the framework for your treatment plan.
- Step 5: This step returns to the athlete. You need to evaluate critically the progress and outcomes, reflecting on steps 1 through 4 and continually aiming to improve outcomes.
Learn more about Evidence-Based Practice in Athletic Training.
Is the Test Reliable?
The first term that is important for a consumer of research to understand is reliability. By strict definition, reliability refers to consistency in measures attained by people or instruments.
Reliability
The first term that is important for a consumer of research to understand is reliability. By strict definition, reliability refers to consistency in measures attained by people or instruments.1,2 Basically, does the tool you are using produce the same outcome each time you use it? An easy example is a pair of tape cutters. If every time you use the cutters to remove tape, you achieve a nice, clean cut, you would consider these cutters reliable. However, if sometimes they cut the tape and other times they do not, you would never know what outcome to expect; you would therefore consider these cutters unreliable. Reliability is important when you are trying to make a clinical decision. If you are performing an anterior drawer test on an ankle and one time you perform the test you feel laxity and the next time you fail to feel laxity, you will have difficulty making a decision. You may question whether the athlete changed, whether you applied different degrees of force, or whether the test itself is unreliable.
Reliability also relates to questionnaires. A well-structured item on a questionnaire should be easy to understand and not subject to misinterpretation. Let's look at an example of an item that uses a Likert scale of 1 to 4, with 1 being low or disagree and 4 being high or strongly agree. Here is the statement: Some of the most important tasks of an athletic trainer are administration duties and the bidding process. Although we can all agree that athletic trainers (ATs) often have some administrative duties and likely need to bid on purchases, this is a poor item that would probably be unreliable. One reader might believe that the most important duties of an AT relate to prevention and mark 1 for this item. Another reader might focus on the bidding portion of the statement and, believing that it's not that important, also mark 1. The researcher in this case has a value of 1 on two occasions but for very different reasons. However, an AT with administrative duties might focus on the administrative portion of the statement and rate this a 4. These varied scores based on individual interpretations of the statement would lead to low reliability and prohibit the researcher from drawing a valid conclusion.
Perhaps a better statement would look like this: Athletic trainers should have administrative skills. The scores for this may also vary, but they are less likely to vary because of misinterpretation. The variance in the scores would most likely be due to a variance in the opinions of the respondents. Thus, it is important to know that a study you are reviewing reports reliability values; without them, it's very difficult to trust the findings and make a judgment based on them.
You should be minimally familiar with several types of reliability measures as an evidence-based clinician. Presented here are four primary types: internal reliability, test - retest reliability, interrater reliability, and intrarater reliability.
Internal Reliability
Internal reliability is often an issue with surveys and questionnaires that employ a Likert scale system of values. The researcher wants to know whether the participant is responding to similar questions consistently. That is, did the person answer different questions about the same topic or construct in a similar manner?1 Let's assume that the researcher has a 30-item survey to find out athletes' opinions of the school's coaching staff. On item 4, which states that the school's coaching staff sincerely cares about athletes, an athlete might respond with a 1 (low or disagree). However, item 22 asks whether the athlete agrees that the coaches exhibit a caring nature; the athlete might score this item as a 4 (high or strongly agree). This is an example of two items asking about the same construct and receiving two very different responses, when the researcher would expect these answers to be similar. However, just because one person completes a survey in this fashion does not mean that the survey is unreliable. Only if a large numberof participants answer the survey in this fashion would it be considered low in internal reliability. This indicates that the sample population misunderstood the items, they deliberately answered in an odd fashion, or the items need to be dropped or reworded to increase consistency. This touches on one assumption of survey research, which is that participants answer honestly.
Test - Retest Reliability
The next form of reliability is test - retest reliability. As the name implies, it entails administering the test or assessment on more than one occasion. Test - retest reliability refers to the ability of an instrument or assessment to produce similar outcomes in the same group of people on multiple occasions in an amount of time that would not lead to a change in the measurement of interest.1,2 Let's assume that you use a dynamometer to assess the triceps extension strength in a group of athletes. The athletes are healthy college males between 18 and 24 years of age who are free of injury or disease and not participating in a strength training program. You would not expect their triceps strength to change in a week's time. It is safe to assume that the scores of the first test and those of the second test will be similar. In this case, large changes in strength measures are likely due to unreliable measures being detected by the dynamometer or test administrator error.
Not all test - retest measures are as easy to visualize or straightforward as the strength testing example. If you take a standardized test such as the SAT or GRE twice, a week apart, and you don't study between the tests, your scores will likely be similar. That's because taking a large standardized test covering topics you might not know won't teach you the topics. If a person scores low in algebra on the SAT, merely taking the SAT again won't improve his ability to apply algebra. This is not the case with certain types of cognitive assessment. A standard practice for the assessment of concussion often includes a computerized concussion assessment. These types of tests, however, result in athletes learning how to take the test, subsequently increasing their scores on follow-up trials. This is referred to as a practice effect.
Two of the most common cognitive-based tests in clinical athletic training practice are the ImPACT Concussion Management Software (ImPACT Applications, Pittsburgh, PA) and Concussion Sentinel (CogState Ltd, Victoria, Australia). These tests present with low to moderate test - retest reliability. This means that the scores of participants completing the assessments varied more than what is considered acceptable between tests, which diminishes their clinical relevance. Test - retest reliability is often reported in the literature as a value between 0 and 1, with value of 1 being more reliable or having fewer random effects that are unaccounted for in the results.3 The closer these reported values are to 1, the more reliable the assessments are, or the more trust you can put in them.
Interrater Reliability
Consensus between professionals on how to best treat an injury is important. It does, however, require that similar results or conditions be interpreted the same way. If you are an experienced AT reviewing injury reports upon returning to work after a few days off, you want to trust that other ATs completed certain orthopedic assessments the same way you would. More important, you hope that those tests would render the same outcome regardless of the person performing them. This is interrater reliability - that is, the agreement between clinicians who perform independent assessments of a condition.6 It is important to note that the consensus should be reached by both clinicians, but independently, without cognitive influence from each other. When the second AT is assessing an injury using the same assessment as the first, she should not know the outcome of the first.6 This ensures that the person performing the second assessment is not influenced by the outcome of the first assessment.
Also important when determining interrater reliability is that the athlete not recall the first assessment and respond differently to the second. This could be problematic when the assessment is on an injured body part and resulted in discomfort; the athlete may respond differently to the second assessment out of fear of re-creating the discomfort.
When reviewing articles and trying to determine the value of the results, remember that the higher the values for interrater reliability are, the more consistent the outcomes will be between assessors. Detailed examples are provided in subsequent chapters.
Intrarater Reliability
Intrarater reliability relates to the reproducibility of a clinical measure when performed by the same clinician on more than one occasion on the same athlete.2 The purpose of determining intrarater reliability is to assess whether the clinician is consistently using a reliable method of assessment. If the assessment method is not established as reliable, we would return to test - retest reliability to establish trust in the assessment or instrument (recall the tape cutters). An example would be assessing the range of motion of a knee. The bony landmarks used to measure knee flexion and the mechanics of the goniometer will not change from day 1 to day 2. However, range of motion assessments often vary when repeated by the same clinician. Imagine the difficulty in creating a treatment plan when on day 1 an athlete is assessed with 100° of flexion and on day 2 has only 90° of flexion. Variations in observed outcomes will confound the care provided to athletes.
Learn more about Evidence-Based Practice in Athletic Training.
Quickly and Easily Identify Low-Quality Articles Using CRAAP
Chapter 2 demonstrated how to perform a database search, and chapter 3 introduced ways to understand and evaluate the evidence. This chapter continues the discussion of evaluation methods by addressing how to appraise the diagnostic articles you have found.
Methods
Chapter 1 outlined the following five steps of evidence-based practice:
- Create aclinically relevant and searchable question concerning your athlete.
- Conduct a database search to find the best evidence.
- Critically appraise the articles or evidence for quality.
- Critically synthesize the evidence to choose and implement a treatment.
- Assess the outcomes by monitoring the athlete.
Chapter 2 demonstrated how to perform a database search, and chapter 3 introduced ways to understand and evaluate the evidence. This chapter continues the discussion of evaluation methods by addressing how to appraise the diagnostic articles you have found. Following are four steps to appraising diagnostic research:1
- Determine the applicability to your athlete.
- Assess the quality of the article.
- Review the results.
- Summarize the clinical bottom line.
As you gather your articles using a variety of search methods, you can apply a simple assessment to quickly and easily eliminate low-quality articles. Review any article based on the following five criteria (CRAAP):
- Currency. Is the information in need of being revised? The 2004 National Athletic Trainers' Association (NATA) position statement on the management of sport-related concussion allowed for same-day return to participation if symptoms resolved and the athlete passed a return-to-play test.3 The 2014 position statement update noted that our knowledge about how to treat concussions had significantly changed and ruled guidelines as recent as 10 years old obsolete.
- Relevance. Is the article written at a level appropriate for your needs, and do you understand it? If not, you may fail to reach a valid conclusion about the use of assessments in your situation. Has the information been peer reviewed, or is it an editorial or opinion piece? Editorials often have a bias and lack the scientific rigor required to make the information useful for sound clinical decisions.
- Authority. Is a sponsor involved that has a product to market? If so, read the information with great care. If an article is promoting an aquatics rehab program to aid in the recovery of knee surgery, but it was sponsored by a company that sells therapy pools, you need to suspect some potential bias. These types of conflicts of interest can be questionable.
- Accuracy. After completing your primary review, it is time to dig a little deeper and appraise the article. The Quality Assessment of Diagnostic Accuracy Studies (QUADAS) is a 14-question instrument to assist you in the appraisal (see the sidebar).1,4 The answer choices are yes, no, or uncertain; a greater number of yes answers is preferred.1 There is no set number of acceptable yes answers, but if any uncertain or no answers make you feel uncomfortable using the data to make a clinical decision, you may want to further research the topic and expand the number of articles you review.
- Purpose. Why is the information presented? Is it an advertisement? Is it funded by someone with a financial stake in selling something? If the answer to the preceding questions is yes, be sure to thoroughly review the academic content. The purpose of a quality article is to inform and provide outcomes, allowing readers to apply it as they see fit.
Each of the questions should be clearly addressed or answered in the article. If you are interested in learning whether a rehab program would be appropriate for high school athletes, the studies you review should have been conducted with people in the same age range. This is called generalizability - that is, can the results of a study be applied to another population based on similarities in the populations?
Authors should clearly tell you how they selected participants and why, as well as any reasons participants were excluded from participation. Did the author report issues with the data that could not be explained, and was the number of participants at the start and end of the study reported? If fewer completed the study than initiated it, was there an explanation for the dropout rate? If an author was researching the use of latex bandages to decrease swelling after ankle sprains and a volunteer participant had an allergy to latex, that person would need to leave the study. In this case, an author should report that 40 subjects volunteered to participate and were initially taking part in the study, and one was removed because of an exclusionary criterion. The author should not say that 39 people agreed to participate.
Similarly, if a new test (index test) is being assessed, it should be compared to an appropriate gold standard reference. If a researcher is testing a new axillary thermometer to establish whether it accurately assesses core body temperature, it should be compared to rectal thermometer measures. Comparing a new axillary thermometer to oral thermometers or others that are not scientifically accepted as the gold standard could introduce significant error, which would make the study results inappropriate to use in clinical practice.
Also important when appraising an article is assessing its methods section. Is it detailed enough that you could replicate it, if desired? You should be able to fully understand what was done during the study when reading the methods section.
The prevention of bias is also important during a study. If a participant was assessed using both an indexed and a referenced test, were the results interpreted separately? In other words, did the person completing the assessments know the results of the comparative assessment? A person who is aware of the other outcomes when completing an assessment creates a potential for biased reporting and a loss of objectivity, which is important in research.
Learn more about Evidence-Based Practice in Athletic Training.
What is Evidence-Based Practice Anyway?
Evidence-based practice (EBP) is a systematic method of reviewing the best evidence, combining it with the art of athletic training or your clinical expertise, and making informed choices to treat the athlete.
Definition of Evidence-Based Practice
Evidence-based practice (EBP) is a systematic method of reviewing the best evidence, combining it with the art of athletic training or your clinical expertise, and making informed choices to treat the athlete. Integrating EBP requires that you incorporate your clinical expertise, your athletes' values, and the best available clinical research into your practice. At its core, EBP is about focusing on athlete and clinical outcomes.
Clinical expertise is the culmination of your experience treating and providing care to athletes. It includes your personal values, preferences, experiences, and wisdom. Clinical expertise develops from hours of observation and trial and error and can result in a sense of pride and emotion. An AT who is vested in caring for an athlete can be uncomfortable being wrong about an assessment or a rehab and treatment program. Likewise, facing a challenging clinical case, deciding on a course of action, and having that action confirmed by other health care providers can be very rewarding. This is the reason many ATs are wary of incorporating new techniques that are unfamiliar to them. However, such a guarded approach can keep them from remembering that optimal athlete outcomes are the most important goal.
Patient values are the preconceived ideas and beliefs each athlete brings to the clinical setting. Athletes all have their own distinctive concerns that you must address. Often, the athlete arrives with a set feeling of trust in or distrust of ATs that has been cultivated from prior experience or social discussions. Furthermore, the athlete's values can be skewed by the appearance or functional use of the athletic training room and the amount of traffic. Some arrive lacking a clear understanding of the body and the healing process and may have set unrealistic time lines for their return to participation. This will influence their adherence to, or desire to complete, therapy. Treatment can be further confounded by comorbidities and the athlete's support network. Although the challenge is unique for each athlete, as an EBP clinician, you must consider all athletes' perspectives.
Clinical research is a scientific and systematic process that generates evidence through hypothesis testing and sound methodology. This is perhaps the most challenging portion of the EBP concept to incorporate. You may have limited experience analyzing peer-reviewed literature, and learning this process may thus appear daunting. Published studies should at a bare minimum introduce a research question, describe the methods used to assess subjects, outline the variables of interest, and explain the participant inclusion and exclusion criteria. The statistical analysis should align with the methods, and the authors should report only facts in the results section. The discussion or impression of the outcomes belongs in the conclusion or discussion section of an article. When assessing the quality of a clinical research article, ask the following questions:
- Were eligibility criteria specified?
- Were subjects randomly assigned to groups, and was assignment concealed, if appropriate?
- Were subjects similar at baseline?
- As appropriate, were subjects, clinicians, and assessors blinded to the treatment of participants?
- Was the completion rate of participants 85% or higher?
Eligibility criteria are a set of delimiters that detail who can be included in a study. These might include age limits, an association with a specific sport, a particular injury or medical condition, ethnicity, or sex. You need to know whether the study will be relevant to the athlete in your care.
Randomization is important in research because it helps to determine whether something truly made an impact, or whether it might have occurred by chance alone or as a result of a preconceived bias. An unethical researcher might place participants in a study group according to a notion that one group will respond more favorably to the treatment, with the purpose of showing support for that treatment. Likewise, whether participants were blinded (not aware that they were placed in a control or experimental group) is an important consideration when reviewing a study. To help the researcher or support a product or treatment they favor, participants who are not blinded may alter their efforts on tests or skew their reporting.
The similarity of participants is also crucial when determining a study's relevance. Participants should be similar in such factors as age, sport, and condition. It would be inappropriate to conclude that a new treatment works if those in the experimental group receiving the treatment were vastly different from those in the control group who did not receive a treatment. Furthermore, blinding the assessors to the participants' treatment limits the possibility of assessor bias.
Finally, researchers should report the participant completion rate to reveal the percentage of participants who dropped out of the study. A completion rate of less than 85% raises suspicion. Reviewers should question why so many participants dropped out. Was the study poorly organized? Was it painful? Was it too cumbersome or difficult?
With all of these aspects of research to consider, always keep the three components of EBP (clinical expertise, patient values, and clinical research) in mind. The bottom line is that EBP should always revolve around the athlete.
Five Steps of Evidence-Based Practice
To use evidence-based practice and retain an athlete focus, follow these five steps:
- Create aclinically relevant and searchable question concerning your athlete.
- Conduct a database search to find the best evidence.
- Critically appraise the articles or evidence for quality.
- Critically synthesize the evidence to choose and implement a treatment.
- Assess the outcomes by monitoring the athlete.
Let's take a closer look at each of these steps.
- Step 1: The first step is to start with the athlete, establish the clinical concern, and build a clinical question that centers on solving the issue or treating the condition. Further details on developing a clinical research question are presented in chapter 2. In the meantime, remember that the clinical question should be focused and searchable. A question that is too narrow in scope will return limited results. One that is too wide will return excessive information, limiting your ability to incorporate the content into a treatment plan. For example, if you are curious about the ability of ultrasound to facilitate increased tissue extensibility and are treating an athlete with ITB syndrome, searching for treat ITB with ultrasound may result in few or no results. It is too narrow and specific. If you search for ultrasound treatment, the results will most likely be too wide. A more appropriate approach might be to search for ultrasound AND connective tissue. This would return studies related to ultrasound and its effects on connective tissue. This will not be specific to ITB, but because ITB involves connective tissue, the treatment parameters and outcomes may be similar enough to draw upon for clinical use.
- Step 2: This step pertains to conducting the database search and looking for the best evidence related to the athlete's condition or injury. Chapter 2 explains how to conduct the search (including Boolean modifiers and search engines and databases), format the question, and conduct the first search. With practice, you will become adept at completing this in minimal time.
- Step 3: Appraise the articles for quality and for whether they are applicable to your athlete.1 Chapters 3 through 6 will help you learn how to do this. It is helpful to rate the quality and applicability of the studies you find on these two scales. A study may be of high quality but not applicable to your current athlete. For example, consider the effect of counterirritants. A research project that applied counterirritants to athletes with arthritis and found no improvement in range of motion at the fingers would not apply to an athlete's use of counterirritants to treat muscle spasms of the gastrocnemius. In this example, the study might be well designed but have no relevance to an athlete with a tight gastrocnemius. Conversely, a study of the effect of counterirritants on athletes with muscle spasms might be very applicable. However, if the article fails to report subjects' baseline measures, the inclusion or exclusion criteria, or the type of counterirritant used, this would indicate low quality. Without a certain amount of requisite data presented in the study, it would be impossible to devise a treatment plan.
- Step 4: This step involves critical processing and synthesizing. Integrating the evidence, your own clinical expertise and comfort in performing certain skills, and the values of the athlete will form the framework for your treatment plan.
- Step 5: This step returns to the athlete. You need to evaluate critically the progress and outcomes, reflecting on steps 1 through 4 and continually aiming to improve outcomes.
Learn more about Evidence-Based Practice in Athletic Training.
Is the Test Reliable?
The first term that is important for a consumer of research to understand is reliability. By strict definition, reliability refers to consistency in measures attained by people or instruments.
Reliability
The first term that is important for a consumer of research to understand is reliability. By strict definition, reliability refers to consistency in measures attained by people or instruments.1,2 Basically, does the tool you are using produce the same outcome each time you use it? An easy example is a pair of tape cutters. If every time you use the cutters to remove tape, you achieve a nice, clean cut, you would consider these cutters reliable. However, if sometimes they cut the tape and other times they do not, you would never know what outcome to expect; you would therefore consider these cutters unreliable. Reliability is important when you are trying to make a clinical decision. If you are performing an anterior drawer test on an ankle and one time you perform the test you feel laxity and the next time you fail to feel laxity, you will have difficulty making a decision. You may question whether the athlete changed, whether you applied different degrees of force, or whether the test itself is unreliable.
Reliability also relates to questionnaires. A well-structured item on a questionnaire should be easy to understand and not subject to misinterpretation. Let's look at an example of an item that uses a Likert scale of 1 to 4, with 1 being low or disagree and 4 being high or strongly agree. Here is the statement: Some of the most important tasks of an athletic trainer are administration duties and the bidding process. Although we can all agree that athletic trainers (ATs) often have some administrative duties and likely need to bid on purchases, this is a poor item that would probably be unreliable. One reader might believe that the most important duties of an AT relate to prevention and mark 1 for this item. Another reader might focus on the bidding portion of the statement and, believing that it's not that important, also mark 1. The researcher in this case has a value of 1 on two occasions but for very different reasons. However, an AT with administrative duties might focus on the administrative portion of the statement and rate this a 4. These varied scores based on individual interpretations of the statement would lead to low reliability and prohibit the researcher from drawing a valid conclusion.
Perhaps a better statement would look like this: Athletic trainers should have administrative skills. The scores for this may also vary, but they are less likely to vary because of misinterpretation. The variance in the scores would most likely be due to a variance in the opinions of the respondents. Thus, it is important to know that a study you are reviewing reports reliability values; without them, it's very difficult to trust the findings and make a judgment based on them.
You should be minimally familiar with several types of reliability measures as an evidence-based clinician. Presented here are four primary types: internal reliability, test - retest reliability, interrater reliability, and intrarater reliability.
Internal Reliability
Internal reliability is often an issue with surveys and questionnaires that employ a Likert scale system of values. The researcher wants to know whether the participant is responding to similar questions consistently. That is, did the person answer different questions about the same topic or construct in a similar manner?1 Let's assume that the researcher has a 30-item survey to find out athletes' opinions of the school's coaching staff. On item 4, which states that the school's coaching staff sincerely cares about athletes, an athlete might respond with a 1 (low or disagree). However, item 22 asks whether the athlete agrees that the coaches exhibit a caring nature; the athlete might score this item as a 4 (high or strongly agree). This is an example of two items asking about the same construct and receiving two very different responses, when the researcher would expect these answers to be similar. However, just because one person completes a survey in this fashion does not mean that the survey is unreliable. Only if a large numberof participants answer the survey in this fashion would it be considered low in internal reliability. This indicates that the sample population misunderstood the items, they deliberately answered in an odd fashion, or the items need to be dropped or reworded to increase consistency. This touches on one assumption of survey research, which is that participants answer honestly.
Test - Retest Reliability
The next form of reliability is test - retest reliability. As the name implies, it entails administering the test or assessment on more than one occasion. Test - retest reliability refers to the ability of an instrument or assessment to produce similar outcomes in the same group of people on multiple occasions in an amount of time that would not lead to a change in the measurement of interest.1,2 Let's assume that you use a dynamometer to assess the triceps extension strength in a group of athletes. The athletes are healthy college males between 18 and 24 years of age who are free of injury or disease and not participating in a strength training program. You would not expect their triceps strength to change in a week's time. It is safe to assume that the scores of the first test and those of the second test will be similar. In this case, large changes in strength measures are likely due to unreliable measures being detected by the dynamometer or test administrator error.
Not all test - retest measures are as easy to visualize or straightforward as the strength testing example. If you take a standardized test such as the SAT or GRE twice, a week apart, and you don't study between the tests, your scores will likely be similar. That's because taking a large standardized test covering topics you might not know won't teach you the topics. If a person scores low in algebra on the SAT, merely taking the SAT again won't improve his ability to apply algebra. This is not the case with certain types of cognitive assessment. A standard practice for the assessment of concussion often includes a computerized concussion assessment. These types of tests, however, result in athletes learning how to take the test, subsequently increasing their scores on follow-up trials. This is referred to as a practice effect.
Two of the most common cognitive-based tests in clinical athletic training practice are the ImPACT Concussion Management Software (ImPACT Applications, Pittsburgh, PA) and Concussion Sentinel (CogState Ltd, Victoria, Australia). These tests present with low to moderate test - retest reliability. This means that the scores of participants completing the assessments varied more than what is considered acceptable between tests, which diminishes their clinical relevance. Test - retest reliability is often reported in the literature as a value between 0 and 1, with value of 1 being more reliable or having fewer random effects that are unaccounted for in the results.3 The closer these reported values are to 1, the more reliable the assessments are, or the more trust you can put in them.
Interrater Reliability
Consensus between professionals on how to best treat an injury is important. It does, however, require that similar results or conditions be interpreted the same way. If you are an experienced AT reviewing injury reports upon returning to work after a few days off, you want to trust that other ATs completed certain orthopedic assessments the same way you would. More important, you hope that those tests would render the same outcome regardless of the person performing them. This is interrater reliability - that is, the agreement between clinicians who perform independent assessments of a condition.6 It is important to note that the consensus should be reached by both clinicians, but independently, without cognitive influence from each other. When the second AT is assessing an injury using the same assessment as the first, she should not know the outcome of the first.6 This ensures that the person performing the second assessment is not influenced by the outcome of the first assessment.
Also important when determining interrater reliability is that the athlete not recall the first assessment and respond differently to the second. This could be problematic when the assessment is on an injured body part and resulted in discomfort; the athlete may respond differently to the second assessment out of fear of re-creating the discomfort.
When reviewing articles and trying to determine the value of the results, remember that the higher the values for interrater reliability are, the more consistent the outcomes will be between assessors. Detailed examples are provided in subsequent chapters.
Intrarater Reliability
Intrarater reliability relates to the reproducibility of a clinical measure when performed by the same clinician on more than one occasion on the same athlete.2 The purpose of determining intrarater reliability is to assess whether the clinician is consistently using a reliable method of assessment. If the assessment method is not established as reliable, we would return to test - retest reliability to establish trust in the assessment or instrument (recall the tape cutters). An example would be assessing the range of motion of a knee. The bony landmarks used to measure knee flexion and the mechanics of the goniometer will not change from day 1 to day 2. However, range of motion assessments often vary when repeated by the same clinician. Imagine the difficulty in creating a treatment plan when on day 1 an athlete is assessed with 100° of flexion and on day 2 has only 90° of flexion. Variations in observed outcomes will confound the care provided to athletes.
Learn more about Evidence-Based Practice in Athletic Training.
Quickly and Easily Identify Low-Quality Articles Using CRAAP
Chapter 2 demonstrated how to perform a database search, and chapter 3 introduced ways to understand and evaluate the evidence. This chapter continues the discussion of evaluation methods by addressing how to appraise the diagnostic articles you have found.
Methods
Chapter 1 outlined the following five steps of evidence-based practice:
- Create aclinically relevant and searchable question concerning your athlete.
- Conduct a database search to find the best evidence.
- Critically appraise the articles or evidence for quality.
- Critically synthesize the evidence to choose and implement a treatment.
- Assess the outcomes by monitoring the athlete.
Chapter 2 demonstrated how to perform a database search, and chapter 3 introduced ways to understand and evaluate the evidence. This chapter continues the discussion of evaluation methods by addressing how to appraise the diagnostic articles you have found. Following are four steps to appraising diagnostic research:1
- Determine the applicability to your athlete.
- Assess the quality of the article.
- Review the results.
- Summarize the clinical bottom line.
As you gather your articles using a variety of search methods, you can apply a simple assessment to quickly and easily eliminate low-quality articles. Review any article based on the following five criteria (CRAAP):
- Currency. Is the information in need of being revised? The 2004 National Athletic Trainers' Association (NATA) position statement on the management of sport-related concussion allowed for same-day return to participation if symptoms resolved and the athlete passed a return-to-play test.3 The 2014 position statement update noted that our knowledge about how to treat concussions had significantly changed and ruled guidelines as recent as 10 years old obsolete.
- Relevance. Is the article written at a level appropriate for your needs, and do you understand it? If not, you may fail to reach a valid conclusion about the use of assessments in your situation. Has the information been peer reviewed, or is it an editorial or opinion piece? Editorials often have a bias and lack the scientific rigor required to make the information useful for sound clinical decisions.
- Authority. Is a sponsor involved that has a product to market? If so, read the information with great care. If an article is promoting an aquatics rehab program to aid in the recovery of knee surgery, but it was sponsored by a company that sells therapy pools, you need to suspect some potential bias. These types of conflicts of interest can be questionable.
- Accuracy. After completing your primary review, it is time to dig a little deeper and appraise the article. The Quality Assessment of Diagnostic Accuracy Studies (QUADAS) is a 14-question instrument to assist you in the appraisal (see the sidebar).1,4 The answer choices are yes, no, or uncertain; a greater number of yes answers is preferred.1 There is no set number of acceptable yes answers, but if any uncertain or no answers make you feel uncomfortable using the data to make a clinical decision, you may want to further research the topic and expand the number of articles you review.
- Purpose. Why is the information presented? Is it an advertisement? Is it funded by someone with a financial stake in selling something? If the answer to the preceding questions is yes, be sure to thoroughly review the academic content. The purpose of a quality article is to inform and provide outcomes, allowing readers to apply it as they see fit.
Each of the questions should be clearly addressed or answered in the article. If you are interested in learning whether a rehab program would be appropriate for high school athletes, the studies you review should have been conducted with people in the same age range. This is called generalizability - that is, can the results of a study be applied to another population based on similarities in the populations?
Authors should clearly tell you how they selected participants and why, as well as any reasons participants were excluded from participation. Did the author report issues with the data that could not be explained, and was the number of participants at the start and end of the study reported? If fewer completed the study than initiated it, was there an explanation for the dropout rate? If an author was researching the use of latex bandages to decrease swelling after ankle sprains and a volunteer participant had an allergy to latex, that person would need to leave the study. In this case, an author should report that 40 subjects volunteered to participate and were initially taking part in the study, and one was removed because of an exclusionary criterion. The author should not say that 39 people agreed to participate.
Similarly, if a new test (index test) is being assessed, it should be compared to an appropriate gold standard reference. If a researcher is testing a new axillary thermometer to establish whether it accurately assesses core body temperature, it should be compared to rectal thermometer measures. Comparing a new axillary thermometer to oral thermometers or others that are not scientifically accepted as the gold standard could introduce significant error, which would make the study results inappropriate to use in clinical practice.
Also important when appraising an article is assessing its methods section. Is it detailed enough that you could replicate it, if desired? You should be able to fully understand what was done during the study when reading the methods section.
The prevention of bias is also important during a study. If a participant was assessed using both an indexed and a referenced test, were the results interpreted separately? In other words, did the person completing the assessments know the results of the comparative assessment? A person who is aware of the other outcomes when completing an assessment creates a potential for biased reporting and a loss of objectivity, which is important in research.
Learn more about Evidence-Based Practice in Athletic Training.
Related Books
