- Home
- Kinesiology/Exercise and Sport Science
- Research Methods, Measurements, and Evaluation
- Recreation and Leisure
- Applied Research and Evaluation Methods in Recreation
Applied Research and Evaluation Methods in Recreation
200 Pages
The importance of research and evaluation in the park and recreation industry is becoming increasingly evident as agencies are being asked to justify their existence to a variety of stakeholders. By properly evaluating programs, recreation professionals can better understand the impact of their programs and, with data in hand, justify future investments. Applied Research and Evaluation Methods in Recreation helps students build the competencies they need in order to meet professional standards.
Applied Research and Evaluation Methods in Recreation is the only text that integrates research, evaluation, and basic statistical analysis and links these concepts directly to the recreation field. Using a logical format and accessible language, the book provides students with the foundational knowledge they need in order to move through the research process. They’ll explore the various types of research commonly used in the field (including qualitative, quantitative, and action research) and then learn about the steps involved in designing a project, from developing the research problem, reviewing literature, and identifying variables and hypotheses to defining the population to study, developing the instrumentation plan, and choosing the appropriate data-collection tools. Then they’ll move on to evaluation concepts, including internal validity, data analysis, and reporting methods.
The author’s approach to the material makes this textbook truly unique. Each chapter builds on the previous one to clearly explain the steps of the research process and show how all of the concepts are interrelated. Students will learn not only why the steps and concepts are important but also how they relate specifically to the field. Each chapter closes with features that help students apply what they’ve learned: a case study, exercises, and For the Investigator, an ongoing research project that uses a hypothetical recreation survey to give students realistic practice in recreation research. The text contains many other features that will help students better understand and apply the concepts they’ve learned. Chapter-opening scenarios offer examples of the kinds of research that students might be expected to undertake. Objectives highlight the key points of each chapter, and a glossary offers easy access to definitions of unfamiliar terms. Professional Perspectives sidebars describe real research projects from current professionals with an explanation of the project’s importance to the organization.
Applied Research and Evaluation Methods in Recreation also includes an online student resource (OSR) that provides additional opportunities for students to apply the information they’ve learned. Students will analyze two scholarly articles—one qualitative and one quantitative—throughout the course based on the concepts learned in each chapter. The OSR also contains a case study for each chapter that can be used in assessing students’ comprehension of the concepts covered or as the basis of in-class discussion and exercises to help students better understand and apply the material. In addition, the For the Investigator project appears in the OSR so students can easily download documents and data related to it.
Through the practical approach of Applied Research and Evaluation Methods in Recreation, students will gain the foundational knowledge they need in order too become confident in using their research and evaluation skills to meet the demands of their profession.
Chapter 1
Investigative Overview
Foundational Knowledge for Researchers
Benefits of Conducting Research and Evaluation
CAPRA Issues Related to Research and Evaluation
Benefits of Research and Evaluation for the Professional
Conclusion
Learning Aids
Chapter 2
The Research and Evaluation Process
Elements of the Scientific Method of Inquiry
Elements of the Evaluation Research Process
Comparing the Scientific Method of Inquiry Research and Evaluation Research
Steps of the Research Process
Quantitative and Qualitative Study Designs
Conclusion
Learning Aids
Chapter 3
The Research Problem
Defining the Research Problem
Purpose of the Study
Research Questions
Quantitative Research Designs
Qualitative Research Designs
Action Research
Conclusion
Learning Aids
Chapter 4
Review of Literature
What Is a Review of Literature?
Why Conduct a Review of Literature?
How to Find Literature
Synthesizing the Literature
Conclusion
Learning Aids
Chapter 5
Variables and Hypotheses in Study Designs
Relationships
Variables
Hypotheses
Measurement Scales
Conclusion
Learning Aids
Chapter 6
Ethics in Research
Protection From Harm
Confidentiality
Parental Permission
Avoiding Deception
Using All Data
Avoiding Coercion
Conclusion
Learning Aids
Chapter 7
Sampling
Defining the Population
Sampling Techniques
Sampling Issues
Conclusion
Learning Aids
Chapter 8
Instrumentation
Selecting a Leader
Defining the Population
Making an Instrumentation Plan
Kinds of Data
Conclusion
Learning Aids
Chapter 9
Data Collection Tools
Data Collection Tools
Instrument Selection
Instrument Considerations
Validity of the Instrument
Reliability of the Instrument
Conclusion
Learning Aids
Chapter 10
Internal Validity
Review of Validity
10 Threats to Internal Validity
Validity in Qualitative Designs
Reliability
Relationship Between Validity and Reliability
Conclusion
Learning Aids
Chapter 11
Data Analysis
Descriptive Statistics
Range
Standard Deviation
Additional Statistical Analysis
Qualitative Data Analysis
Conclusion
Learning Aids
Chapter 12
The Research Report
Executive Summary
Purpose of the Study
Background
Methods
Results
Recommendations
Conclusion
Learning Aids
Diane C. Blankenship, EdD, is an associate professor at Frostburg State University in Frostburg, Maryland, where she teaches courses in research methods in the recreation and parks management program. She also serves on the program’s graduate thesis and project committee.
Blankenship has extensive and varied experience, having worked as a professional Boy Scout and a recreation therapist; in residential camps, aquatic facilities, recreation centers, and outdoor recreation facilities; and for the Department of the Army in Germany.
Blankenship has used her skills in recreation and leisure research methods to assist local and state organizations in the evaluation of their programs and facilities and in strategic planning. She is a member of the Maryland Recreation and Parks Association (MRPA) and has served numerous times on the conference committee and as the coordinator of data analysis for all conference evaluations. In addition, she serves as a site visitor for the Commission for Accreditation of Park and Recreation Agencies (CAPRA) and as a recreation commissioner for the City of Frostburg.
In her free time, Blankenship enjoys walking rail-trails, knitting, and traveling with her daughter to Native American reservations. She and her husband, Calvin, and their daughter reside in Frostburg.
The importance of research and evaluation in recreation
Parks and recreation departments provide a wide variety of programs and services to their local communities. But are these programs accomplishing what they are intended to do for the individual, the organization, or the community?
Foundational Knowledge for Researchers
To become comfortable with research and evaluation, you first need to understand the following:
- The mission and purpose of parks and recreation
- The need for research and evaluation
- How research and evaluation assist an organization
- The expectations for evaluation and research within the industry related to professional certification and agency accreditation
Mission and Purpose of Recreation Agencies
In the United States, the field of recreation and parks has deep roots grounded within societal needs, community needs, and individual needs (see figure 1.1). As you probably remember, the original needs of the community emerged during the Industrial Revolution. With the population shifting from rural areas to urban centers, children needed a safe place to play and socialize. The scope of facilities and services expanded from that point to include providing public parks, public recreation facilities, and instructional classes. The focus of this expansion was to (1) provide places for people to play, (2) offer programs that were educational and considered “wholesome” for the participant, and (3) provide recreational opportunities for individuals, families, and seniors within local communities.
Every park and recreation agency is challenged by—and exists to address—social concerns that become the needs within the community. For example, many American communities are concerned about gang activity, risk-taking behavior of youth, single-parent households, drug and alcohol abuse, and the growing need for sport fields and other public open spaces. Obesity is a primary example of a social concern that park and recreation programs are well positioned to address. This epidemic is prevalent in children, teenagers, and adults within the United States. Researchers predict that many members of the current generation of children will die before their parents because of factors related to obesity (high cholesterol, high blood pressure, diabetes, and a lack of exercise). The National Recreation and Park Association has partnered with other organizations to provide initiatives that address obesity factors, such as Hearts N' Parks, Step Up to Health, and Teens Outside. Each initiative is a community mobilization model that is designed to assist local communities in planning, marketing, and promoting healthy lifestyles related to diet and exercise. In addition, park and recreation agencies provide a wide variety of health and fitness programs, such as swimming lessons, open swim, aerobic classes, sport skill classes, movement classes for preschoolers, weight rooms for working out, and the simple reminder to get outside and play an hour a day. Healthy lifestyles go beyond physical fitness to emotional and psychological wellness. Park and recreation agencies also provide a wide variety of art classes, dance lessons, and concerts. All these kinds of programs and services work toward a healthier community.
Parks and recreation departments provide a wide variety of programs and services to their local communities. But are these programs accomplishing what they are intended to do for the individual, the organization, or the community?
The Need for Research and Evaluation
Every park and recreation agency needs to answer a number of questions about its programs, services, and operations which are noted in figure 1.2. These questions include the following:
- Is the agency meeting the objectives of the programs?
- Is the agency meeting the needs of the customer?
- Is the agency financially stable?
These are only a small sample of the questions that a park and recreation agency needs to answer. How does a park and recreation agency go about answering these questions? The best way is to use the process of research and evaluation. Let's explore these three questions further in order to gain an understanding of the importance of evaluation and research in providing the answers.
Is the Agency Meeting the Objectives of the Programs? All programs provided by park and recreation agencies have a purpose or an objective related to the outcomes for the participants. For example, the objective of a beginning swimming class is to have the participants move through water adjustment skills and learn a basic stroke on their front and back. For the organization, the objective for swimming classes is to have more people learn to swim. This can help reduce the number of rescues or deaths at the organization's aquatic facilities. In addition, the programs should cover costs associated with the programs and generate revenues for the agency. The agency must always consider the cost and revenues generated from the programs. For the community, the objective of swimming classes could be to have a variety of opportunities available for family members to learn to swim and to enjoy the aquatic facilities year-round. Many park and recreation agencies have indoor facilities, outdoor facilities, or both. Swimming is also a physical activity that people can enjoy throughout their lifetime, either in formal classes or individually. Park and recreation agencies must work to gather the information they need to determine if the objectives of the program are being met and whether these are class objectives, agency objectives, or objectives for the community. The research and evaluation process is the primary method for gathering this information and making this assessment. This process gives recreation professionals specific information and evidence to support or negate the notion that their agencies are achieving their objectives. Without research and evaluation, professionals might have only a vague sense of whether objectives are being met, or they may have no idea at all.
Is the Agency Meeting the Needs of the Customer? What are the customer needs and expectations in relation to recreational activities? For swimming classes, the needs of the customers are generally to have a safe environment for the class and to have the class conducted by a qualified instructor. The needs of the participants' parents are to have their children learn to swim in a safe environment so that the children can be safer at pools, lakes, and beaches during the summer. The swimming classes provide one way to meet these needs. Additionally, customers have expectations about their experience or their children's experience in the program. These expectations may be associated with the registration process, how to get to the facility, cleanliness of the locker rooms, cleanliness of the facility, and the quality of the instructor. The agency must gather information to determine if the needs and expectations of the customers are being met during the program.
Is the Agency Financially Stable? Historically, park and recreation departments received a majority of their funding for operations from local taxes. The agency was considered a type of social service that needed tax-based funding for the annual operations. This perception regarding park and recreation agencies has shifted from a social service model to a business-based model. Today, park and recreation agencies are expected to operate as a business. This has resulted in departments receiving very little tax-based funding for operations and has increased the need to generate revenues from programs and services. The agency gathers financial information on a regular basis to determine if the agency is generating sufficient funds to meet the financial needs of the organization. These research activities provide valuable information to assist the managers in making decisions related to the future operations of the agency. The information gained through research and evaluation helps managers decide which programs and services need to be revised, which ones should be added, and which ones should be phased out to ensure that the agency is financially stable. The decision on whether to add or phase out a program cannot be properly made unless the managers have current information about new community needs or needs within the community that no longer exist. The best method for gathering this information is through research and evaluation.
For agencies to answer these fundamental questions concerning their programs, services, and financial operations, research and evaluation have become a necessary activity. Historically, park and recreation agencies did not regularly evaluate program outcomes for the participants. In the past, evaluation efforts centered on the agency's need to know “the numbers”—the number of participants, the hours of the programs, the revenues generated by the programs, and the expenses associated with the programs. This type of evaluation misses the essence and purpose of the park and recreation agency (i.e., meeting the program objectives for the participants and meeting the needs of the customers and community). Evaluating and researching participant outcomes, needs, and expectations are critically important to the future of every park and recreation agency. Agencies that regularly conduct evaluation and research projects gain many benefits that help to sustain the operation of the agency.
This is an excerpt from Applied Research and Evaluation Methods in Recreation.
Steps of the research process
Scientific research involves a systematic process that focuses on being objective and gathering a multitude of information for analysis so that the researcher can come to a conclusion.
Scientific research involves a systematic process that focuses on being objective and gathering a multitude of information for analysis so that the researcher can come to a conclusion. This process is used in all research and evaluation projects, regardless of the research method (scientific method of inquiry, evaluation research, or action research). The process focuses on testing hunches or ideas in a park and recreation setting through a systematic process. In this process, the study is documented in such a way that another individual can conduct the same study again. This is referred to as replicating the study. Any research done without documenting the study so that others can review the process and results is not an investigation using the scientific research process. The scientific research process is a multiple-step process where the steps are interlinked with the other steps in the process. If changes are made in one step of the process, the researcher must review all the other steps to ensure that the changes are reflected throughout the process. Parks and recreation professionals are often involved in conducting research or evaluation projects within the agency. These professionals need to understand the eight steps of the research process as they apply to conducting a study. Table 2.4 lists the steps of the research process and provides an example of each step for a sample research study.
Step 1: Identify the Problem
The first step in the process is to identify a problem or develop a research question. The research problem may be something the agency identifies as a problem, some knowledge or information that is needed by the agency, or the desire to identify a recreation trend nationally. In the example in table 2.4, the problem that the agency has identified is childhood obesity, which is a local problem and concern within the community. This serves as the focus of the study.
Step 2: Review the Literature
Now that the problem has been identified, the researcher must learn more about the topic under investigation. To do this, the researcher must review the literature related to the research problem. This step provides foundational knowledge about the problem area. The review of literature also educates the researcher about what studies have been conducted in the past, how these studies were conducted, and the conclusions in the problem area. In the obesity study, the review of literature enables the programmer to discover horrifying statistics related to the long-term effects of childhood obesity in terms of health issues, death rates, and projected medical costs. In addition, the programmer finds several articles and information from the Centers for Disease Control and Prevention that describe the benefits of walking 10,000 steps a day. The information discovered during this step helps the programmer fully understand the magnitude of the problem, recognize the future consequences of obesity, and identify a strategy to combat obesity (i.e., walking).
Step 3: Clarify the Problem
Many times the initial problem identified in the first step of the process is too large or broad in scope. In step 3 of the process, the researcher clarifies the problem and narrows the scope of the study. This can only be done after the literature has been reviewed. The knowledge gained through the review of literature guides the researcher in clarifying and narrowing the research project. In the example, the programmer has identified childhood obesity as the problem and the purpose of the study. This topic is very broad and could be studied based on genetics, family environment, diet, exercise, self-confidence, leisure activities, or health issues. All of these areas cannot be investigated in a single study; therefore, the problem and purpose of the study must be more clearly defined. The programmer has decided that the purpose of the study is to determine if walking 10,000 steps a day for three days a week will improve the individual's health. This purpose is more narrowly focused and researchable than the original problem.
Step 4: Clearly Define Terms and Concepts
Terms and concepts are words or phrases used in the purpose statement of the study or the description of the study. These items need to be specifically defined as they apply to the study. Terms or concepts often have different definitions depending on who is reading the study. To minimize confusion about what the terms and phrases mean, the researcher must specifically define them for the study. In the obesity study, the concept of “individual's health” can be defined in hundreds of ways, such as physical, mental, emotional, or spiritual health. For this study, the individual's health is defined as physical health. The concept of physical health may also be defined and measured in many ways. In this case, the programmer decides to more narrowly define “individual health” to refer to the areas of weight, percentage of body fat, and cholesterol. By defining the terms or concepts more narrowly, the scope of the study is more manageable for the programmer, making it easier to collect the necessary data for the study. This also makes the concepts more understandable to the reader.
Step 5: Define the Population
Research projects can focus on a specific group of people, facilities, park development, employee evaluations, programs, financial status, marketing efforts, or the integration of technology into the operations. For example, if a researcher wants to examine a specific group of people in the community, the study could examine a specific age group, males or females, people living in a specific geographic area, or a specific ethnic group. Literally thousands of options are available to the researcher to specifically identify the group to study. The research problem and the purpose of the study assist the researcher in identifying the group to involve in the study. In research terms, the group to involve in the study is always called the population. Defining the population assists the researcher in several ways. First, it narrows the scope of the study from a very large population to one that is manageable. Second, the population identifies the group that the researcher's efforts will be focused on within the study. This helps ensure that the researcher stays on the right path during the study. Finally, by defining the population, the researcher identifies the group that the results will apply to at the conclusion of the study. In the example in table 2.4, the programmer has identified the population of the study as children ages 10 to 12 years. This narrower population makes the study more manageable in terms of time and resources.
Step 6: Develop the Instrumentation Plan
The plan for the study is referred to as the instrumentation plan. The instrumentation plan serves as the road map for the entire study, specifying who will participate in the study; how, when, and where data will be collected; and the content of the program. This plan is composed of numerous decisions and considerations that are addressed in chapter 8 of this text. In the obesity study, the researcher has decided to have the children participate in a walking program for six months. The group of participants is called the sample, which is a smaller group selected from the population specified for the study. The study cannot possibly include every 10- to 12-year-old child in the community, so a smaller group is used to represent the population. The researcher develops the plan for the walking program, indicating what data will be collected, when and how the data will be collected, who will collect the data, and how the data will be analyzed. The instrumentation plan specifies all the steps that must be completed for the study. This ensures that the programmer has carefully thought through all these decisions and that she provides a step-by-step plan to be followed in the study.
Step 7: Collect Data
Once the instrumentation plan is completed, the actual study begins with the collection of data. The collection of data is a critical step in providing the information needed to answer the research question. Every study includes the collection of some type of data—whether it is from the literature or from subjects—to answer the research question. Data can be collected in the form of words on a survey, with a questionnaire, through observations, or from the literature. In the obesity study, the programmers will be collecting data on the defined variables: weight, percentage of body fat, cholesterol levels, and the number of days the person walked a total of 10,000 steps during the class.
The researcher collects these data at the first session and at the last session of the program. These two sets of data are necessary to determine the effect of the walking program on weight, body fat, and cholesterol level. Once the data are collected on the variables, the researcher is ready to move to the final step of the process, which is the data analysis.
Step 8: Analyze the Data
All the time, effort, and resources dedicated to steps 1 through 7 of the research process culminate in this final step. The researcher finally has data to analyze so that the research question can be answered. In the instrumentation plan, the researcher specified how the data will be analyzed. The researcher now analyzes the data according to the plan. The results of this analysis are then reviewed and summarized in a manner directly related to the research questions. In the obesity study, the researcher compares the measurements of weight, percentage of body fat, and cholesterol that were taken at the first meeting of the subjects to the measurements of the same variables at the final program session. These two sets of data will be analyzed to determine if there was a difference between the first measurement and the second measurement for each individual in the program. Then, the data will be analyzed to determine if the differences are statistically significant. If the differences are statistically significant, the study validates the theory that was the focus of the study. The results of the study also provide valuable information about one strategy to combat childhood obesity in the community.
As you have probably concluded, conducting studies using the eight steps of the scientific research process requires you to dedicate time and effort to the planning process. You cannot conduct a study using the scientific research process when time is limited or the study is done at the last minute. Researchers who do this conduct studies that result in either false conclusions or conclusions that are not of any value to the organization.
This is an excerpt from Applied Research and Evaluation Methods in Recreation.
Creating instrumentation plans for research studies
The instrumentation plan helps guide the progress of the study to the ultimate goal of gathering data and formulating conclusions to answer the research question.
The instrumentation plan is composed of a number of decisions that need to be made before beginning the study. These decisions are made to determine
what data are needed to answer the research questions,
- how to gather the data,
- when to gather the data,
- where to gather the data, and
- how to analyze the data.
These decisions must be made as part of the instrumentation plan for the study. They help guide the progress of the study to the ultimate goal of gathering data and formulating conclusions to answer the research question.
Selecting Data
The previous decisions help define what data need to be gathered and why the data are important. What is data? Data refers to the information that is gathered to answer the research question. Data can be numbers, words, or actual objects, such as photos, articles, or video. The instrumentation process defines what data need to be collected and the timing of the data collection process.
What data are needed if the researcher is examining the relationship between taking swimming lessons and purchasing a summer session pass? The researcher needs the records of swimming lesson participants and season pass holders. Examining any other type of data, such as instructor evaluations, is a waste of time if it does not focus on answering the research question. If the researcher is examining the factors that lead a person to purchase a summer pass for the pool, interview data are needed to fully understand the factors in the decision-making process of the customer. By using the interview process, the researcher can conduct a two-way conversation in order to explore the factors that lead to purchasing the season pass. This type of data is very comprehensive in exploring the decision-making process, and it is more effective than looking at only one or two variables. In the example of the fitness center evaluating the personal trainers, a written survey will provide the data needed to evaluate the performances of the trainers. By identifying the specific type of data needed to answer the research question, the researchers' efforts are properly focused. Table 8.3 summarizes what data are needed in relation to the previous decisions in the instrumentation plan.
Gathering Data
The next natural question is, how will the data be collected? To answer this question, the researcher needs to identify whether the study is an evaluation, a quantitative study, or a qualitative study. This helps determine how to collect data. What instrument will be used to collect data? The researcher has a variety of options that may be used as a data collection instrument, such as surveys, interviews, observations, or rating instruments. Table 8.4 summarizes the decision of how to gather data along with the previous pieces of the instrumentation plan.
Determining When to Collect Data
Once the researcher decides what data to collect and how to collect them, the researcher must determine when to collect the data, where to collect them, and who should collect them. The results of the study can be influenced by the time that data are gathered, where the data are gathered, and who gathers the data. Studies that only use literature or the records of the agency are not influenced by these decisions. This is the case with the first research question concerning the relationship between taking swimming lessons and purchasing a summer season pass. Once the majority of season passes have been sold, the data should be gathered at the beginning of the summer.
This is not the case when data are being gathered from human subjects. Timing considerations that must be addressed are the month, the day of the week, and the time of day that the data are collected. In the example of identifying what factors lead a person to purchase a season pass to the pool, the interview should be conducted as soon as possible after the purchase of the pass. If the researchers wait weeks or months after the purchase, the subjects may not recall what factors led them to make the purchase. This interview could be done over the phone or in a small group.
For the fitness center evaluating the personal trainers, a specific time frame is necessary in order to ensure a high response rate. The manager decides to give the clients an evaluation form to complete after their last session with their personal trainer (before they leave the facility). This is a better plan than mailing the surveys to the clients' home weeks after the clients' last session with their trainer. The timing of data collection can affect the quality of the data received from the subjects. Table 8.5 presents this information, along with the previous decisions in the instrumentation plan.
Determining Where to Collect Data
The place that data are collected and the person collecting data must be specifically defined in the instrumentation plan. The best plan is to standardize the place and person for the data collection. This standardization helps enhance the truthfulness and validity of the data. The subjects should be in an environment where they feel at ease so that they will answer questions honestly. These two considerations are not an issue for the first research question because the data are coming from the documents of the agency and not from individuals.
In the case of identifying the factors that lead to purchasing a season pass for the pool, the interviews will be conducted by telephone. The people conducting the interviews should be trained in how to conduct an interview and how to record the information accurately. Each person being interviewed will be at home while participating in the interview, which is a comfortable environment for that individual.
In the example of evaluating personal trainers, having the clients' trainer administer the survey in the gym will most likely provide invalid data. The ideal situation would be to have one staff person provide the clients with a quiet room to complete the survey. Then the subject should return the survey to the same staff member who gave the survey to the subject. This standardizes where the data are collected and by whom. Table 8.6 provides a summary of all the instrument pieces and provides examples of each decision that needs to be made in developing the instrumentation plan for a study.
Analyzing Data
The next planning decision to make is how to analyze the data and what to do with the information once the analysis is completed. The type of analysis used with data is determined by whether the data are quantitative or qualitative data. To analyze quantitative data, some type of statistical analysis is used to provide the results. The type of statistical analysis used with data must be thought out and documented in the instrumentation plan.
Some of the most frequently used options for statistical analysis will be covered later in this text. Qualitative data are analyzed through a coding process that identifies themes; these themes become the foundation for the conclusions of the study. This type of data analysis will also be covered later in this text. The results of the data analysis should be summarized and presented in a report to supervisors and other parties for review. Research and evaluation efforts yield a wealth of information that can be used to educate commissioners, city councils, customers, and other decision makers. Studies that document outcomes of the programs and benefits to the community serve as a powerful tool that allows the agency to document its benefits to the community through facts and data, not perceptions and speculations. In today's cost-conscious society, the agency must prove that it is contributing to the quality of life of the community and must document the outcomes from the programs. This type of documentation and evidence is also a requirement for CAPRA accredited agencies and represents an ongoing evaluation process within the agency. Table 8.7 summarizes the key decisions that need to be made in the development of the instrumentation plan.
This is an excerpt from Applied Research and Evaluation Methods in Recreation.
The importance of research and evaluation in recreation
Parks and recreation departments provide a wide variety of programs and services to their local communities. But are these programs accomplishing what they are intended to do for the individual, the organization, or the community?
Foundational Knowledge for Researchers
To become comfortable with research and evaluation, you first need to understand the following:
- The mission and purpose of parks and recreation
- The need for research and evaluation
- How research and evaluation assist an organization
- The expectations for evaluation and research within the industry related to professional certification and agency accreditation
Mission and Purpose of Recreation Agencies
In the United States, the field of recreation and parks has deep roots grounded within societal needs, community needs, and individual needs (see figure 1.1). As you probably remember, the original needs of the community emerged during the Industrial Revolution. With the population shifting from rural areas to urban centers, children needed a safe place to play and socialize. The scope of facilities and services expanded from that point to include providing public parks, public recreation facilities, and instructional classes. The focus of this expansion was to (1) provide places for people to play, (2) offer programs that were educational and considered “wholesome” for the participant, and (3) provide recreational opportunities for individuals, families, and seniors within local communities.
Every park and recreation agency is challenged by—and exists to address—social concerns that become the needs within the community. For example, many American communities are concerned about gang activity, risk-taking behavior of youth, single-parent households, drug and alcohol abuse, and the growing need for sport fields and other public open spaces. Obesity is a primary example of a social concern that park and recreation programs are well positioned to address. This epidemic is prevalent in children, teenagers, and adults within the United States. Researchers predict that many members of the current generation of children will die before their parents because of factors related to obesity (high cholesterol, high blood pressure, diabetes, and a lack of exercise). The National Recreation and Park Association has partnered with other organizations to provide initiatives that address obesity factors, such as Hearts N' Parks, Step Up to Health, and Teens Outside. Each initiative is a community mobilization model that is designed to assist local communities in planning, marketing, and promoting healthy lifestyles related to diet and exercise. In addition, park and recreation agencies provide a wide variety of health and fitness programs, such as swimming lessons, open swim, aerobic classes, sport skill classes, movement classes for preschoolers, weight rooms for working out, and the simple reminder to get outside and play an hour a day. Healthy lifestyles go beyond physical fitness to emotional and psychological wellness. Park and recreation agencies also provide a wide variety of art classes, dance lessons, and concerts. All these kinds of programs and services work toward a healthier community.
Parks and recreation departments provide a wide variety of programs and services to their local communities. But are these programs accomplishing what they are intended to do for the individual, the organization, or the community?
The Need for Research and Evaluation
Every park and recreation agency needs to answer a number of questions about its programs, services, and operations which are noted in figure 1.2. These questions include the following:
- Is the agency meeting the objectives of the programs?
- Is the agency meeting the needs of the customer?
- Is the agency financially stable?
These are only a small sample of the questions that a park and recreation agency needs to answer. How does a park and recreation agency go about answering these questions? The best way is to use the process of research and evaluation. Let's explore these three questions further in order to gain an understanding of the importance of evaluation and research in providing the answers.
Is the Agency Meeting the Objectives of the Programs? All programs provided by park and recreation agencies have a purpose or an objective related to the outcomes for the participants. For example, the objective of a beginning swimming class is to have the participants move through water adjustment skills and learn a basic stroke on their front and back. For the organization, the objective for swimming classes is to have more people learn to swim. This can help reduce the number of rescues or deaths at the organization's aquatic facilities. In addition, the programs should cover costs associated with the programs and generate revenues for the agency. The agency must always consider the cost and revenues generated from the programs. For the community, the objective of swimming classes could be to have a variety of opportunities available for family members to learn to swim and to enjoy the aquatic facilities year-round. Many park and recreation agencies have indoor facilities, outdoor facilities, or both. Swimming is also a physical activity that people can enjoy throughout their lifetime, either in formal classes or individually. Park and recreation agencies must work to gather the information they need to determine if the objectives of the program are being met and whether these are class objectives, agency objectives, or objectives for the community. The research and evaluation process is the primary method for gathering this information and making this assessment. This process gives recreation professionals specific information and evidence to support or negate the notion that their agencies are achieving their objectives. Without research and evaluation, professionals might have only a vague sense of whether objectives are being met, or they may have no idea at all.
Is the Agency Meeting the Needs of the Customer? What are the customer needs and expectations in relation to recreational activities? For swimming classes, the needs of the customers are generally to have a safe environment for the class and to have the class conducted by a qualified instructor. The needs of the participants' parents are to have their children learn to swim in a safe environment so that the children can be safer at pools, lakes, and beaches during the summer. The swimming classes provide one way to meet these needs. Additionally, customers have expectations about their experience or their children's experience in the program. These expectations may be associated with the registration process, how to get to the facility, cleanliness of the locker rooms, cleanliness of the facility, and the quality of the instructor. The agency must gather information to determine if the needs and expectations of the customers are being met during the program.
Is the Agency Financially Stable? Historically, park and recreation departments received a majority of their funding for operations from local taxes. The agency was considered a type of social service that needed tax-based funding for the annual operations. This perception regarding park and recreation agencies has shifted from a social service model to a business-based model. Today, park and recreation agencies are expected to operate as a business. This has resulted in departments receiving very little tax-based funding for operations and has increased the need to generate revenues from programs and services. The agency gathers financial information on a regular basis to determine if the agency is generating sufficient funds to meet the financial needs of the organization. These research activities provide valuable information to assist the managers in making decisions related to the future operations of the agency. The information gained through research and evaluation helps managers decide which programs and services need to be revised, which ones should be added, and which ones should be phased out to ensure that the agency is financially stable. The decision on whether to add or phase out a program cannot be properly made unless the managers have current information about new community needs or needs within the community that no longer exist. The best method for gathering this information is through research and evaluation.
For agencies to answer these fundamental questions concerning their programs, services, and financial operations, research and evaluation have become a necessary activity. Historically, park and recreation agencies did not regularly evaluate program outcomes for the participants. In the past, evaluation efforts centered on the agency's need to know “the numbers”—the number of participants, the hours of the programs, the revenues generated by the programs, and the expenses associated with the programs. This type of evaluation misses the essence and purpose of the park and recreation agency (i.e., meeting the program objectives for the participants and meeting the needs of the customers and community). Evaluating and researching participant outcomes, needs, and expectations are critically important to the future of every park and recreation agency. Agencies that regularly conduct evaluation and research projects gain many benefits that help to sustain the operation of the agency.
This is an excerpt from Applied Research and Evaluation Methods in Recreation.
Steps of the research process
Scientific research involves a systematic process that focuses on being objective and gathering a multitude of information for analysis so that the researcher can come to a conclusion.
Scientific research involves a systematic process that focuses on being objective and gathering a multitude of information for analysis so that the researcher can come to a conclusion. This process is used in all research and evaluation projects, regardless of the research method (scientific method of inquiry, evaluation research, or action research). The process focuses on testing hunches or ideas in a park and recreation setting through a systematic process. In this process, the study is documented in such a way that another individual can conduct the same study again. This is referred to as replicating the study. Any research done without documenting the study so that others can review the process and results is not an investigation using the scientific research process. The scientific research process is a multiple-step process where the steps are interlinked with the other steps in the process. If changes are made in one step of the process, the researcher must review all the other steps to ensure that the changes are reflected throughout the process. Parks and recreation professionals are often involved in conducting research or evaluation projects within the agency. These professionals need to understand the eight steps of the research process as they apply to conducting a study. Table 2.4 lists the steps of the research process and provides an example of each step for a sample research study.
Step 1: Identify the Problem
The first step in the process is to identify a problem or develop a research question. The research problem may be something the agency identifies as a problem, some knowledge or information that is needed by the agency, or the desire to identify a recreation trend nationally. In the example in table 2.4, the problem that the agency has identified is childhood obesity, which is a local problem and concern within the community. This serves as the focus of the study.
Step 2: Review the Literature
Now that the problem has been identified, the researcher must learn more about the topic under investigation. To do this, the researcher must review the literature related to the research problem. This step provides foundational knowledge about the problem area. The review of literature also educates the researcher about what studies have been conducted in the past, how these studies were conducted, and the conclusions in the problem area. In the obesity study, the review of literature enables the programmer to discover horrifying statistics related to the long-term effects of childhood obesity in terms of health issues, death rates, and projected medical costs. In addition, the programmer finds several articles and information from the Centers for Disease Control and Prevention that describe the benefits of walking 10,000 steps a day. The information discovered during this step helps the programmer fully understand the magnitude of the problem, recognize the future consequences of obesity, and identify a strategy to combat obesity (i.e., walking).
Step 3: Clarify the Problem
Many times the initial problem identified in the first step of the process is too large or broad in scope. In step 3 of the process, the researcher clarifies the problem and narrows the scope of the study. This can only be done after the literature has been reviewed. The knowledge gained through the review of literature guides the researcher in clarifying and narrowing the research project. In the example, the programmer has identified childhood obesity as the problem and the purpose of the study. This topic is very broad and could be studied based on genetics, family environment, diet, exercise, self-confidence, leisure activities, or health issues. All of these areas cannot be investigated in a single study; therefore, the problem and purpose of the study must be more clearly defined. The programmer has decided that the purpose of the study is to determine if walking 10,000 steps a day for three days a week will improve the individual's health. This purpose is more narrowly focused and researchable than the original problem.
Step 4: Clearly Define Terms and Concepts
Terms and concepts are words or phrases used in the purpose statement of the study or the description of the study. These items need to be specifically defined as they apply to the study. Terms or concepts often have different definitions depending on who is reading the study. To minimize confusion about what the terms and phrases mean, the researcher must specifically define them for the study. In the obesity study, the concept of “individual's health” can be defined in hundreds of ways, such as physical, mental, emotional, or spiritual health. For this study, the individual's health is defined as physical health. The concept of physical health may also be defined and measured in many ways. In this case, the programmer decides to more narrowly define “individual health” to refer to the areas of weight, percentage of body fat, and cholesterol. By defining the terms or concepts more narrowly, the scope of the study is more manageable for the programmer, making it easier to collect the necessary data for the study. This also makes the concepts more understandable to the reader.
Step 5: Define the Population
Research projects can focus on a specific group of people, facilities, park development, employee evaluations, programs, financial status, marketing efforts, or the integration of technology into the operations. For example, if a researcher wants to examine a specific group of people in the community, the study could examine a specific age group, males or females, people living in a specific geographic area, or a specific ethnic group. Literally thousands of options are available to the researcher to specifically identify the group to study. The research problem and the purpose of the study assist the researcher in identifying the group to involve in the study. In research terms, the group to involve in the study is always called the population. Defining the population assists the researcher in several ways. First, it narrows the scope of the study from a very large population to one that is manageable. Second, the population identifies the group that the researcher's efforts will be focused on within the study. This helps ensure that the researcher stays on the right path during the study. Finally, by defining the population, the researcher identifies the group that the results will apply to at the conclusion of the study. In the example in table 2.4, the programmer has identified the population of the study as children ages 10 to 12 years. This narrower population makes the study more manageable in terms of time and resources.
Step 6: Develop the Instrumentation Plan
The plan for the study is referred to as the instrumentation plan. The instrumentation plan serves as the road map for the entire study, specifying who will participate in the study; how, when, and where data will be collected; and the content of the program. This plan is composed of numerous decisions and considerations that are addressed in chapter 8 of this text. In the obesity study, the researcher has decided to have the children participate in a walking program for six months. The group of participants is called the sample, which is a smaller group selected from the population specified for the study. The study cannot possibly include every 10- to 12-year-old child in the community, so a smaller group is used to represent the population. The researcher develops the plan for the walking program, indicating what data will be collected, when and how the data will be collected, who will collect the data, and how the data will be analyzed. The instrumentation plan specifies all the steps that must be completed for the study. This ensures that the programmer has carefully thought through all these decisions and that she provides a step-by-step plan to be followed in the study.
Step 7: Collect Data
Once the instrumentation plan is completed, the actual study begins with the collection of data. The collection of data is a critical step in providing the information needed to answer the research question. Every study includes the collection of some type of data—whether it is from the literature or from subjects—to answer the research question. Data can be collected in the form of words on a survey, with a questionnaire, through observations, or from the literature. In the obesity study, the programmers will be collecting data on the defined variables: weight, percentage of body fat, cholesterol levels, and the number of days the person walked a total of 10,000 steps during the class.
The researcher collects these data at the first session and at the last session of the program. These two sets of data are necessary to determine the effect of the walking program on weight, body fat, and cholesterol level. Once the data are collected on the variables, the researcher is ready to move to the final step of the process, which is the data analysis.
Step 8: Analyze the Data
All the time, effort, and resources dedicated to steps 1 through 7 of the research process culminate in this final step. The researcher finally has data to analyze so that the research question can be answered. In the instrumentation plan, the researcher specified how the data will be analyzed. The researcher now analyzes the data according to the plan. The results of this analysis are then reviewed and summarized in a manner directly related to the research questions. In the obesity study, the researcher compares the measurements of weight, percentage of body fat, and cholesterol that were taken at the first meeting of the subjects to the measurements of the same variables at the final program session. These two sets of data will be analyzed to determine if there was a difference between the first measurement and the second measurement for each individual in the program. Then, the data will be analyzed to determine if the differences are statistically significant. If the differences are statistically significant, the study validates the theory that was the focus of the study. The results of the study also provide valuable information about one strategy to combat childhood obesity in the community.
As you have probably concluded, conducting studies using the eight steps of the scientific research process requires you to dedicate time and effort to the planning process. You cannot conduct a study using the scientific research process when time is limited or the study is done at the last minute. Researchers who do this conduct studies that result in either false conclusions or conclusions that are not of any value to the organization.
This is an excerpt from Applied Research and Evaluation Methods in Recreation.
Creating instrumentation plans for research studies
The instrumentation plan helps guide the progress of the study to the ultimate goal of gathering data and formulating conclusions to answer the research question.
The instrumentation plan is composed of a number of decisions that need to be made before beginning the study. These decisions are made to determine
what data are needed to answer the research questions,
- how to gather the data,
- when to gather the data,
- where to gather the data, and
- how to analyze the data.
These decisions must be made as part of the instrumentation plan for the study. They help guide the progress of the study to the ultimate goal of gathering data and formulating conclusions to answer the research question.
Selecting Data
The previous decisions help define what data need to be gathered and why the data are important. What is data? Data refers to the information that is gathered to answer the research question. Data can be numbers, words, or actual objects, such as photos, articles, or video. The instrumentation process defines what data need to be collected and the timing of the data collection process.
What data are needed if the researcher is examining the relationship between taking swimming lessons and purchasing a summer session pass? The researcher needs the records of swimming lesson participants and season pass holders. Examining any other type of data, such as instructor evaluations, is a waste of time if it does not focus on answering the research question. If the researcher is examining the factors that lead a person to purchase a summer pass for the pool, interview data are needed to fully understand the factors in the decision-making process of the customer. By using the interview process, the researcher can conduct a two-way conversation in order to explore the factors that lead to purchasing the season pass. This type of data is very comprehensive in exploring the decision-making process, and it is more effective than looking at only one or two variables. In the example of the fitness center evaluating the personal trainers, a written survey will provide the data needed to evaluate the performances of the trainers. By identifying the specific type of data needed to answer the research question, the researchers' efforts are properly focused. Table 8.3 summarizes what data are needed in relation to the previous decisions in the instrumentation plan.
Gathering Data
The next natural question is, how will the data be collected? To answer this question, the researcher needs to identify whether the study is an evaluation, a quantitative study, or a qualitative study. This helps determine how to collect data. What instrument will be used to collect data? The researcher has a variety of options that may be used as a data collection instrument, such as surveys, interviews, observations, or rating instruments. Table 8.4 summarizes the decision of how to gather data along with the previous pieces of the instrumentation plan.
Determining When to Collect Data
Once the researcher decides what data to collect and how to collect them, the researcher must determine when to collect the data, where to collect them, and who should collect them. The results of the study can be influenced by the time that data are gathered, where the data are gathered, and who gathers the data. Studies that only use literature or the records of the agency are not influenced by these decisions. This is the case with the first research question concerning the relationship between taking swimming lessons and purchasing a summer season pass. Once the majority of season passes have been sold, the data should be gathered at the beginning of the summer.
This is not the case when data are being gathered from human subjects. Timing considerations that must be addressed are the month, the day of the week, and the time of day that the data are collected. In the example of identifying what factors lead a person to purchase a season pass to the pool, the interview should be conducted as soon as possible after the purchase of the pass. If the researchers wait weeks or months after the purchase, the subjects may not recall what factors led them to make the purchase. This interview could be done over the phone or in a small group.
For the fitness center evaluating the personal trainers, a specific time frame is necessary in order to ensure a high response rate. The manager decides to give the clients an evaluation form to complete after their last session with their personal trainer (before they leave the facility). This is a better plan than mailing the surveys to the clients' home weeks after the clients' last session with their trainer. The timing of data collection can affect the quality of the data received from the subjects. Table 8.5 presents this information, along with the previous decisions in the instrumentation plan.
Determining Where to Collect Data
The place that data are collected and the person collecting data must be specifically defined in the instrumentation plan. The best plan is to standardize the place and person for the data collection. This standardization helps enhance the truthfulness and validity of the data. The subjects should be in an environment where they feel at ease so that they will answer questions honestly. These two considerations are not an issue for the first research question because the data are coming from the documents of the agency and not from individuals.
In the case of identifying the factors that lead to purchasing a season pass for the pool, the interviews will be conducted by telephone. The people conducting the interviews should be trained in how to conduct an interview and how to record the information accurately. Each person being interviewed will be at home while participating in the interview, which is a comfortable environment for that individual.
In the example of evaluating personal trainers, having the clients' trainer administer the survey in the gym will most likely provide invalid data. The ideal situation would be to have one staff person provide the clients with a quiet room to complete the survey. Then the subject should return the survey to the same staff member who gave the survey to the subject. This standardizes where the data are collected and by whom. Table 8.6 provides a summary of all the instrument pieces and provides examples of each decision that needs to be made in developing the instrumentation plan for a study.
Analyzing Data
The next planning decision to make is how to analyze the data and what to do with the information once the analysis is completed. The type of analysis used with data is determined by whether the data are quantitative or qualitative data. To analyze quantitative data, some type of statistical analysis is used to provide the results. The type of statistical analysis used with data must be thought out and documented in the instrumentation plan.
Some of the most frequently used options for statistical analysis will be covered later in this text. Qualitative data are analyzed through a coding process that identifies themes; these themes become the foundation for the conclusions of the study. This type of data analysis will also be covered later in this text. The results of the data analysis should be summarized and presented in a report to supervisors and other parties for review. Research and evaluation efforts yield a wealth of information that can be used to educate commissioners, city councils, customers, and other decision makers. Studies that document outcomes of the programs and benefits to the community serve as a powerful tool that allows the agency to document its benefits to the community through facts and data, not perceptions and speculations. In today's cost-conscious society, the agency must prove that it is contributing to the quality of life of the community and must document the outcomes from the programs. This type of documentation and evidence is also a requirement for CAPRA accredited agencies and represents an ongoing evaluation process within the agency. Table 8.7 summarizes the key decisions that need to be made in the development of the instrumentation plan.
This is an excerpt from Applied Research and Evaluation Methods in Recreation.
The importance of research and evaluation in recreation
Parks and recreation departments provide a wide variety of programs and services to their local communities. But are these programs accomplishing what they are intended to do for the individual, the organization, or the community?
Foundational Knowledge for Researchers
To become comfortable with research and evaluation, you first need to understand the following:
- The mission and purpose of parks and recreation
- The need for research and evaluation
- How research and evaluation assist an organization
- The expectations for evaluation and research within the industry related to professional certification and agency accreditation
Mission and Purpose of Recreation Agencies
In the United States, the field of recreation and parks has deep roots grounded within societal needs, community needs, and individual needs (see figure 1.1). As you probably remember, the original needs of the community emerged during the Industrial Revolution. With the population shifting from rural areas to urban centers, children needed a safe place to play and socialize. The scope of facilities and services expanded from that point to include providing public parks, public recreation facilities, and instructional classes. The focus of this expansion was to (1) provide places for people to play, (2) offer programs that were educational and considered “wholesome” for the participant, and (3) provide recreational opportunities for individuals, families, and seniors within local communities.
Every park and recreation agency is challenged by—and exists to address—social concerns that become the needs within the community. For example, many American communities are concerned about gang activity, risk-taking behavior of youth, single-parent households, drug and alcohol abuse, and the growing need for sport fields and other public open spaces. Obesity is a primary example of a social concern that park and recreation programs are well positioned to address. This epidemic is prevalent in children, teenagers, and adults within the United States. Researchers predict that many members of the current generation of children will die before their parents because of factors related to obesity (high cholesterol, high blood pressure, diabetes, and a lack of exercise). The National Recreation and Park Association has partnered with other organizations to provide initiatives that address obesity factors, such as Hearts N' Parks, Step Up to Health, and Teens Outside. Each initiative is a community mobilization model that is designed to assist local communities in planning, marketing, and promoting healthy lifestyles related to diet and exercise. In addition, park and recreation agencies provide a wide variety of health and fitness programs, such as swimming lessons, open swim, aerobic classes, sport skill classes, movement classes for preschoolers, weight rooms for working out, and the simple reminder to get outside and play an hour a day. Healthy lifestyles go beyond physical fitness to emotional and psychological wellness. Park and recreation agencies also provide a wide variety of art classes, dance lessons, and concerts. All these kinds of programs and services work toward a healthier community.
Parks and recreation departments provide a wide variety of programs and services to their local communities. But are these programs accomplishing what they are intended to do for the individual, the organization, or the community?
The Need for Research and Evaluation
Every park and recreation agency needs to answer a number of questions about its programs, services, and operations which are noted in figure 1.2. These questions include the following:
- Is the agency meeting the objectives of the programs?
- Is the agency meeting the needs of the customer?
- Is the agency financially stable?
These are only a small sample of the questions that a park and recreation agency needs to answer. How does a park and recreation agency go about answering these questions? The best way is to use the process of research and evaluation. Let's explore these three questions further in order to gain an understanding of the importance of evaluation and research in providing the answers.
Is the Agency Meeting the Objectives of the Programs? All programs provided by park and recreation agencies have a purpose or an objective related to the outcomes for the participants. For example, the objective of a beginning swimming class is to have the participants move through water adjustment skills and learn a basic stroke on their front and back. For the organization, the objective for swimming classes is to have more people learn to swim. This can help reduce the number of rescues or deaths at the organization's aquatic facilities. In addition, the programs should cover costs associated with the programs and generate revenues for the agency. The agency must always consider the cost and revenues generated from the programs. For the community, the objective of swimming classes could be to have a variety of opportunities available for family members to learn to swim and to enjoy the aquatic facilities year-round. Many park and recreation agencies have indoor facilities, outdoor facilities, or both. Swimming is also a physical activity that people can enjoy throughout their lifetime, either in formal classes or individually. Park and recreation agencies must work to gather the information they need to determine if the objectives of the program are being met and whether these are class objectives, agency objectives, or objectives for the community. The research and evaluation process is the primary method for gathering this information and making this assessment. This process gives recreation professionals specific information and evidence to support or negate the notion that their agencies are achieving their objectives. Without research and evaluation, professionals might have only a vague sense of whether objectives are being met, or they may have no idea at all.
Is the Agency Meeting the Needs of the Customer? What are the customer needs and expectations in relation to recreational activities? For swimming classes, the needs of the customers are generally to have a safe environment for the class and to have the class conducted by a qualified instructor. The needs of the participants' parents are to have their children learn to swim in a safe environment so that the children can be safer at pools, lakes, and beaches during the summer. The swimming classes provide one way to meet these needs. Additionally, customers have expectations about their experience or their children's experience in the program. These expectations may be associated with the registration process, how to get to the facility, cleanliness of the locker rooms, cleanliness of the facility, and the quality of the instructor. The agency must gather information to determine if the needs and expectations of the customers are being met during the program.
Is the Agency Financially Stable? Historically, park and recreation departments received a majority of their funding for operations from local taxes. The agency was considered a type of social service that needed tax-based funding for the annual operations. This perception regarding park and recreation agencies has shifted from a social service model to a business-based model. Today, park and recreation agencies are expected to operate as a business. This has resulted in departments receiving very little tax-based funding for operations and has increased the need to generate revenues from programs and services. The agency gathers financial information on a regular basis to determine if the agency is generating sufficient funds to meet the financial needs of the organization. These research activities provide valuable information to assist the managers in making decisions related to the future operations of the agency. The information gained through research and evaluation helps managers decide which programs and services need to be revised, which ones should be added, and which ones should be phased out to ensure that the agency is financially stable. The decision on whether to add or phase out a program cannot be properly made unless the managers have current information about new community needs or needs within the community that no longer exist. The best method for gathering this information is through research and evaluation.
For agencies to answer these fundamental questions concerning their programs, services, and financial operations, research and evaluation have become a necessary activity. Historically, park and recreation agencies did not regularly evaluate program outcomes for the participants. In the past, evaluation efforts centered on the agency's need to know “the numbers”—the number of participants, the hours of the programs, the revenues generated by the programs, and the expenses associated with the programs. This type of evaluation misses the essence and purpose of the park and recreation agency (i.e., meeting the program objectives for the participants and meeting the needs of the customers and community). Evaluating and researching participant outcomes, needs, and expectations are critically important to the future of every park and recreation agency. Agencies that regularly conduct evaluation and research projects gain many benefits that help to sustain the operation of the agency.
This is an excerpt from Applied Research and Evaluation Methods in Recreation.
Steps of the research process
Scientific research involves a systematic process that focuses on being objective and gathering a multitude of information for analysis so that the researcher can come to a conclusion.
Scientific research involves a systematic process that focuses on being objective and gathering a multitude of information for analysis so that the researcher can come to a conclusion. This process is used in all research and evaluation projects, regardless of the research method (scientific method of inquiry, evaluation research, or action research). The process focuses on testing hunches or ideas in a park and recreation setting through a systematic process. In this process, the study is documented in such a way that another individual can conduct the same study again. This is referred to as replicating the study. Any research done without documenting the study so that others can review the process and results is not an investigation using the scientific research process. The scientific research process is a multiple-step process where the steps are interlinked with the other steps in the process. If changes are made in one step of the process, the researcher must review all the other steps to ensure that the changes are reflected throughout the process. Parks and recreation professionals are often involved in conducting research or evaluation projects within the agency. These professionals need to understand the eight steps of the research process as they apply to conducting a study. Table 2.4 lists the steps of the research process and provides an example of each step for a sample research study.
Step 1: Identify the Problem
The first step in the process is to identify a problem or develop a research question. The research problem may be something the agency identifies as a problem, some knowledge or information that is needed by the agency, or the desire to identify a recreation trend nationally. In the example in table 2.4, the problem that the agency has identified is childhood obesity, which is a local problem and concern within the community. This serves as the focus of the study.
Step 2: Review the Literature
Now that the problem has been identified, the researcher must learn more about the topic under investigation. To do this, the researcher must review the literature related to the research problem. This step provides foundational knowledge about the problem area. The review of literature also educates the researcher about what studies have been conducted in the past, how these studies were conducted, and the conclusions in the problem area. In the obesity study, the review of literature enables the programmer to discover horrifying statistics related to the long-term effects of childhood obesity in terms of health issues, death rates, and projected medical costs. In addition, the programmer finds several articles and information from the Centers for Disease Control and Prevention that describe the benefits of walking 10,000 steps a day. The information discovered during this step helps the programmer fully understand the magnitude of the problem, recognize the future consequences of obesity, and identify a strategy to combat obesity (i.e., walking).
Step 3: Clarify the Problem
Many times the initial problem identified in the first step of the process is too large or broad in scope. In step 3 of the process, the researcher clarifies the problem and narrows the scope of the study. This can only be done after the literature has been reviewed. The knowledge gained through the review of literature guides the researcher in clarifying and narrowing the research project. In the example, the programmer has identified childhood obesity as the problem and the purpose of the study. This topic is very broad and could be studied based on genetics, family environment, diet, exercise, self-confidence, leisure activities, or health issues. All of these areas cannot be investigated in a single study; therefore, the problem and purpose of the study must be more clearly defined. The programmer has decided that the purpose of the study is to determine if walking 10,000 steps a day for three days a week will improve the individual's health. This purpose is more narrowly focused and researchable than the original problem.
Step 4: Clearly Define Terms and Concepts
Terms and concepts are words or phrases used in the purpose statement of the study or the description of the study. These items need to be specifically defined as they apply to the study. Terms or concepts often have different definitions depending on who is reading the study. To minimize confusion about what the terms and phrases mean, the researcher must specifically define them for the study. In the obesity study, the concept of “individual's health” can be defined in hundreds of ways, such as physical, mental, emotional, or spiritual health. For this study, the individual's health is defined as physical health. The concept of physical health may also be defined and measured in many ways. In this case, the programmer decides to more narrowly define “individual health” to refer to the areas of weight, percentage of body fat, and cholesterol. By defining the terms or concepts more narrowly, the scope of the study is more manageable for the programmer, making it easier to collect the necessary data for the study. This also makes the concepts more understandable to the reader.
Step 5: Define the Population
Research projects can focus on a specific group of people, facilities, park development, employee evaluations, programs, financial status, marketing efforts, or the integration of technology into the operations. For example, if a researcher wants to examine a specific group of people in the community, the study could examine a specific age group, males or females, people living in a specific geographic area, or a specific ethnic group. Literally thousands of options are available to the researcher to specifically identify the group to study. The research problem and the purpose of the study assist the researcher in identifying the group to involve in the study. In research terms, the group to involve in the study is always called the population. Defining the population assists the researcher in several ways. First, it narrows the scope of the study from a very large population to one that is manageable. Second, the population identifies the group that the researcher's efforts will be focused on within the study. This helps ensure that the researcher stays on the right path during the study. Finally, by defining the population, the researcher identifies the group that the results will apply to at the conclusion of the study. In the example in table 2.4, the programmer has identified the population of the study as children ages 10 to 12 years. This narrower population makes the study more manageable in terms of time and resources.
Step 6: Develop the Instrumentation Plan
The plan for the study is referred to as the instrumentation plan. The instrumentation plan serves as the road map for the entire study, specifying who will participate in the study; how, when, and where data will be collected; and the content of the program. This plan is composed of numerous decisions and considerations that are addressed in chapter 8 of this text. In the obesity study, the researcher has decided to have the children participate in a walking program for six months. The group of participants is called the sample, which is a smaller group selected from the population specified for the study. The study cannot possibly include every 10- to 12-year-old child in the community, so a smaller group is used to represent the population. The researcher develops the plan for the walking program, indicating what data will be collected, when and how the data will be collected, who will collect the data, and how the data will be analyzed. The instrumentation plan specifies all the steps that must be completed for the study. This ensures that the programmer has carefully thought through all these decisions and that she provides a step-by-step plan to be followed in the study.
Step 7: Collect Data
Once the instrumentation plan is completed, the actual study begins with the collection of data. The collection of data is a critical step in providing the information needed to answer the research question. Every study includes the collection of some type of data—whether it is from the literature or from subjects—to answer the research question. Data can be collected in the form of words on a survey, with a questionnaire, through observations, or from the literature. In the obesity study, the programmers will be collecting data on the defined variables: weight, percentage of body fat, cholesterol levels, and the number of days the person walked a total of 10,000 steps during the class.
The researcher collects these data at the first session and at the last session of the program. These two sets of data are necessary to determine the effect of the walking program on weight, body fat, and cholesterol level. Once the data are collected on the variables, the researcher is ready to move to the final step of the process, which is the data analysis.
Step 8: Analyze the Data
All the time, effort, and resources dedicated to steps 1 through 7 of the research process culminate in this final step. The researcher finally has data to analyze so that the research question can be answered. In the instrumentation plan, the researcher specified how the data will be analyzed. The researcher now analyzes the data according to the plan. The results of this analysis are then reviewed and summarized in a manner directly related to the research questions. In the obesity study, the researcher compares the measurements of weight, percentage of body fat, and cholesterol that were taken at the first meeting of the subjects to the measurements of the same variables at the final program session. These two sets of data will be analyzed to determine if there was a difference between the first measurement and the second measurement for each individual in the program. Then, the data will be analyzed to determine if the differences are statistically significant. If the differences are statistically significant, the study validates the theory that was the focus of the study. The results of the study also provide valuable information about one strategy to combat childhood obesity in the community.
As you have probably concluded, conducting studies using the eight steps of the scientific research process requires you to dedicate time and effort to the planning process. You cannot conduct a study using the scientific research process when time is limited or the study is done at the last minute. Researchers who do this conduct studies that result in either false conclusions or conclusions that are not of any value to the organization.
This is an excerpt from Applied Research and Evaluation Methods in Recreation.
Creating instrumentation plans for research studies
The instrumentation plan helps guide the progress of the study to the ultimate goal of gathering data and formulating conclusions to answer the research question.
The instrumentation plan is composed of a number of decisions that need to be made before beginning the study. These decisions are made to determine
what data are needed to answer the research questions,
- how to gather the data,
- when to gather the data,
- where to gather the data, and
- how to analyze the data.
These decisions must be made as part of the instrumentation plan for the study. They help guide the progress of the study to the ultimate goal of gathering data and formulating conclusions to answer the research question.
Selecting Data
The previous decisions help define what data need to be gathered and why the data are important. What is data? Data refers to the information that is gathered to answer the research question. Data can be numbers, words, or actual objects, such as photos, articles, or video. The instrumentation process defines what data need to be collected and the timing of the data collection process.
What data are needed if the researcher is examining the relationship between taking swimming lessons and purchasing a summer session pass? The researcher needs the records of swimming lesson participants and season pass holders. Examining any other type of data, such as instructor evaluations, is a waste of time if it does not focus on answering the research question. If the researcher is examining the factors that lead a person to purchase a summer pass for the pool, interview data are needed to fully understand the factors in the decision-making process of the customer. By using the interview process, the researcher can conduct a two-way conversation in order to explore the factors that lead to purchasing the season pass. This type of data is very comprehensive in exploring the decision-making process, and it is more effective than looking at only one or two variables. In the example of the fitness center evaluating the personal trainers, a written survey will provide the data needed to evaluate the performances of the trainers. By identifying the specific type of data needed to answer the research question, the researchers' efforts are properly focused. Table 8.3 summarizes what data are needed in relation to the previous decisions in the instrumentation plan.
Gathering Data
The next natural question is, how will the data be collected? To answer this question, the researcher needs to identify whether the study is an evaluation, a quantitative study, or a qualitative study. This helps determine how to collect data. What instrument will be used to collect data? The researcher has a variety of options that may be used as a data collection instrument, such as surveys, interviews, observations, or rating instruments. Table 8.4 summarizes the decision of how to gather data along with the previous pieces of the instrumentation plan.
Determining When to Collect Data
Once the researcher decides what data to collect and how to collect them, the researcher must determine when to collect the data, where to collect them, and who should collect them. The results of the study can be influenced by the time that data are gathered, where the data are gathered, and who gathers the data. Studies that only use literature or the records of the agency are not influenced by these decisions. This is the case with the first research question concerning the relationship between taking swimming lessons and purchasing a summer season pass. Once the majority of season passes have been sold, the data should be gathered at the beginning of the summer.
This is not the case when data are being gathered from human subjects. Timing considerations that must be addressed are the month, the day of the week, and the time of day that the data are collected. In the example of identifying what factors lead a person to purchase a season pass to the pool, the interview should be conducted as soon as possible after the purchase of the pass. If the researchers wait weeks or months after the purchase, the subjects may not recall what factors led them to make the purchase. This interview could be done over the phone or in a small group.
For the fitness center evaluating the personal trainers, a specific time frame is necessary in order to ensure a high response rate. The manager decides to give the clients an evaluation form to complete after their last session with their personal trainer (before they leave the facility). This is a better plan than mailing the surveys to the clients' home weeks after the clients' last session with their trainer. The timing of data collection can affect the quality of the data received from the subjects. Table 8.5 presents this information, along with the previous decisions in the instrumentation plan.
Determining Where to Collect Data
The place that data are collected and the person collecting data must be specifically defined in the instrumentation plan. The best plan is to standardize the place and person for the data collection. This standardization helps enhance the truthfulness and validity of the data. The subjects should be in an environment where they feel at ease so that they will answer questions honestly. These two considerations are not an issue for the first research question because the data are coming from the documents of the agency and not from individuals.
In the case of identifying the factors that lead to purchasing a season pass for the pool, the interviews will be conducted by telephone. The people conducting the interviews should be trained in how to conduct an interview and how to record the information accurately. Each person being interviewed will be at home while participating in the interview, which is a comfortable environment for that individual.
In the example of evaluating personal trainers, having the clients' trainer administer the survey in the gym will most likely provide invalid data. The ideal situation would be to have one staff person provide the clients with a quiet room to complete the survey. Then the subject should return the survey to the same staff member who gave the survey to the subject. This standardizes where the data are collected and by whom. Table 8.6 provides a summary of all the instrument pieces and provides examples of each decision that needs to be made in developing the instrumentation plan for a study.
Analyzing Data
The next planning decision to make is how to analyze the data and what to do with the information once the analysis is completed. The type of analysis used with data is determined by whether the data are quantitative or qualitative data. To analyze quantitative data, some type of statistical analysis is used to provide the results. The type of statistical analysis used with data must be thought out and documented in the instrumentation plan.
Some of the most frequently used options for statistical analysis will be covered later in this text. Qualitative data are analyzed through a coding process that identifies themes; these themes become the foundation for the conclusions of the study. This type of data analysis will also be covered later in this text. The results of the data analysis should be summarized and presented in a report to supervisors and other parties for review. Research and evaluation efforts yield a wealth of information that can be used to educate commissioners, city councils, customers, and other decision makers. Studies that document outcomes of the programs and benefits to the community serve as a powerful tool that allows the agency to document its benefits to the community through facts and data, not perceptions and speculations. In today's cost-conscious society, the agency must prove that it is contributing to the quality of life of the community and must document the outcomes from the programs. This type of documentation and evidence is also a requirement for CAPRA accredited agencies and represents an ongoing evaluation process within the agency. Table 8.7 summarizes the key decisions that need to be made in the development of the instrumentation plan.
This is an excerpt from Applied Research and Evaluation Methods in Recreation.
The importance of research and evaluation in recreation
Parks and recreation departments provide a wide variety of programs and services to their local communities. But are these programs accomplishing what they are intended to do for the individual, the organization, or the community?
Foundational Knowledge for Researchers
To become comfortable with research and evaluation, you first need to understand the following:
- The mission and purpose of parks and recreation
- The need for research and evaluation
- How research and evaluation assist an organization
- The expectations for evaluation and research within the industry related to professional certification and agency accreditation
Mission and Purpose of Recreation Agencies
In the United States, the field of recreation and parks has deep roots grounded within societal needs, community needs, and individual needs (see figure 1.1). As you probably remember, the original needs of the community emerged during the Industrial Revolution. With the population shifting from rural areas to urban centers, children needed a safe place to play and socialize. The scope of facilities and services expanded from that point to include providing public parks, public recreation facilities, and instructional classes. The focus of this expansion was to (1) provide places for people to play, (2) offer programs that were educational and considered “wholesome” for the participant, and (3) provide recreational opportunities for individuals, families, and seniors within local communities.
Every park and recreation agency is challenged by—and exists to address—social concerns that become the needs within the community. For example, many American communities are concerned about gang activity, risk-taking behavior of youth, single-parent households, drug and alcohol abuse, and the growing need for sport fields and other public open spaces. Obesity is a primary example of a social concern that park and recreation programs are well positioned to address. This epidemic is prevalent in children, teenagers, and adults within the United States. Researchers predict that many members of the current generation of children will die before their parents because of factors related to obesity (high cholesterol, high blood pressure, diabetes, and a lack of exercise). The National Recreation and Park Association has partnered with other organizations to provide initiatives that address obesity factors, such as Hearts N' Parks, Step Up to Health, and Teens Outside. Each initiative is a community mobilization model that is designed to assist local communities in planning, marketing, and promoting healthy lifestyles related to diet and exercise. In addition, park and recreation agencies provide a wide variety of health and fitness programs, such as swimming lessons, open swim, aerobic classes, sport skill classes, movement classes for preschoolers, weight rooms for working out, and the simple reminder to get outside and play an hour a day. Healthy lifestyles go beyond physical fitness to emotional and psychological wellness. Park and recreation agencies also provide a wide variety of art classes, dance lessons, and concerts. All these kinds of programs and services work toward a healthier community.
Parks and recreation departments provide a wide variety of programs and services to their local communities. But are these programs accomplishing what they are intended to do for the individual, the organization, or the community?
The Need for Research and Evaluation
Every park and recreation agency needs to answer a number of questions about its programs, services, and operations which are noted in figure 1.2. These questions include the following:
- Is the agency meeting the objectives of the programs?
- Is the agency meeting the needs of the customer?
- Is the agency financially stable?
These are only a small sample of the questions that a park and recreation agency needs to answer. How does a park and recreation agency go about answering these questions? The best way is to use the process of research and evaluation. Let's explore these three questions further in order to gain an understanding of the importance of evaluation and research in providing the answers.
Is the Agency Meeting the Objectives of the Programs? All programs provided by park and recreation agencies have a purpose or an objective related to the outcomes for the participants. For example, the objective of a beginning swimming class is to have the participants move through water adjustment skills and learn a basic stroke on their front and back. For the organization, the objective for swimming classes is to have more people learn to swim. This can help reduce the number of rescues or deaths at the organization's aquatic facilities. In addition, the programs should cover costs associated with the programs and generate revenues for the agency. The agency must always consider the cost and revenues generated from the programs. For the community, the objective of swimming classes could be to have a variety of opportunities available for family members to learn to swim and to enjoy the aquatic facilities year-round. Many park and recreation agencies have indoor facilities, outdoor facilities, or both. Swimming is also a physical activity that people can enjoy throughout their lifetime, either in formal classes or individually. Park and recreation agencies must work to gather the information they need to determine if the objectives of the program are being met and whether these are class objectives, agency objectives, or objectives for the community. The research and evaluation process is the primary method for gathering this information and making this assessment. This process gives recreation professionals specific information and evidence to support or negate the notion that their agencies are achieving their objectives. Without research and evaluation, professionals might have only a vague sense of whether objectives are being met, or they may have no idea at all.
Is the Agency Meeting the Needs of the Customer? What are the customer needs and expectations in relation to recreational activities? For swimming classes, the needs of the customers are generally to have a safe environment for the class and to have the class conducted by a qualified instructor. The needs of the participants' parents are to have their children learn to swim in a safe environment so that the children can be safer at pools, lakes, and beaches during the summer. The swimming classes provide one way to meet these needs. Additionally, customers have expectations about their experience or their children's experience in the program. These expectations may be associated with the registration process, how to get to the facility, cleanliness of the locker rooms, cleanliness of the facility, and the quality of the instructor. The agency must gather information to determine if the needs and expectations of the customers are being met during the program.
Is the Agency Financially Stable? Historically, park and recreation departments received a majority of their funding for operations from local taxes. The agency was considered a type of social service that needed tax-based funding for the annual operations. This perception regarding park and recreation agencies has shifted from a social service model to a business-based model. Today, park and recreation agencies are expected to operate as a business. This has resulted in departments receiving very little tax-based funding for operations and has increased the need to generate revenues from programs and services. The agency gathers financial information on a regular basis to determine if the agency is generating sufficient funds to meet the financial needs of the organization. These research activities provide valuable information to assist the managers in making decisions related to the future operations of the agency. The information gained through research and evaluation helps managers decide which programs and services need to be revised, which ones should be added, and which ones should be phased out to ensure that the agency is financially stable. The decision on whether to add or phase out a program cannot be properly made unless the managers have current information about new community needs or needs within the community that no longer exist. The best method for gathering this information is through research and evaluation.
For agencies to answer these fundamental questions concerning their programs, services, and financial operations, research and evaluation have become a necessary activity. Historically, park and recreation agencies did not regularly evaluate program outcomes for the participants. In the past, evaluation efforts centered on the agency's need to know “the numbers”—the number of participants, the hours of the programs, the revenues generated by the programs, and the expenses associated with the programs. This type of evaluation misses the essence and purpose of the park and recreation agency (i.e., meeting the program objectives for the participants and meeting the needs of the customers and community). Evaluating and researching participant outcomes, needs, and expectations are critically important to the future of every park and recreation agency. Agencies that regularly conduct evaluation and research projects gain many benefits that help to sustain the operation of the agency.
This is an excerpt from Applied Research and Evaluation Methods in Recreation.
Steps of the research process
Scientific research involves a systematic process that focuses on being objective and gathering a multitude of information for analysis so that the researcher can come to a conclusion.
Scientific research involves a systematic process that focuses on being objective and gathering a multitude of information for analysis so that the researcher can come to a conclusion. This process is used in all research and evaluation projects, regardless of the research method (scientific method of inquiry, evaluation research, or action research). The process focuses on testing hunches or ideas in a park and recreation setting through a systematic process. In this process, the study is documented in such a way that another individual can conduct the same study again. This is referred to as replicating the study. Any research done without documenting the study so that others can review the process and results is not an investigation using the scientific research process. The scientific research process is a multiple-step process where the steps are interlinked with the other steps in the process. If changes are made in one step of the process, the researcher must review all the other steps to ensure that the changes are reflected throughout the process. Parks and recreation professionals are often involved in conducting research or evaluation projects within the agency. These professionals need to understand the eight steps of the research process as they apply to conducting a study. Table 2.4 lists the steps of the research process and provides an example of each step for a sample research study.
Step 1: Identify the Problem
The first step in the process is to identify a problem or develop a research question. The research problem may be something the agency identifies as a problem, some knowledge or information that is needed by the agency, or the desire to identify a recreation trend nationally. In the example in table 2.4, the problem that the agency has identified is childhood obesity, which is a local problem and concern within the community. This serves as the focus of the study.
Step 2: Review the Literature
Now that the problem has been identified, the researcher must learn more about the topic under investigation. To do this, the researcher must review the literature related to the research problem. This step provides foundational knowledge about the problem area. The review of literature also educates the researcher about what studies have been conducted in the past, how these studies were conducted, and the conclusions in the problem area. In the obesity study, the review of literature enables the programmer to discover horrifying statistics related to the long-term effects of childhood obesity in terms of health issues, death rates, and projected medical costs. In addition, the programmer finds several articles and information from the Centers for Disease Control and Prevention that describe the benefits of walking 10,000 steps a day. The information discovered during this step helps the programmer fully understand the magnitude of the problem, recognize the future consequences of obesity, and identify a strategy to combat obesity (i.e., walking).
Step 3: Clarify the Problem
Many times the initial problem identified in the first step of the process is too large or broad in scope. In step 3 of the process, the researcher clarifies the problem and narrows the scope of the study. This can only be done after the literature has been reviewed. The knowledge gained through the review of literature guides the researcher in clarifying and narrowing the research project. In the example, the programmer has identified childhood obesity as the problem and the purpose of the study. This topic is very broad and could be studied based on genetics, family environment, diet, exercise, self-confidence, leisure activities, or health issues. All of these areas cannot be investigated in a single study; therefore, the problem and purpose of the study must be more clearly defined. The programmer has decided that the purpose of the study is to determine if walking 10,000 steps a day for three days a week will improve the individual's health. This purpose is more narrowly focused and researchable than the original problem.
Step 4: Clearly Define Terms and Concepts
Terms and concepts are words or phrases used in the purpose statement of the study or the description of the study. These items need to be specifically defined as they apply to the study. Terms or concepts often have different definitions depending on who is reading the study. To minimize confusion about what the terms and phrases mean, the researcher must specifically define them for the study. In the obesity study, the concept of “individual's health” can be defined in hundreds of ways, such as physical, mental, emotional, or spiritual health. For this study, the individual's health is defined as physical health. The concept of physical health may also be defined and measured in many ways. In this case, the programmer decides to more narrowly define “individual health” to refer to the areas of weight, percentage of body fat, and cholesterol. By defining the terms or concepts more narrowly, the scope of the study is more manageable for the programmer, making it easier to collect the necessary data for the study. This also makes the concepts more understandable to the reader.
Step 5: Define the Population
Research projects can focus on a specific group of people, facilities, park development, employee evaluations, programs, financial status, marketing efforts, or the integration of technology into the operations. For example, if a researcher wants to examine a specific group of people in the community, the study could examine a specific age group, males or females, people living in a specific geographic area, or a specific ethnic group. Literally thousands of options are available to the researcher to specifically identify the group to study. The research problem and the purpose of the study assist the researcher in identifying the group to involve in the study. In research terms, the group to involve in the study is always called the population. Defining the population assists the researcher in several ways. First, it narrows the scope of the study from a very large population to one that is manageable. Second, the population identifies the group that the researcher's efforts will be focused on within the study. This helps ensure that the researcher stays on the right path during the study. Finally, by defining the population, the researcher identifies the group that the results will apply to at the conclusion of the study. In the example in table 2.4, the programmer has identified the population of the study as children ages 10 to 12 years. This narrower population makes the study more manageable in terms of time and resources.
Step 6: Develop the Instrumentation Plan
The plan for the study is referred to as the instrumentation plan. The instrumentation plan serves as the road map for the entire study, specifying who will participate in the study; how, when, and where data will be collected; and the content of the program. This plan is composed of numerous decisions and considerations that are addressed in chapter 8 of this text. In the obesity study, the researcher has decided to have the children participate in a walking program for six months. The group of participants is called the sample, which is a smaller group selected from the population specified for the study. The study cannot possibly include every 10- to 12-year-old child in the community, so a smaller group is used to represent the population. The researcher develops the plan for the walking program, indicating what data will be collected, when and how the data will be collected, who will collect the data, and how the data will be analyzed. The instrumentation plan specifies all the steps that must be completed for the study. This ensures that the programmer has carefully thought through all these decisions and that she provides a step-by-step plan to be followed in the study.
Step 7: Collect Data
Once the instrumentation plan is completed, the actual study begins with the collection of data. The collection of data is a critical step in providing the information needed to answer the research question. Every study includes the collection of some type of data—whether it is from the literature or from subjects—to answer the research question. Data can be collected in the form of words on a survey, with a questionnaire, through observations, or from the literature. In the obesity study, the programmers will be collecting data on the defined variables: weight, percentage of body fat, cholesterol levels, and the number of days the person walked a total of 10,000 steps during the class.
The researcher collects these data at the first session and at the last session of the program. These two sets of data are necessary to determine the effect of the walking program on weight, body fat, and cholesterol level. Once the data are collected on the variables, the researcher is ready to move to the final step of the process, which is the data analysis.
Step 8: Analyze the Data
All the time, effort, and resources dedicated to steps 1 through 7 of the research process culminate in this final step. The researcher finally has data to analyze so that the research question can be answered. In the instrumentation plan, the researcher specified how the data will be analyzed. The researcher now analyzes the data according to the plan. The results of this analysis are then reviewed and summarized in a manner directly related to the research questions. In the obesity study, the researcher compares the measurements of weight, percentage of body fat, and cholesterol that were taken at the first meeting of the subjects to the measurements of the same variables at the final program session. These two sets of data will be analyzed to determine if there was a difference between the first measurement and the second measurement for each individual in the program. Then, the data will be analyzed to determine if the differences are statistically significant. If the differences are statistically significant, the study validates the theory that was the focus of the study. The results of the study also provide valuable information about one strategy to combat childhood obesity in the community.
As you have probably concluded, conducting studies using the eight steps of the scientific research process requires you to dedicate time and effort to the planning process. You cannot conduct a study using the scientific research process when time is limited or the study is done at the last minute. Researchers who do this conduct studies that result in either false conclusions or conclusions that are not of any value to the organization.
This is an excerpt from Applied Research and Evaluation Methods in Recreation.
Creating instrumentation plans for research studies
The instrumentation plan helps guide the progress of the study to the ultimate goal of gathering data and formulating conclusions to answer the research question.
The instrumentation plan is composed of a number of decisions that need to be made before beginning the study. These decisions are made to determine
what data are needed to answer the research questions,
- how to gather the data,
- when to gather the data,
- where to gather the data, and
- how to analyze the data.
These decisions must be made as part of the instrumentation plan for the study. They help guide the progress of the study to the ultimate goal of gathering data and formulating conclusions to answer the research question.
Selecting Data
The previous decisions help define what data need to be gathered and why the data are important. What is data? Data refers to the information that is gathered to answer the research question. Data can be numbers, words, or actual objects, such as photos, articles, or video. The instrumentation process defines what data need to be collected and the timing of the data collection process.
What data are needed if the researcher is examining the relationship between taking swimming lessons and purchasing a summer session pass? The researcher needs the records of swimming lesson participants and season pass holders. Examining any other type of data, such as instructor evaluations, is a waste of time if it does not focus on answering the research question. If the researcher is examining the factors that lead a person to purchase a summer pass for the pool, interview data are needed to fully understand the factors in the decision-making process of the customer. By using the interview process, the researcher can conduct a two-way conversation in order to explore the factors that lead to purchasing the season pass. This type of data is very comprehensive in exploring the decision-making process, and it is more effective than looking at only one or two variables. In the example of the fitness center evaluating the personal trainers, a written survey will provide the data needed to evaluate the performances of the trainers. By identifying the specific type of data needed to answer the research question, the researchers' efforts are properly focused. Table 8.3 summarizes what data are needed in relation to the previous decisions in the instrumentation plan.
Gathering Data
The next natural question is, how will the data be collected? To answer this question, the researcher needs to identify whether the study is an evaluation, a quantitative study, or a qualitative study. This helps determine how to collect data. What instrument will be used to collect data? The researcher has a variety of options that may be used as a data collection instrument, such as surveys, interviews, observations, or rating instruments. Table 8.4 summarizes the decision of how to gather data along with the previous pieces of the instrumentation plan.
Determining When to Collect Data
Once the researcher decides what data to collect and how to collect them, the researcher must determine when to collect the data, where to collect them, and who should collect them. The results of the study can be influenced by the time that data are gathered, where the data are gathered, and who gathers the data. Studies that only use literature or the records of the agency are not influenced by these decisions. This is the case with the first research question concerning the relationship between taking swimming lessons and purchasing a summer season pass. Once the majority of season passes have been sold, the data should be gathered at the beginning of the summer.
This is not the case when data are being gathered from human subjects. Timing considerations that must be addressed are the month, the day of the week, and the time of day that the data are collected. In the example of identifying what factors lead a person to purchase a season pass to the pool, the interview should be conducted as soon as possible after the purchase of the pass. If the researchers wait weeks or months after the purchase, the subjects may not recall what factors led them to make the purchase. This interview could be done over the phone or in a small group.
For the fitness center evaluating the personal trainers, a specific time frame is necessary in order to ensure a high response rate. The manager decides to give the clients an evaluation form to complete after their last session with their personal trainer (before they leave the facility). This is a better plan than mailing the surveys to the clients' home weeks after the clients' last session with their trainer. The timing of data collection can affect the quality of the data received from the subjects. Table 8.5 presents this information, along with the previous decisions in the instrumentation plan.
Determining Where to Collect Data
The place that data are collected and the person collecting data must be specifically defined in the instrumentation plan. The best plan is to standardize the place and person for the data collection. This standardization helps enhance the truthfulness and validity of the data. The subjects should be in an environment where they feel at ease so that they will answer questions honestly. These two considerations are not an issue for the first research question because the data are coming from the documents of the agency and not from individuals.
In the case of identifying the factors that lead to purchasing a season pass for the pool, the interviews will be conducted by telephone. The people conducting the interviews should be trained in how to conduct an interview and how to record the information accurately. Each person being interviewed will be at home while participating in the interview, which is a comfortable environment for that individual.
In the example of evaluating personal trainers, having the clients' trainer administer the survey in the gym will most likely provide invalid data. The ideal situation would be to have one staff person provide the clients with a quiet room to complete the survey. Then the subject should return the survey to the same staff member who gave the survey to the subject. This standardizes where the data are collected and by whom. Table 8.6 provides a summary of all the instrument pieces and provides examples of each decision that needs to be made in developing the instrumentation plan for a study.
Analyzing Data
The next planning decision to make is how to analyze the data and what to do with the information once the analysis is completed. The type of analysis used with data is determined by whether the data are quantitative or qualitative data. To analyze quantitative data, some type of statistical analysis is used to provide the results. The type of statistical analysis used with data must be thought out and documented in the instrumentation plan.
Some of the most frequently used options for statistical analysis will be covered later in this text. Qualitative data are analyzed through a coding process that identifies themes; these themes become the foundation for the conclusions of the study. This type of data analysis will also be covered later in this text. The results of the data analysis should be summarized and presented in a report to supervisors and other parties for review. Research and evaluation efforts yield a wealth of information that can be used to educate commissioners, city councils, customers, and other decision makers. Studies that document outcomes of the programs and benefits to the community serve as a powerful tool that allows the agency to document its benefits to the community through facts and data, not perceptions and speculations. In today's cost-conscious society, the agency must prove that it is contributing to the quality of life of the community and must document the outcomes from the programs. This type of documentation and evidence is also a requirement for CAPRA accredited agencies and represents an ongoing evaluation process within the agency. Table 8.7 summarizes the key decisions that need to be made in the development of the instrumentation plan.
This is an excerpt from Applied Research and Evaluation Methods in Recreation.
The importance of research and evaluation in recreation
Parks and recreation departments provide a wide variety of programs and services to their local communities. But are these programs accomplishing what they are intended to do for the individual, the organization, or the community?
Foundational Knowledge for Researchers
To become comfortable with research and evaluation, you first need to understand the following:
- The mission and purpose of parks and recreation
- The need for research and evaluation
- How research and evaluation assist an organization
- The expectations for evaluation and research within the industry related to professional certification and agency accreditation
Mission and Purpose of Recreation Agencies
In the United States, the field of recreation and parks has deep roots grounded within societal needs, community needs, and individual needs (see figure 1.1). As you probably remember, the original needs of the community emerged during the Industrial Revolution. With the population shifting from rural areas to urban centers, children needed a safe place to play and socialize. The scope of facilities and services expanded from that point to include providing public parks, public recreation facilities, and instructional classes. The focus of this expansion was to (1) provide places for people to play, (2) offer programs that were educational and considered “wholesome” for the participant, and (3) provide recreational opportunities for individuals, families, and seniors within local communities.
Every park and recreation agency is challenged by—and exists to address—social concerns that become the needs within the community. For example, many American communities are concerned about gang activity, risk-taking behavior of youth, single-parent households, drug and alcohol abuse, and the growing need for sport fields and other public open spaces. Obesity is a primary example of a social concern that park and recreation programs are well positioned to address. This epidemic is prevalent in children, teenagers, and adults within the United States. Researchers predict that many members of the current generation of children will die before their parents because of factors related to obesity (high cholesterol, high blood pressure, diabetes, and a lack of exercise). The National Recreation and Park Association has partnered with other organizations to provide initiatives that address obesity factors, such as Hearts N' Parks, Step Up to Health, and Teens Outside. Each initiative is a community mobilization model that is designed to assist local communities in planning, marketing, and promoting healthy lifestyles related to diet and exercise. In addition, park and recreation agencies provide a wide variety of health and fitness programs, such as swimming lessons, open swim, aerobic classes, sport skill classes, movement classes for preschoolers, weight rooms for working out, and the simple reminder to get outside and play an hour a day. Healthy lifestyles go beyond physical fitness to emotional and psychological wellness. Park and recreation agencies also provide a wide variety of art classes, dance lessons, and concerts. All these kinds of programs and services work toward a healthier community.
Parks and recreation departments provide a wide variety of programs and services to their local communities. But are these programs accomplishing what they are intended to do for the individual, the organization, or the community?
The Need for Research and Evaluation
Every park and recreation agency needs to answer a number of questions about its programs, services, and operations which are noted in figure 1.2. These questions include the following:
- Is the agency meeting the objectives of the programs?
- Is the agency meeting the needs of the customer?
- Is the agency financially stable?
These are only a small sample of the questions that a park and recreation agency needs to answer. How does a park and recreation agency go about answering these questions? The best way is to use the process of research and evaluation. Let's explore these three questions further in order to gain an understanding of the importance of evaluation and research in providing the answers.
Is the Agency Meeting the Objectives of the Programs? All programs provided by park and recreation agencies have a purpose or an objective related to the outcomes for the participants. For example, the objective of a beginning swimming class is to have the participants move through water adjustment skills and learn a basic stroke on their front and back. For the organization, the objective for swimming classes is to have more people learn to swim. This can help reduce the number of rescues or deaths at the organization's aquatic facilities. In addition, the programs should cover costs associated with the programs and generate revenues for the agency. The agency must always consider the cost and revenues generated from the programs. For the community, the objective of swimming classes could be to have a variety of opportunities available for family members to learn to swim and to enjoy the aquatic facilities year-round. Many park and recreation agencies have indoor facilities, outdoor facilities, or both. Swimming is also a physical activity that people can enjoy throughout their lifetime, either in formal classes or individually. Park and recreation agencies must work to gather the information they need to determine if the objectives of the program are being met and whether these are class objectives, agency objectives, or objectives for the community. The research and evaluation process is the primary method for gathering this information and making this assessment. This process gives recreation professionals specific information and evidence to support or negate the notion that their agencies are achieving their objectives. Without research and evaluation, professionals might have only a vague sense of whether objectives are being met, or they may have no idea at all.
Is the Agency Meeting the Needs of the Customer? What are the customer needs and expectations in relation to recreational activities? For swimming classes, the needs of the customers are generally to have a safe environment for the class and to have the class conducted by a qualified instructor. The needs of the participants' parents are to have their children learn to swim in a safe environment so that the children can be safer at pools, lakes, and beaches during the summer. The swimming classes provide one way to meet these needs. Additionally, customers have expectations about their experience or their children's experience in the program. These expectations may be associated with the registration process, how to get to the facility, cleanliness of the locker rooms, cleanliness of the facility, and the quality of the instructor. The agency must gather information to determine if the needs and expectations of the customers are being met during the program.
Is the Agency Financially Stable? Historically, park and recreation departments received a majority of their funding for operations from local taxes. The agency was considered a type of social service that needed tax-based funding for the annual operations. This perception regarding park and recreation agencies has shifted from a social service model to a business-based model. Today, park and recreation agencies are expected to operate as a business. This has resulted in departments receiving very little tax-based funding for operations and has increased the need to generate revenues from programs and services. The agency gathers financial information on a regular basis to determine if the agency is generating sufficient funds to meet the financial needs of the organization. These research activities provide valuable information to assist the managers in making decisions related to the future operations of the agency. The information gained through research and evaluation helps managers decide which programs and services need to be revised, which ones should be added, and which ones should be phased out to ensure that the agency is financially stable. The decision on whether to add or phase out a program cannot be properly made unless the managers have current information about new community needs or needs within the community that no longer exist. The best method for gathering this information is through research and evaluation.
For agencies to answer these fundamental questions concerning their programs, services, and financial operations, research and evaluation have become a necessary activity. Historically, park and recreation agencies did not regularly evaluate program outcomes for the participants. In the past, evaluation efforts centered on the agency's need to know “the numbers”—the number of participants, the hours of the programs, the revenues generated by the programs, and the expenses associated with the programs. This type of evaluation misses the essence and purpose of the park and recreation agency (i.e., meeting the program objectives for the participants and meeting the needs of the customers and community). Evaluating and researching participant outcomes, needs, and expectations are critically important to the future of every park and recreation agency. Agencies that regularly conduct evaluation and research projects gain many benefits that help to sustain the operation of the agency.
This is an excerpt from Applied Research and Evaluation Methods in Recreation.
Steps of the research process
Scientific research involves a systematic process that focuses on being objective and gathering a multitude of information for analysis so that the researcher can come to a conclusion.
Scientific research involves a systematic process that focuses on being objective and gathering a multitude of information for analysis so that the researcher can come to a conclusion. This process is used in all research and evaluation projects, regardless of the research method (scientific method of inquiry, evaluation research, or action research). The process focuses on testing hunches or ideas in a park and recreation setting through a systematic process. In this process, the study is documented in such a way that another individual can conduct the same study again. This is referred to as replicating the study. Any research done without documenting the study so that others can review the process and results is not an investigation using the scientific research process. The scientific research process is a multiple-step process where the steps are interlinked with the other steps in the process. If changes are made in one step of the process, the researcher must review all the other steps to ensure that the changes are reflected throughout the process. Parks and recreation professionals are often involved in conducting research or evaluation projects within the agency. These professionals need to understand the eight steps of the research process as they apply to conducting a study. Table 2.4 lists the steps of the research process and provides an example of each step for a sample research study.
Step 1: Identify the Problem
The first step in the process is to identify a problem or develop a research question. The research problem may be something the agency identifies as a problem, some knowledge or information that is needed by the agency, or the desire to identify a recreation trend nationally. In the example in table 2.4, the problem that the agency has identified is childhood obesity, which is a local problem and concern within the community. This serves as the focus of the study.
Step 2: Review the Literature
Now that the problem has been identified, the researcher must learn more about the topic under investigation. To do this, the researcher must review the literature related to the research problem. This step provides foundational knowledge about the problem area. The review of literature also educates the researcher about what studies have been conducted in the past, how these studies were conducted, and the conclusions in the problem area. In the obesity study, the review of literature enables the programmer to discover horrifying statistics related to the long-term effects of childhood obesity in terms of health issues, death rates, and projected medical costs. In addition, the programmer finds several articles and information from the Centers for Disease Control and Prevention that describe the benefits of walking 10,000 steps a day. The information discovered during this step helps the programmer fully understand the magnitude of the problem, recognize the future consequences of obesity, and identify a strategy to combat obesity (i.e., walking).
Step 3: Clarify the Problem
Many times the initial problem identified in the first step of the process is too large or broad in scope. In step 3 of the process, the researcher clarifies the problem and narrows the scope of the study. This can only be done after the literature has been reviewed. The knowledge gained through the review of literature guides the researcher in clarifying and narrowing the research project. In the example, the programmer has identified childhood obesity as the problem and the purpose of the study. This topic is very broad and could be studied based on genetics, family environment, diet, exercise, self-confidence, leisure activities, or health issues. All of these areas cannot be investigated in a single study; therefore, the problem and purpose of the study must be more clearly defined. The programmer has decided that the purpose of the study is to determine if walking 10,000 steps a day for three days a week will improve the individual's health. This purpose is more narrowly focused and researchable than the original problem.
Step 4: Clearly Define Terms and Concepts
Terms and concepts are words or phrases used in the purpose statement of the study or the description of the study. These items need to be specifically defined as they apply to the study. Terms or concepts often have different definitions depending on who is reading the study. To minimize confusion about what the terms and phrases mean, the researcher must specifically define them for the study. In the obesity study, the concept of “individual's health” can be defined in hundreds of ways, such as physical, mental, emotional, or spiritual health. For this study, the individual's health is defined as physical health. The concept of physical health may also be defined and measured in many ways. In this case, the programmer decides to more narrowly define “individual health” to refer to the areas of weight, percentage of body fat, and cholesterol. By defining the terms or concepts more narrowly, the scope of the study is more manageable for the programmer, making it easier to collect the necessary data for the study. This also makes the concepts more understandable to the reader.
Step 5: Define the Population
Research projects can focus on a specific group of people, facilities, park development, employee evaluations, programs, financial status, marketing efforts, or the integration of technology into the operations. For example, if a researcher wants to examine a specific group of people in the community, the study could examine a specific age group, males or females, people living in a specific geographic area, or a specific ethnic group. Literally thousands of options are available to the researcher to specifically identify the group to study. The research problem and the purpose of the study assist the researcher in identifying the group to involve in the study. In research terms, the group to involve in the study is always called the population. Defining the population assists the researcher in several ways. First, it narrows the scope of the study from a very large population to one that is manageable. Second, the population identifies the group that the researcher's efforts will be focused on within the study. This helps ensure that the researcher stays on the right path during the study. Finally, by defining the population, the researcher identifies the group that the results will apply to at the conclusion of the study. In the example in table 2.4, the programmer has identified the population of the study as children ages 10 to 12 years. This narrower population makes the study more manageable in terms of time and resources.
Step 6: Develop the Instrumentation Plan
The plan for the study is referred to as the instrumentation plan. The instrumentation plan serves as the road map for the entire study, specifying who will participate in the study; how, when, and where data will be collected; and the content of the program. This plan is composed of numerous decisions and considerations that are addressed in chapter 8 of this text. In the obesity study, the researcher has decided to have the children participate in a walking program for six months. The group of participants is called the sample, which is a smaller group selected from the population specified for the study. The study cannot possibly include every 10- to 12-year-old child in the community, so a smaller group is used to represent the population. The researcher develops the plan for the walking program, indicating what data will be collected, when and how the data will be collected, who will collect the data, and how the data will be analyzed. The instrumentation plan specifies all the steps that must be completed for the study. This ensures that the programmer has carefully thought through all these decisions and that she provides a step-by-step plan to be followed in the study.
Step 7: Collect Data
Once the instrumentation plan is completed, the actual study begins with the collection of data. The collection of data is a critical step in providing the information needed to answer the research question. Every study includes the collection of some type of data—whether it is from the literature or from subjects—to answer the research question. Data can be collected in the form of words on a survey, with a questionnaire, through observations, or from the literature. In the obesity study, the programmers will be collecting data on the defined variables: weight, percentage of body fat, cholesterol levels, and the number of days the person walked a total of 10,000 steps during the class.
The researcher collects these data at the first session and at the last session of the program. These two sets of data are necessary to determine the effect of the walking program on weight, body fat, and cholesterol level. Once the data are collected on the variables, the researcher is ready to move to the final step of the process, which is the data analysis.
Step 8: Analyze the Data
All the time, effort, and resources dedicated to steps 1 through 7 of the research process culminate in this final step. The researcher finally has data to analyze so that the research question can be answered. In the instrumentation plan, the researcher specified how the data will be analyzed. The researcher now analyzes the data according to the plan. The results of this analysis are then reviewed and summarized in a manner directly related to the research questions. In the obesity study, the researcher compares the measurements of weight, percentage of body fat, and cholesterol that were taken at the first meeting of the subjects to the measurements of the same variables at the final program session. These two sets of data will be analyzed to determine if there was a difference between the first measurement and the second measurement for each individual in the program. Then, the data will be analyzed to determine if the differences are statistically significant. If the differences are statistically significant, the study validates the theory that was the focus of the study. The results of the study also provide valuable information about one strategy to combat childhood obesity in the community.
As you have probably concluded, conducting studies using the eight steps of the scientific research process requires you to dedicate time and effort to the planning process. You cannot conduct a study using the scientific research process when time is limited or the study is done at the last minute. Researchers who do this conduct studies that result in either false conclusions or conclusions that are not of any value to the organization.
This is an excerpt from Applied Research and Evaluation Methods in Recreation.
Creating instrumentation plans for research studies
The instrumentation plan helps guide the progress of the study to the ultimate goal of gathering data and formulating conclusions to answer the research question.
The instrumentation plan is composed of a number of decisions that need to be made before beginning the study. These decisions are made to determine
what data are needed to answer the research questions,
- how to gather the data,
- when to gather the data,
- where to gather the data, and
- how to analyze the data.
These decisions must be made as part of the instrumentation plan for the study. They help guide the progress of the study to the ultimate goal of gathering data and formulating conclusions to answer the research question.
Selecting Data
The previous decisions help define what data need to be gathered and why the data are important. What is data? Data refers to the information that is gathered to answer the research question. Data can be numbers, words, or actual objects, such as photos, articles, or video. The instrumentation process defines what data need to be collected and the timing of the data collection process.
What data are needed if the researcher is examining the relationship between taking swimming lessons and purchasing a summer session pass? The researcher needs the records of swimming lesson participants and season pass holders. Examining any other type of data, such as instructor evaluations, is a waste of time if it does not focus on answering the research question. If the researcher is examining the factors that lead a person to purchase a summer pass for the pool, interview data are needed to fully understand the factors in the decision-making process of the customer. By using the interview process, the researcher can conduct a two-way conversation in order to explore the factors that lead to purchasing the season pass. This type of data is very comprehensive in exploring the decision-making process, and it is more effective than looking at only one or two variables. In the example of the fitness center evaluating the personal trainers, a written survey will provide the data needed to evaluate the performances of the trainers. By identifying the specific type of data needed to answer the research question, the researchers' efforts are properly focused. Table 8.3 summarizes what data are needed in relation to the previous decisions in the instrumentation plan.
Gathering Data
The next natural question is, how will the data be collected? To answer this question, the researcher needs to identify whether the study is an evaluation, a quantitative study, or a qualitative study. This helps determine how to collect data. What instrument will be used to collect data? The researcher has a variety of options that may be used as a data collection instrument, such as surveys, interviews, observations, or rating instruments. Table 8.4 summarizes the decision of how to gather data along with the previous pieces of the instrumentation plan.
Determining When to Collect Data
Once the researcher decides what data to collect and how to collect them, the researcher must determine when to collect the data, where to collect them, and who should collect them. The results of the study can be influenced by the time that data are gathered, where the data are gathered, and who gathers the data. Studies that only use literature or the records of the agency are not influenced by these decisions. This is the case with the first research question concerning the relationship between taking swimming lessons and purchasing a summer season pass. Once the majority of season passes have been sold, the data should be gathered at the beginning of the summer.
This is not the case when data are being gathered from human subjects. Timing considerations that must be addressed are the month, the day of the week, and the time of day that the data are collected. In the example of identifying what factors lead a person to purchase a season pass to the pool, the interview should be conducted as soon as possible after the purchase of the pass. If the researchers wait weeks or months after the purchase, the subjects may not recall what factors led them to make the purchase. This interview could be done over the phone or in a small group.
For the fitness center evaluating the personal trainers, a specific time frame is necessary in order to ensure a high response rate. The manager decides to give the clients an evaluation form to complete after their last session with their personal trainer (before they leave the facility). This is a better plan than mailing the surveys to the clients' home weeks after the clients' last session with their trainer. The timing of data collection can affect the quality of the data received from the subjects. Table 8.5 presents this information, along with the previous decisions in the instrumentation plan.
Determining Where to Collect Data
The place that data are collected and the person collecting data must be specifically defined in the instrumentation plan. The best plan is to standardize the place and person for the data collection. This standardization helps enhance the truthfulness and validity of the data. The subjects should be in an environment where they feel at ease so that they will answer questions honestly. These two considerations are not an issue for the first research question because the data are coming from the documents of the agency and not from individuals.
In the case of identifying the factors that lead to purchasing a season pass for the pool, the interviews will be conducted by telephone. The people conducting the interviews should be trained in how to conduct an interview and how to record the information accurately. Each person being interviewed will be at home while participating in the interview, which is a comfortable environment for that individual.
In the example of evaluating personal trainers, having the clients' trainer administer the survey in the gym will most likely provide invalid data. The ideal situation would be to have one staff person provide the clients with a quiet room to complete the survey. Then the subject should return the survey to the same staff member who gave the survey to the subject. This standardizes where the data are collected and by whom. Table 8.6 provides a summary of all the instrument pieces and provides examples of each decision that needs to be made in developing the instrumentation plan for a study.
Analyzing Data
The next planning decision to make is how to analyze the data and what to do with the information once the analysis is completed. The type of analysis used with data is determined by whether the data are quantitative or qualitative data. To analyze quantitative data, some type of statistical analysis is used to provide the results. The type of statistical analysis used with data must be thought out and documented in the instrumentation plan.
Some of the most frequently used options for statistical analysis will be covered later in this text. Qualitative data are analyzed through a coding process that identifies themes; these themes become the foundation for the conclusions of the study. This type of data analysis will also be covered later in this text. The results of the data analysis should be summarized and presented in a report to supervisors and other parties for review. Research and evaluation efforts yield a wealth of information that can be used to educate commissioners, city councils, customers, and other decision makers. Studies that document outcomes of the programs and benefits to the community serve as a powerful tool that allows the agency to document its benefits to the community through facts and data, not perceptions and speculations. In today's cost-conscious society, the agency must prove that it is contributing to the quality of life of the community and must document the outcomes from the programs. This type of documentation and evidence is also a requirement for CAPRA accredited agencies and represents an ongoing evaluation process within the agency. Table 8.7 summarizes the key decisions that need to be made in the development of the instrumentation plan.
This is an excerpt from Applied Research and Evaluation Methods in Recreation.
The importance of research and evaluation in recreation
Parks and recreation departments provide a wide variety of programs and services to their local communities. But are these programs accomplishing what they are intended to do for the individual, the organization, or the community?
Foundational Knowledge for Researchers
To become comfortable with research and evaluation, you first need to understand the following:
- The mission and purpose of parks and recreation
- The need for research and evaluation
- How research and evaluation assist an organization
- The expectations for evaluation and research within the industry related to professional certification and agency accreditation
Mission and Purpose of Recreation Agencies
In the United States, the field of recreation and parks has deep roots grounded within societal needs, community needs, and individual needs (see figure 1.1). As you probably remember, the original needs of the community emerged during the Industrial Revolution. With the population shifting from rural areas to urban centers, children needed a safe place to play and socialize. The scope of facilities and services expanded from that point to include providing public parks, public recreation facilities, and instructional classes. The focus of this expansion was to (1) provide places for people to play, (2) offer programs that were educational and considered “wholesome” for the participant, and (3) provide recreational opportunities for individuals, families, and seniors within local communities.
Every park and recreation agency is challenged by—and exists to address—social concerns that become the needs within the community. For example, many American communities are concerned about gang activity, risk-taking behavior of youth, single-parent households, drug and alcohol abuse, and the growing need for sport fields and other public open spaces. Obesity is a primary example of a social concern that park and recreation programs are well positioned to address. This epidemic is prevalent in children, teenagers, and adults within the United States. Researchers predict that many members of the current generation of children will die before their parents because of factors related to obesity (high cholesterol, high blood pressure, diabetes, and a lack of exercise). The National Recreation and Park Association has partnered with other organizations to provide initiatives that address obesity factors, such as Hearts N' Parks, Step Up to Health, and Teens Outside. Each initiative is a community mobilization model that is designed to assist local communities in planning, marketing, and promoting healthy lifestyles related to diet and exercise. In addition, park and recreation agencies provide a wide variety of health and fitness programs, such as swimming lessons, open swim, aerobic classes, sport skill classes, movement classes for preschoolers, weight rooms for working out, and the simple reminder to get outside and play an hour a day. Healthy lifestyles go beyond physical fitness to emotional and psychological wellness. Park and recreation agencies also provide a wide variety of art classes, dance lessons, and concerts. All these kinds of programs and services work toward a healthier community.
Parks and recreation departments provide a wide variety of programs and services to their local communities. But are these programs accomplishing what they are intended to do for the individual, the organization, or the community?
The Need for Research and Evaluation
Every park and recreation agency needs to answer a number of questions about its programs, services, and operations which are noted in figure 1.2. These questions include the following:
- Is the agency meeting the objectives of the programs?
- Is the agency meeting the needs of the customer?
- Is the agency financially stable?
These are only a small sample of the questions that a park and recreation agency needs to answer. How does a park and recreation agency go about answering these questions? The best way is to use the process of research and evaluation. Let's explore these three questions further in order to gain an understanding of the importance of evaluation and research in providing the answers.
Is the Agency Meeting the Objectives of the Programs? All programs provided by park and recreation agencies have a purpose or an objective related to the outcomes for the participants. For example, the objective of a beginning swimming class is to have the participants move through water adjustment skills and learn a basic stroke on their front and back. For the organization, the objective for swimming classes is to have more people learn to swim. This can help reduce the number of rescues or deaths at the organization's aquatic facilities. In addition, the programs should cover costs associated with the programs and generate revenues for the agency. The agency must always consider the cost and revenues generated from the programs. For the community, the objective of swimming classes could be to have a variety of opportunities available for family members to learn to swim and to enjoy the aquatic facilities year-round. Many park and recreation agencies have indoor facilities, outdoor facilities, or both. Swimming is also a physical activity that people can enjoy throughout their lifetime, either in formal classes or individually. Park and recreation agencies must work to gather the information they need to determine if the objectives of the program are being met and whether these are class objectives, agency objectives, or objectives for the community. The research and evaluation process is the primary method for gathering this information and making this assessment. This process gives recreation professionals specific information and evidence to support or negate the notion that their agencies are achieving their objectives. Without research and evaluation, professionals might have only a vague sense of whether objectives are being met, or they may have no idea at all.
Is the Agency Meeting the Needs of the Customer? What are the customer needs and expectations in relation to recreational activities? For swimming classes, the needs of the customers are generally to have a safe environment for the class and to have the class conducted by a qualified instructor. The needs of the participants' parents are to have their children learn to swim in a safe environment so that the children can be safer at pools, lakes, and beaches during the summer. The swimming classes provide one way to meet these needs. Additionally, customers have expectations about their experience or their children's experience in the program. These expectations may be associated with the registration process, how to get to the facility, cleanliness of the locker rooms, cleanliness of the facility, and the quality of the instructor. The agency must gather information to determine if the needs and expectations of the customers are being met during the program.
Is the Agency Financially Stable? Historically, park and recreation departments received a majority of their funding for operations from local taxes. The agency was considered a type of social service that needed tax-based funding for the annual operations. This perception regarding park and recreation agencies has shifted from a social service model to a business-based model. Today, park and recreation agencies are expected to operate as a business. This has resulted in departments receiving very little tax-based funding for operations and has increased the need to generate revenues from programs and services. The agency gathers financial information on a regular basis to determine if the agency is generating sufficient funds to meet the financial needs of the organization. These research activities provide valuable information to assist the managers in making decisions related to the future operations of the agency. The information gained through research and evaluation helps managers decide which programs and services need to be revised, which ones should be added, and which ones should be phased out to ensure that the agency is financially stable. The decision on whether to add or phase out a program cannot be properly made unless the managers have current information about new community needs or needs within the community that no longer exist. The best method for gathering this information is through research and evaluation.
For agencies to answer these fundamental questions concerning their programs, services, and financial operations, research and evaluation have become a necessary activity. Historically, park and recreation agencies did not regularly evaluate program outcomes for the participants. In the past, evaluation efforts centered on the agency's need to know “the numbers”—the number of participants, the hours of the programs, the revenues generated by the programs, and the expenses associated with the programs. This type of evaluation misses the essence and purpose of the park and recreation agency (i.e., meeting the program objectives for the participants and meeting the needs of the customers and community). Evaluating and researching participant outcomes, needs, and expectations are critically important to the future of every park and recreation agency. Agencies that regularly conduct evaluation and research projects gain many benefits that help to sustain the operation of the agency.
This is an excerpt from Applied Research and Evaluation Methods in Recreation.
Steps of the research process
Scientific research involves a systematic process that focuses on being objective and gathering a multitude of information for analysis so that the researcher can come to a conclusion.
Scientific research involves a systematic process that focuses on being objective and gathering a multitude of information for analysis so that the researcher can come to a conclusion. This process is used in all research and evaluation projects, regardless of the research method (scientific method of inquiry, evaluation research, or action research). The process focuses on testing hunches or ideas in a park and recreation setting through a systematic process. In this process, the study is documented in such a way that another individual can conduct the same study again. This is referred to as replicating the study. Any research done without documenting the study so that others can review the process and results is not an investigation using the scientific research process. The scientific research process is a multiple-step process where the steps are interlinked with the other steps in the process. If changes are made in one step of the process, the researcher must review all the other steps to ensure that the changes are reflected throughout the process. Parks and recreation professionals are often involved in conducting research or evaluation projects within the agency. These professionals need to understand the eight steps of the research process as they apply to conducting a study. Table 2.4 lists the steps of the research process and provides an example of each step for a sample research study.
Step 1: Identify the Problem
The first step in the process is to identify a problem or develop a research question. The research problem may be something the agency identifies as a problem, some knowledge or information that is needed by the agency, or the desire to identify a recreation trend nationally. In the example in table 2.4, the problem that the agency has identified is childhood obesity, which is a local problem and concern within the community. This serves as the focus of the study.
Step 2: Review the Literature
Now that the problem has been identified, the researcher must learn more about the topic under investigation. To do this, the researcher must review the literature related to the research problem. This step provides foundational knowledge about the problem area. The review of literature also educates the researcher about what studies have been conducted in the past, how these studies were conducted, and the conclusions in the problem area. In the obesity study, the review of literature enables the programmer to discover horrifying statistics related to the long-term effects of childhood obesity in terms of health issues, death rates, and projected medical costs. In addition, the programmer finds several articles and information from the Centers for Disease Control and Prevention that describe the benefits of walking 10,000 steps a day. The information discovered during this step helps the programmer fully understand the magnitude of the problem, recognize the future consequences of obesity, and identify a strategy to combat obesity (i.e., walking).
Step 3: Clarify the Problem
Many times the initial problem identified in the first step of the process is too large or broad in scope. In step 3 of the process, the researcher clarifies the problem and narrows the scope of the study. This can only be done after the literature has been reviewed. The knowledge gained through the review of literature guides the researcher in clarifying and narrowing the research project. In the example, the programmer has identified childhood obesity as the problem and the purpose of the study. This topic is very broad and could be studied based on genetics, family environment, diet, exercise, self-confidence, leisure activities, or health issues. All of these areas cannot be investigated in a single study; therefore, the problem and purpose of the study must be more clearly defined. The programmer has decided that the purpose of the study is to determine if walking 10,000 steps a day for three days a week will improve the individual's health. This purpose is more narrowly focused and researchable than the original problem.
Step 4: Clearly Define Terms and Concepts
Terms and concepts are words or phrases used in the purpose statement of the study or the description of the study. These items need to be specifically defined as they apply to the study. Terms or concepts often have different definitions depending on who is reading the study. To minimize confusion about what the terms and phrases mean, the researcher must specifically define them for the study. In the obesity study, the concept of “individual's health” can be defined in hundreds of ways, such as physical, mental, emotional, or spiritual health. For this study, the individual's health is defined as physical health. The concept of physical health may also be defined and measured in many ways. In this case, the programmer decides to more narrowly define “individual health” to refer to the areas of weight, percentage of body fat, and cholesterol. By defining the terms or concepts more narrowly, the scope of the study is more manageable for the programmer, making it easier to collect the necessary data for the study. This also makes the concepts more understandable to the reader.
Step 5: Define the Population
Research projects can focus on a specific group of people, facilities, park development, employee evaluations, programs, financial status, marketing efforts, or the integration of technology into the operations. For example, if a researcher wants to examine a specific group of people in the community, the study could examine a specific age group, males or females, people living in a specific geographic area, or a specific ethnic group. Literally thousands of options are available to the researcher to specifically identify the group to study. The research problem and the purpose of the study assist the researcher in identifying the group to involve in the study. In research terms, the group to involve in the study is always called the population. Defining the population assists the researcher in several ways. First, it narrows the scope of the study from a very large population to one that is manageable. Second, the population identifies the group that the researcher's efforts will be focused on within the study. This helps ensure that the researcher stays on the right path during the study. Finally, by defining the population, the researcher identifies the group that the results will apply to at the conclusion of the study. In the example in table 2.4, the programmer has identified the population of the study as children ages 10 to 12 years. This narrower population makes the study more manageable in terms of time and resources.
Step 6: Develop the Instrumentation Plan
The plan for the study is referred to as the instrumentation plan. The instrumentation plan serves as the road map for the entire study, specifying who will participate in the study; how, when, and where data will be collected; and the content of the program. This plan is composed of numerous decisions and considerations that are addressed in chapter 8 of this text. In the obesity study, the researcher has decided to have the children participate in a walking program for six months. The group of participants is called the sample, which is a smaller group selected from the population specified for the study. The study cannot possibly include every 10- to 12-year-old child in the community, so a smaller group is used to represent the population. The researcher develops the plan for the walking program, indicating what data will be collected, when and how the data will be collected, who will collect the data, and how the data will be analyzed. The instrumentation plan specifies all the steps that must be completed for the study. This ensures that the programmer has carefully thought through all these decisions and that she provides a step-by-step plan to be followed in the study.
Step 7: Collect Data
Once the instrumentation plan is completed, the actual study begins with the collection of data. The collection of data is a critical step in providing the information needed to answer the research question. Every study includes the collection of some type of data—whether it is from the literature or from subjects—to answer the research question. Data can be collected in the form of words on a survey, with a questionnaire, through observations, or from the literature. In the obesity study, the programmers will be collecting data on the defined variables: weight, percentage of body fat, cholesterol levels, and the number of days the person walked a total of 10,000 steps during the class.
The researcher collects these data at the first session and at the last session of the program. These two sets of data are necessary to determine the effect of the walking program on weight, body fat, and cholesterol level. Once the data are collected on the variables, the researcher is ready to move to the final step of the process, which is the data analysis.
Step 8: Analyze the Data
All the time, effort, and resources dedicated to steps 1 through 7 of the research process culminate in this final step. The researcher finally has data to analyze so that the research question can be answered. In the instrumentation plan, the researcher specified how the data will be analyzed. The researcher now analyzes the data according to the plan. The results of this analysis are then reviewed and summarized in a manner directly related to the research questions. In the obesity study, the researcher compares the measurements of weight, percentage of body fat, and cholesterol that were taken at the first meeting of the subjects to the measurements of the same variables at the final program session. These two sets of data will be analyzed to determine if there was a difference between the first measurement and the second measurement for each individual in the program. Then, the data will be analyzed to determine if the differences are statistically significant. If the differences are statistically significant, the study validates the theory that was the focus of the study. The results of the study also provide valuable information about one strategy to combat childhood obesity in the community.
As you have probably concluded, conducting studies using the eight steps of the scientific research process requires you to dedicate time and effort to the planning process. You cannot conduct a study using the scientific research process when time is limited or the study is done at the last minute. Researchers who do this conduct studies that result in either false conclusions or conclusions that are not of any value to the organization.
This is an excerpt from Applied Research and Evaluation Methods in Recreation.
Creating instrumentation plans for research studies
The instrumentation plan helps guide the progress of the study to the ultimate goal of gathering data and formulating conclusions to answer the research question.
The instrumentation plan is composed of a number of decisions that need to be made before beginning the study. These decisions are made to determine
what data are needed to answer the research questions,
- how to gather the data,
- when to gather the data,
- where to gather the data, and
- how to analyze the data.
These decisions must be made as part of the instrumentation plan for the study. They help guide the progress of the study to the ultimate goal of gathering data and formulating conclusions to answer the research question.
Selecting Data
The previous decisions help define what data need to be gathered and why the data are important. What is data? Data refers to the information that is gathered to answer the research question. Data can be numbers, words, or actual objects, such as photos, articles, or video. The instrumentation process defines what data need to be collected and the timing of the data collection process.
What data are needed if the researcher is examining the relationship between taking swimming lessons and purchasing a summer session pass? The researcher needs the records of swimming lesson participants and season pass holders. Examining any other type of data, such as instructor evaluations, is a waste of time if it does not focus on answering the research question. If the researcher is examining the factors that lead a person to purchase a summer pass for the pool, interview data are needed to fully understand the factors in the decision-making process of the customer. By using the interview process, the researcher can conduct a two-way conversation in order to explore the factors that lead to purchasing the season pass. This type of data is very comprehensive in exploring the decision-making process, and it is more effective than looking at only one or two variables. In the example of the fitness center evaluating the personal trainers, a written survey will provide the data needed to evaluate the performances of the trainers. By identifying the specific type of data needed to answer the research question, the researchers' efforts are properly focused. Table 8.3 summarizes what data are needed in relation to the previous decisions in the instrumentation plan.
Gathering Data
The next natural question is, how will the data be collected? To answer this question, the researcher needs to identify whether the study is an evaluation, a quantitative study, or a qualitative study. This helps determine how to collect data. What instrument will be used to collect data? The researcher has a variety of options that may be used as a data collection instrument, such as surveys, interviews, observations, or rating instruments. Table 8.4 summarizes the decision of how to gather data along with the previous pieces of the instrumentation plan.
Determining When to Collect Data
Once the researcher decides what data to collect and how to collect them, the researcher must determine when to collect the data, where to collect them, and who should collect them. The results of the study can be influenced by the time that data are gathered, where the data are gathered, and who gathers the data. Studies that only use literature or the records of the agency are not influenced by these decisions. This is the case with the first research question concerning the relationship between taking swimming lessons and purchasing a summer season pass. Once the majority of season passes have been sold, the data should be gathered at the beginning of the summer.
This is not the case when data are being gathered from human subjects. Timing considerations that must be addressed are the month, the day of the week, and the time of day that the data are collected. In the example of identifying what factors lead a person to purchase a season pass to the pool, the interview should be conducted as soon as possible after the purchase of the pass. If the researchers wait weeks or months after the purchase, the subjects may not recall what factors led them to make the purchase. This interview could be done over the phone or in a small group.
For the fitness center evaluating the personal trainers, a specific time frame is necessary in order to ensure a high response rate. The manager decides to give the clients an evaluation form to complete after their last session with their personal trainer (before they leave the facility). This is a better plan than mailing the surveys to the clients' home weeks after the clients' last session with their trainer. The timing of data collection can affect the quality of the data received from the subjects. Table 8.5 presents this information, along with the previous decisions in the instrumentation plan.
Determining Where to Collect Data
The place that data are collected and the person collecting data must be specifically defined in the instrumentation plan. The best plan is to standardize the place and person for the data collection. This standardization helps enhance the truthfulness and validity of the data. The subjects should be in an environment where they feel at ease so that they will answer questions honestly. These two considerations are not an issue for the first research question because the data are coming from the documents of the agency and not from individuals.
In the case of identifying the factors that lead to purchasing a season pass for the pool, the interviews will be conducted by telephone. The people conducting the interviews should be trained in how to conduct an interview and how to record the information accurately. Each person being interviewed will be at home while participating in the interview, which is a comfortable environment for that individual.
In the example of evaluating personal trainers, having the clients' trainer administer the survey in the gym will most likely provide invalid data. The ideal situation would be to have one staff person provide the clients with a quiet room to complete the survey. Then the subject should return the survey to the same staff member who gave the survey to the subject. This standardizes where the data are collected and by whom. Table 8.6 provides a summary of all the instrument pieces and provides examples of each decision that needs to be made in developing the instrumentation plan for a study.
Analyzing Data
The next planning decision to make is how to analyze the data and what to do with the information once the analysis is completed. The type of analysis used with data is determined by whether the data are quantitative or qualitative data. To analyze quantitative data, some type of statistical analysis is used to provide the results. The type of statistical analysis used with data must be thought out and documented in the instrumentation plan.
Some of the most frequently used options for statistical analysis will be covered later in this text. Qualitative data are analyzed through a coding process that identifies themes; these themes become the foundation for the conclusions of the study. This type of data analysis will also be covered later in this text. The results of the data analysis should be summarized and presented in a report to supervisors and other parties for review. Research and evaluation efforts yield a wealth of information that can be used to educate commissioners, city councils, customers, and other decision makers. Studies that document outcomes of the programs and benefits to the community serve as a powerful tool that allows the agency to document its benefits to the community through facts and data, not perceptions and speculations. In today's cost-conscious society, the agency must prove that it is contributing to the quality of life of the community and must document the outcomes from the programs. This type of documentation and evidence is also a requirement for CAPRA accredited agencies and represents an ongoing evaluation process within the agency. Table 8.7 summarizes the key decisions that need to be made in the development of the instrumentation plan.
This is an excerpt from Applied Research and Evaluation Methods in Recreation.
The importance of research and evaluation in recreation
Parks and recreation departments provide a wide variety of programs and services to their local communities. But are these programs accomplishing what they are intended to do for the individual, the organization, or the community?
Foundational Knowledge for Researchers
To become comfortable with research and evaluation, you first need to understand the following:
- The mission and purpose of parks and recreation
- The need for research and evaluation
- How research and evaluation assist an organization
- The expectations for evaluation and research within the industry related to professional certification and agency accreditation
Mission and Purpose of Recreation Agencies
In the United States, the field of recreation and parks has deep roots grounded within societal needs, community needs, and individual needs (see figure 1.1). As you probably remember, the original needs of the community emerged during the Industrial Revolution. With the population shifting from rural areas to urban centers, children needed a safe place to play and socialize. The scope of facilities and services expanded from that point to include providing public parks, public recreation facilities, and instructional classes. The focus of this expansion was to (1) provide places for people to play, (2) offer programs that were educational and considered “wholesome” for the participant, and (3) provide recreational opportunities for individuals, families, and seniors within local communities.
Every park and recreation agency is challenged by—and exists to address—social concerns that become the needs within the community. For example, many American communities are concerned about gang activity, risk-taking behavior of youth, single-parent households, drug and alcohol abuse, and the growing need for sport fields and other public open spaces. Obesity is a primary example of a social concern that park and recreation programs are well positioned to address. This epidemic is prevalent in children, teenagers, and adults within the United States. Researchers predict that many members of the current generation of children will die before their parents because of factors related to obesity (high cholesterol, high blood pressure, diabetes, and a lack of exercise). The National Recreation and Park Association has partnered with other organizations to provide initiatives that address obesity factors, such as Hearts N' Parks, Step Up to Health, and Teens Outside. Each initiative is a community mobilization model that is designed to assist local communities in planning, marketing, and promoting healthy lifestyles related to diet and exercise. In addition, park and recreation agencies provide a wide variety of health and fitness programs, such as swimming lessons, open swim, aerobic classes, sport skill classes, movement classes for preschoolers, weight rooms for working out, and the simple reminder to get outside and play an hour a day. Healthy lifestyles go beyond physical fitness to emotional and psychological wellness. Park and recreation agencies also provide a wide variety of art classes, dance lessons, and concerts. All these kinds of programs and services work toward a healthier community.
Parks and recreation departments provide a wide variety of programs and services to their local communities. But are these programs accomplishing what they are intended to do for the individual, the organization, or the community?
The Need for Research and Evaluation
Every park and recreation agency needs to answer a number of questions about its programs, services, and operations which are noted in figure 1.2. These questions include the following:
- Is the agency meeting the objectives of the programs?
- Is the agency meeting the needs of the customer?
- Is the agency financially stable?
These are only a small sample of the questions that a park and recreation agency needs to answer. How does a park and recreation agency go about answering these questions? The best way is to use the process of research and evaluation. Let's explore these three questions further in order to gain an understanding of the importance of evaluation and research in providing the answers.
Is the Agency Meeting the Objectives of the Programs? All programs provided by park and recreation agencies have a purpose or an objective related to the outcomes for the participants. For example, the objective of a beginning swimming class is to have the participants move through water adjustment skills and learn a basic stroke on their front and back. For the organization, the objective for swimming classes is to have more people learn to swim. This can help reduce the number of rescues or deaths at the organization's aquatic facilities. In addition, the programs should cover costs associated with the programs and generate revenues for the agency. The agency must always consider the cost and revenues generated from the programs. For the community, the objective of swimming classes could be to have a variety of opportunities available for family members to learn to swim and to enjoy the aquatic facilities year-round. Many park and recreation agencies have indoor facilities, outdoor facilities, or both. Swimming is also a physical activity that people can enjoy throughout their lifetime, either in formal classes or individually. Park and recreation agencies must work to gather the information they need to determine if the objectives of the program are being met and whether these are class objectives, agency objectives, or objectives for the community. The research and evaluation process is the primary method for gathering this information and making this assessment. This process gives recreation professionals specific information and evidence to support or negate the notion that their agencies are achieving their objectives. Without research and evaluation, professionals might have only a vague sense of whether objectives are being met, or they may have no idea at all.
Is the Agency Meeting the Needs of the Customer? What are the customer needs and expectations in relation to recreational activities? For swimming classes, the needs of the customers are generally to have a safe environment for the class and to have the class conducted by a qualified instructor. The needs of the participants' parents are to have their children learn to swim in a safe environment so that the children can be safer at pools, lakes, and beaches during the summer. The swimming classes provide one way to meet these needs. Additionally, customers have expectations about their experience or their children's experience in the program. These expectations may be associated with the registration process, how to get to the facility, cleanliness of the locker rooms, cleanliness of the facility, and the quality of the instructor. The agency must gather information to determine if the needs and expectations of the customers are being met during the program.
Is the Agency Financially Stable? Historically, park and recreation departments received a majority of their funding for operations from local taxes. The agency was considered a type of social service that needed tax-based funding for the annual operations. This perception regarding park and recreation agencies has shifted from a social service model to a business-based model. Today, park and recreation agencies are expected to operate as a business. This has resulted in departments receiving very little tax-based funding for operations and has increased the need to generate revenues from programs and services. The agency gathers financial information on a regular basis to determine if the agency is generating sufficient funds to meet the financial needs of the organization. These research activities provide valuable information to assist the managers in making decisions related to the future operations of the agency. The information gained through research and evaluation helps managers decide which programs and services need to be revised, which ones should be added, and which ones should be phased out to ensure that the agency is financially stable. The decision on whether to add or phase out a program cannot be properly made unless the managers have current information about new community needs or needs within the community that no longer exist. The best method for gathering this information is through research and evaluation.
For agencies to answer these fundamental questions concerning their programs, services, and financial operations, research and evaluation have become a necessary activity. Historically, park and recreation agencies did not regularly evaluate program outcomes for the participants. In the past, evaluation efforts centered on the agency's need to know “the numbers”—the number of participants, the hours of the programs, the revenues generated by the programs, and the expenses associated with the programs. This type of evaluation misses the essence and purpose of the park and recreation agency (i.e., meeting the program objectives for the participants and meeting the needs of the customers and community). Evaluating and researching participant outcomes, needs, and expectations are critically important to the future of every park and recreation agency. Agencies that regularly conduct evaluation and research projects gain many benefits that help to sustain the operation of the agency.
This is an excerpt from Applied Research and Evaluation Methods in Recreation.
Steps of the research process
Scientific research involves a systematic process that focuses on being objective and gathering a multitude of information for analysis so that the researcher can come to a conclusion.
Scientific research involves a systematic process that focuses on being objective and gathering a multitude of information for analysis so that the researcher can come to a conclusion. This process is used in all research and evaluation projects, regardless of the research method (scientific method of inquiry, evaluation research, or action research). The process focuses on testing hunches or ideas in a park and recreation setting through a systematic process. In this process, the study is documented in such a way that another individual can conduct the same study again. This is referred to as replicating the study. Any research done without documenting the study so that others can review the process and results is not an investigation using the scientific research process. The scientific research process is a multiple-step process where the steps are interlinked with the other steps in the process. If changes are made in one step of the process, the researcher must review all the other steps to ensure that the changes are reflected throughout the process. Parks and recreation professionals are often involved in conducting research or evaluation projects within the agency. These professionals need to understand the eight steps of the research process as they apply to conducting a study. Table 2.4 lists the steps of the research process and provides an example of each step for a sample research study.
Step 1: Identify the Problem
The first step in the process is to identify a problem or develop a research question. The research problem may be something the agency identifies as a problem, some knowledge or information that is needed by the agency, or the desire to identify a recreation trend nationally. In the example in table 2.4, the problem that the agency has identified is childhood obesity, which is a local problem and concern within the community. This serves as the focus of the study.
Step 2: Review the Literature
Now that the problem has been identified, the researcher must learn more about the topic under investigation. To do this, the researcher must review the literature related to the research problem. This step provides foundational knowledge about the problem area. The review of literature also educates the researcher about what studies have been conducted in the past, how these studies were conducted, and the conclusions in the problem area. In the obesity study, the review of literature enables the programmer to discover horrifying statistics related to the long-term effects of childhood obesity in terms of health issues, death rates, and projected medical costs. In addition, the programmer finds several articles and information from the Centers for Disease Control and Prevention that describe the benefits of walking 10,000 steps a day. The information discovered during this step helps the programmer fully understand the magnitude of the problem, recognize the future consequences of obesity, and identify a strategy to combat obesity (i.e., walking).
Step 3: Clarify the Problem
Many times the initial problem identified in the first step of the process is too large or broad in scope. In step 3 of the process, the researcher clarifies the problem and narrows the scope of the study. This can only be done after the literature has been reviewed. The knowledge gained through the review of literature guides the researcher in clarifying and narrowing the research project. In the example, the programmer has identified childhood obesity as the problem and the purpose of the study. This topic is very broad and could be studied based on genetics, family environment, diet, exercise, self-confidence, leisure activities, or health issues. All of these areas cannot be investigated in a single study; therefore, the problem and purpose of the study must be more clearly defined. The programmer has decided that the purpose of the study is to determine if walking 10,000 steps a day for three days a week will improve the individual's health. This purpose is more narrowly focused and researchable than the original problem.
Step 4: Clearly Define Terms and Concepts
Terms and concepts are words or phrases used in the purpose statement of the study or the description of the study. These items need to be specifically defined as they apply to the study. Terms or concepts often have different definitions depending on who is reading the study. To minimize confusion about what the terms and phrases mean, the researcher must specifically define them for the study. In the obesity study, the concept of “individual's health” can be defined in hundreds of ways, such as physical, mental, emotional, or spiritual health. For this study, the individual's health is defined as physical health. The concept of physical health may also be defined and measured in many ways. In this case, the programmer decides to more narrowly define “individual health” to refer to the areas of weight, percentage of body fat, and cholesterol. By defining the terms or concepts more narrowly, the scope of the study is more manageable for the programmer, making it easier to collect the necessary data for the study. This also makes the concepts more understandable to the reader.
Step 5: Define the Population
Research projects can focus on a specific group of people, facilities, park development, employee evaluations, programs, financial status, marketing efforts, or the integration of technology into the operations. For example, if a researcher wants to examine a specific group of people in the community, the study could examine a specific age group, males or females, people living in a specific geographic area, or a specific ethnic group. Literally thousands of options are available to the researcher to specifically identify the group to study. The research problem and the purpose of the study assist the researcher in identifying the group to involve in the study. In research terms, the group to involve in the study is always called the population. Defining the population assists the researcher in several ways. First, it narrows the scope of the study from a very large population to one that is manageable. Second, the population identifies the group that the researcher's efforts will be focused on within the study. This helps ensure that the researcher stays on the right path during the study. Finally, by defining the population, the researcher identifies the group that the results will apply to at the conclusion of the study. In the example in table 2.4, the programmer has identified the population of the study as children ages 10 to 12 years. This narrower population makes the study more manageable in terms of time and resources.
Step 6: Develop the Instrumentation Plan
The plan for the study is referred to as the instrumentation plan. The instrumentation plan serves as the road map for the entire study, specifying who will participate in the study; how, when, and where data will be collected; and the content of the program. This plan is composed of numerous decisions and considerations that are addressed in chapter 8 of this text. In the obesity study, the researcher has decided to have the children participate in a walking program for six months. The group of participants is called the sample, which is a smaller group selected from the population specified for the study. The study cannot possibly include every 10- to 12-year-old child in the community, so a smaller group is used to represent the population. The researcher develops the plan for the walking program, indicating what data will be collected, when and how the data will be collected, who will collect the data, and how the data will be analyzed. The instrumentation plan specifies all the steps that must be completed for the study. This ensures that the programmer has carefully thought through all these decisions and that she provides a step-by-step plan to be followed in the study.
Step 7: Collect Data
Once the instrumentation plan is completed, the actual study begins with the collection of data. The collection of data is a critical step in providing the information needed to answer the research question. Every study includes the collection of some type of data—whether it is from the literature or from subjects—to answer the research question. Data can be collected in the form of words on a survey, with a questionnaire, through observations, or from the literature. In the obesity study, the programmers will be collecting data on the defined variables: weight, percentage of body fat, cholesterol levels, and the number of days the person walked a total of 10,000 steps during the class.
The researcher collects these data at the first session and at the last session of the program. These two sets of data are necessary to determine the effect of the walking program on weight, body fat, and cholesterol level. Once the data are collected on the variables, the researcher is ready to move to the final step of the process, which is the data analysis.
Step 8: Analyze the Data
All the time, effort, and resources dedicated to steps 1 through 7 of the research process culminate in this final step. The researcher finally has data to analyze so that the research question can be answered. In the instrumentation plan, the researcher specified how the data will be analyzed. The researcher now analyzes the data according to the plan. The results of this analysis are then reviewed and summarized in a manner directly related to the research questions. In the obesity study, the researcher compares the measurements of weight, percentage of body fat, and cholesterol that were taken at the first meeting of the subjects to the measurements of the same variables at the final program session. These two sets of data will be analyzed to determine if there was a difference between the first measurement and the second measurement for each individual in the program. Then, the data will be analyzed to determine if the differences are statistically significant. If the differences are statistically significant, the study validates the theory that was the focus of the study. The results of the study also provide valuable information about one strategy to combat childhood obesity in the community.
As you have probably concluded, conducting studies using the eight steps of the scientific research process requires you to dedicate time and effort to the planning process. You cannot conduct a study using the scientific research process when time is limited or the study is done at the last minute. Researchers who do this conduct studies that result in either false conclusions or conclusions that are not of any value to the organization.
This is an excerpt from Applied Research and Evaluation Methods in Recreation.
Creating instrumentation plans for research studies
The instrumentation plan helps guide the progress of the study to the ultimate goal of gathering data and formulating conclusions to answer the research question.
The instrumentation plan is composed of a number of decisions that need to be made before beginning the study. These decisions are made to determine
what data are needed to answer the research questions,
- how to gather the data,
- when to gather the data,
- where to gather the data, and
- how to analyze the data.
These decisions must be made as part of the instrumentation plan for the study. They help guide the progress of the study to the ultimate goal of gathering data and formulating conclusions to answer the research question.
Selecting Data
The previous decisions help define what data need to be gathered and why the data are important. What is data? Data refers to the information that is gathered to answer the research question. Data can be numbers, words, or actual objects, such as photos, articles, or video. The instrumentation process defines what data need to be collected and the timing of the data collection process.
What data are needed if the researcher is examining the relationship between taking swimming lessons and purchasing a summer session pass? The researcher needs the records of swimming lesson participants and season pass holders. Examining any other type of data, such as instructor evaluations, is a waste of time if it does not focus on answering the research question. If the researcher is examining the factors that lead a person to purchase a summer pass for the pool, interview data are needed to fully understand the factors in the decision-making process of the customer. By using the interview process, the researcher can conduct a two-way conversation in order to explore the factors that lead to purchasing the season pass. This type of data is very comprehensive in exploring the decision-making process, and it is more effective than looking at only one or two variables. In the example of the fitness center evaluating the personal trainers, a written survey will provide the data needed to evaluate the performances of the trainers. By identifying the specific type of data needed to answer the research question, the researchers' efforts are properly focused. Table 8.3 summarizes what data are needed in relation to the previous decisions in the instrumentation plan.
Gathering Data
The next natural question is, how will the data be collected? To answer this question, the researcher needs to identify whether the study is an evaluation, a quantitative study, or a qualitative study. This helps determine how to collect data. What instrument will be used to collect data? The researcher has a variety of options that may be used as a data collection instrument, such as surveys, interviews, observations, or rating instruments. Table 8.4 summarizes the decision of how to gather data along with the previous pieces of the instrumentation plan.
Determining When to Collect Data
Once the researcher decides what data to collect and how to collect them, the researcher must determine when to collect the data, where to collect them, and who should collect them. The results of the study can be influenced by the time that data are gathered, where the data are gathered, and who gathers the data. Studies that only use literature or the records of the agency are not influenced by these decisions. This is the case with the first research question concerning the relationship between taking swimming lessons and purchasing a summer season pass. Once the majority of season passes have been sold, the data should be gathered at the beginning of the summer.
This is not the case when data are being gathered from human subjects. Timing considerations that must be addressed are the month, the day of the week, and the time of day that the data are collected. In the example of identifying what factors lead a person to purchase a season pass to the pool, the interview should be conducted as soon as possible after the purchase of the pass. If the researchers wait weeks or months after the purchase, the subjects may not recall what factors led them to make the purchase. This interview could be done over the phone or in a small group.
For the fitness center evaluating the personal trainers, a specific time frame is necessary in order to ensure a high response rate. The manager decides to give the clients an evaluation form to complete after their last session with their personal trainer (before they leave the facility). This is a better plan than mailing the surveys to the clients' home weeks after the clients' last session with their trainer. The timing of data collection can affect the quality of the data received from the subjects. Table 8.5 presents this information, along with the previous decisions in the instrumentation plan.
Determining Where to Collect Data
The place that data are collected and the person collecting data must be specifically defined in the instrumentation plan. The best plan is to standardize the place and person for the data collection. This standardization helps enhance the truthfulness and validity of the data. The subjects should be in an environment where they feel at ease so that they will answer questions honestly. These two considerations are not an issue for the first research question because the data are coming from the documents of the agency and not from individuals.
In the case of identifying the factors that lead to purchasing a season pass for the pool, the interviews will be conducted by telephone. The people conducting the interviews should be trained in how to conduct an interview and how to record the information accurately. Each person being interviewed will be at home while participating in the interview, which is a comfortable environment for that individual.
In the example of evaluating personal trainers, having the clients' trainer administer the survey in the gym will most likely provide invalid data. The ideal situation would be to have one staff person provide the clients with a quiet room to complete the survey. Then the subject should return the survey to the same staff member who gave the survey to the subject. This standardizes where the data are collected and by whom. Table 8.6 provides a summary of all the instrument pieces and provides examples of each decision that needs to be made in developing the instrumentation plan for a study.
Analyzing Data
The next planning decision to make is how to analyze the data and what to do with the information once the analysis is completed. The type of analysis used with data is determined by whether the data are quantitative or qualitative data. To analyze quantitative data, some type of statistical analysis is used to provide the results. The type of statistical analysis used with data must be thought out and documented in the instrumentation plan.
Some of the most frequently used options for statistical analysis will be covered later in this text. Qualitative data are analyzed through a coding process that identifies themes; these themes become the foundation for the conclusions of the study. This type of data analysis will also be covered later in this text. The results of the data analysis should be summarized and presented in a report to supervisors and other parties for review. Research and evaluation efforts yield a wealth of information that can be used to educate commissioners, city councils, customers, and other decision makers. Studies that document outcomes of the programs and benefits to the community serve as a powerful tool that allows the agency to document its benefits to the community through facts and data, not perceptions and speculations. In today's cost-conscious society, the agency must prove that it is contributing to the quality of life of the community and must document the outcomes from the programs. This type of documentation and evidence is also a requirement for CAPRA accredited agencies and represents an ongoing evaluation process within the agency. Table 8.7 summarizes the key decisions that need to be made in the development of the instrumentation plan.
This is an excerpt from Applied Research and Evaluation Methods in Recreation.
The importance of research and evaluation in recreation
Parks and recreation departments provide a wide variety of programs and services to their local communities. But are these programs accomplishing what they are intended to do for the individual, the organization, or the community?
Foundational Knowledge for Researchers
To become comfortable with research and evaluation, you first need to understand the following:
- The mission and purpose of parks and recreation
- The need for research and evaluation
- How research and evaluation assist an organization
- The expectations for evaluation and research within the industry related to professional certification and agency accreditation
Mission and Purpose of Recreation Agencies
In the United States, the field of recreation and parks has deep roots grounded within societal needs, community needs, and individual needs (see figure 1.1). As you probably remember, the original needs of the community emerged during the Industrial Revolution. With the population shifting from rural areas to urban centers, children needed a safe place to play and socialize. The scope of facilities and services expanded from that point to include providing public parks, public recreation facilities, and instructional classes. The focus of this expansion was to (1) provide places for people to play, (2) offer programs that were educational and considered “wholesome” for the participant, and (3) provide recreational opportunities for individuals, families, and seniors within local communities.
Every park and recreation agency is challenged by—and exists to address—social concerns that become the needs within the community. For example, many American communities are concerned about gang activity, risk-taking behavior of youth, single-parent households, drug and alcohol abuse, and the growing need for sport fields and other public open spaces. Obesity is a primary example of a social concern that park and recreation programs are well positioned to address. This epidemic is prevalent in children, teenagers, and adults within the United States. Researchers predict that many members of the current generation of children will die before their parents because of factors related to obesity (high cholesterol, high blood pressure, diabetes, and a lack of exercise). The National Recreation and Park Association has partnered with other organizations to provide initiatives that address obesity factors, such as Hearts N' Parks, Step Up to Health, and Teens Outside. Each initiative is a community mobilization model that is designed to assist local communities in planning, marketing, and promoting healthy lifestyles related to diet and exercise. In addition, park and recreation agencies provide a wide variety of health and fitness programs, such as swimming lessons, open swim, aerobic classes, sport skill classes, movement classes for preschoolers, weight rooms for working out, and the simple reminder to get outside and play an hour a day. Healthy lifestyles go beyond physical fitness to emotional and psychological wellness. Park and recreation agencies also provide a wide variety of art classes, dance lessons, and concerts. All these kinds of programs and services work toward a healthier community.
Parks and recreation departments provide a wide variety of programs and services to their local communities. But are these programs accomplishing what they are intended to do for the individual, the organization, or the community?
The Need for Research and Evaluation
Every park and recreation agency needs to answer a number of questions about its programs, services, and operations which are noted in figure 1.2. These questions include the following:
- Is the agency meeting the objectives of the programs?
- Is the agency meeting the needs of the customer?
- Is the agency financially stable?
These are only a small sample of the questions that a park and recreation agency needs to answer. How does a park and recreation agency go about answering these questions? The best way is to use the process of research and evaluation. Let's explore these three questions further in order to gain an understanding of the importance of evaluation and research in providing the answers.
Is the Agency Meeting the Objectives of the Programs? All programs provided by park and recreation agencies have a purpose or an objective related to the outcomes for the participants. For example, the objective of a beginning swimming class is to have the participants move through water adjustment skills and learn a basic stroke on their front and back. For the organization, the objective for swimming classes is to have more people learn to swim. This can help reduce the number of rescues or deaths at the organization's aquatic facilities. In addition, the programs should cover costs associated with the programs and generate revenues for the agency. The agency must always consider the cost and revenues generated from the programs. For the community, the objective of swimming classes could be to have a variety of opportunities available for family members to learn to swim and to enjoy the aquatic facilities year-round. Many park and recreation agencies have indoor facilities, outdoor facilities, or both. Swimming is also a physical activity that people can enjoy throughout their lifetime, either in formal classes or individually. Park and recreation agencies must work to gather the information they need to determine if the objectives of the program are being met and whether these are class objectives, agency objectives, or objectives for the community. The research and evaluation process is the primary method for gathering this information and making this assessment. This process gives recreation professionals specific information and evidence to support or negate the notion that their agencies are achieving their objectives. Without research and evaluation, professionals might have only a vague sense of whether objectives are being met, or they may have no idea at all.
Is the Agency Meeting the Needs of the Customer? What are the customer needs and expectations in relation to recreational activities? For swimming classes, the needs of the customers are generally to have a safe environment for the class and to have the class conducted by a qualified instructor. The needs of the participants' parents are to have their children learn to swim in a safe environment so that the children can be safer at pools, lakes, and beaches during the summer. The swimming classes provide one way to meet these needs. Additionally, customers have expectations about their experience or their children's experience in the program. These expectations may be associated with the registration process, how to get to the facility, cleanliness of the locker rooms, cleanliness of the facility, and the quality of the instructor. The agency must gather information to determine if the needs and expectations of the customers are being met during the program.
Is the Agency Financially Stable? Historically, park and recreation departments received a majority of their funding for operations from local taxes. The agency was considered a type of social service that needed tax-based funding for the annual operations. This perception regarding park and recreation agencies has shifted from a social service model to a business-based model. Today, park and recreation agencies are expected to operate as a business. This has resulted in departments receiving very little tax-based funding for operations and has increased the need to generate revenues from programs and services. The agency gathers financial information on a regular basis to determine if the agency is generating sufficient funds to meet the financial needs of the organization. These research activities provide valuable information to assist the managers in making decisions related to the future operations of the agency. The information gained through research and evaluation helps managers decide which programs and services need to be revised, which ones should be added, and which ones should be phased out to ensure that the agency is financially stable. The decision on whether to add or phase out a program cannot be properly made unless the managers have current information about new community needs or needs within the community that no longer exist. The best method for gathering this information is through research and evaluation.
For agencies to answer these fundamental questions concerning their programs, services, and financial operations, research and evaluation have become a necessary activity. Historically, park and recreation agencies did not regularly evaluate program outcomes for the participants. In the past, evaluation efforts centered on the agency's need to know “the numbers”—the number of participants, the hours of the programs, the revenues generated by the programs, and the expenses associated with the programs. This type of evaluation misses the essence and purpose of the park and recreation agency (i.e., meeting the program objectives for the participants and meeting the needs of the customers and community). Evaluating and researching participant outcomes, needs, and expectations are critically important to the future of every park and recreation agency. Agencies that regularly conduct evaluation and research projects gain many benefits that help to sustain the operation of the agency.
This is an excerpt from Applied Research and Evaluation Methods in Recreation.
Steps of the research process
Scientific research involves a systematic process that focuses on being objective and gathering a multitude of information for analysis so that the researcher can come to a conclusion.
Scientific research involves a systematic process that focuses on being objective and gathering a multitude of information for analysis so that the researcher can come to a conclusion. This process is used in all research and evaluation projects, regardless of the research method (scientific method of inquiry, evaluation research, or action research). The process focuses on testing hunches or ideas in a park and recreation setting through a systematic process. In this process, the study is documented in such a way that another individual can conduct the same study again. This is referred to as replicating the study. Any research done without documenting the study so that others can review the process and results is not an investigation using the scientific research process. The scientific research process is a multiple-step process where the steps are interlinked with the other steps in the process. If changes are made in one step of the process, the researcher must review all the other steps to ensure that the changes are reflected throughout the process. Parks and recreation professionals are often involved in conducting research or evaluation projects within the agency. These professionals need to understand the eight steps of the research process as they apply to conducting a study. Table 2.4 lists the steps of the research process and provides an example of each step for a sample research study.
Step 1: Identify the Problem
The first step in the process is to identify a problem or develop a research question. The research problem may be something the agency identifies as a problem, some knowledge or information that is needed by the agency, or the desire to identify a recreation trend nationally. In the example in table 2.4, the problem that the agency has identified is childhood obesity, which is a local problem and concern within the community. This serves as the focus of the study.
Step 2: Review the Literature
Now that the problem has been identified, the researcher must learn more about the topic under investigation. To do this, the researcher must review the literature related to the research problem. This step provides foundational knowledge about the problem area. The review of literature also educates the researcher about what studies have been conducted in the past, how these studies were conducted, and the conclusions in the problem area. In the obesity study, the review of literature enables the programmer to discover horrifying statistics related to the long-term effects of childhood obesity in terms of health issues, death rates, and projected medical costs. In addition, the programmer finds several articles and information from the Centers for Disease Control and Prevention that describe the benefits of walking 10,000 steps a day. The information discovered during this step helps the programmer fully understand the magnitude of the problem, recognize the future consequences of obesity, and identify a strategy to combat obesity (i.e., walking).
Step 3: Clarify the Problem
Many times the initial problem identified in the first step of the process is too large or broad in scope. In step 3 of the process, the researcher clarifies the problem and narrows the scope of the study. This can only be done after the literature has been reviewed. The knowledge gained through the review of literature guides the researcher in clarifying and narrowing the research project. In the example, the programmer has identified childhood obesity as the problem and the purpose of the study. This topic is very broad and could be studied based on genetics, family environment, diet, exercise, self-confidence, leisure activities, or health issues. All of these areas cannot be investigated in a single study; therefore, the problem and purpose of the study must be more clearly defined. The programmer has decided that the purpose of the study is to determine if walking 10,000 steps a day for three days a week will improve the individual's health. This purpose is more narrowly focused and researchable than the original problem.
Step 4: Clearly Define Terms and Concepts
Terms and concepts are words or phrases used in the purpose statement of the study or the description of the study. These items need to be specifically defined as they apply to the study. Terms or concepts often have different definitions depending on who is reading the study. To minimize confusion about what the terms and phrases mean, the researcher must specifically define them for the study. In the obesity study, the concept of “individual's health” can be defined in hundreds of ways, such as physical, mental, emotional, or spiritual health. For this study, the individual's health is defined as physical health. The concept of physical health may also be defined and measured in many ways. In this case, the programmer decides to more narrowly define “individual health” to refer to the areas of weight, percentage of body fat, and cholesterol. By defining the terms or concepts more narrowly, the scope of the study is more manageable for the programmer, making it easier to collect the necessary data for the study. This also makes the concepts more understandable to the reader.
Step 5: Define the Population
Research projects can focus on a specific group of people, facilities, park development, employee evaluations, programs, financial status, marketing efforts, or the integration of technology into the operations. For example, if a researcher wants to examine a specific group of people in the community, the study could examine a specific age group, males or females, people living in a specific geographic area, or a specific ethnic group. Literally thousands of options are available to the researcher to specifically identify the group to study. The research problem and the purpose of the study assist the researcher in identifying the group to involve in the study. In research terms, the group to involve in the study is always called the population. Defining the population assists the researcher in several ways. First, it narrows the scope of the study from a very large population to one that is manageable. Second, the population identifies the group that the researcher's efforts will be focused on within the study. This helps ensure that the researcher stays on the right path during the study. Finally, by defining the population, the researcher identifies the group that the results will apply to at the conclusion of the study. In the example in table 2.4, the programmer has identified the population of the study as children ages 10 to 12 years. This narrower population makes the study more manageable in terms of time and resources.
Step 6: Develop the Instrumentation Plan
The plan for the study is referred to as the instrumentation plan. The instrumentation plan serves as the road map for the entire study, specifying who will participate in the study; how, when, and where data will be collected; and the content of the program. This plan is composed of numerous decisions and considerations that are addressed in chapter 8 of this text. In the obesity study, the researcher has decided to have the children participate in a walking program for six months. The group of participants is called the sample, which is a smaller group selected from the population specified for the study. The study cannot possibly include every 10- to 12-year-old child in the community, so a smaller group is used to represent the population. The researcher develops the plan for the walking program, indicating what data will be collected, when and how the data will be collected, who will collect the data, and how the data will be analyzed. The instrumentation plan specifies all the steps that must be completed for the study. This ensures that the programmer has carefully thought through all these decisions and that she provides a step-by-step plan to be followed in the study.
Step 7: Collect Data
Once the instrumentation plan is completed, the actual study begins with the collection of data. The collection of data is a critical step in providing the information needed to answer the research question. Every study includes the collection of some type of data—whether it is from the literature or from subjects—to answer the research question. Data can be collected in the form of words on a survey, with a questionnaire, through observations, or from the literature. In the obesity study, the programmers will be collecting data on the defined variables: weight, percentage of body fat, cholesterol levels, and the number of days the person walked a total of 10,000 steps during the class.
The researcher collects these data at the first session and at the last session of the program. These two sets of data are necessary to determine the effect of the walking program on weight, body fat, and cholesterol level. Once the data are collected on the variables, the researcher is ready to move to the final step of the process, which is the data analysis.
Step 8: Analyze the Data
All the time, effort, and resources dedicated to steps 1 through 7 of the research process culminate in this final step. The researcher finally has data to analyze so that the research question can be answered. In the instrumentation plan, the researcher specified how the data will be analyzed. The researcher now analyzes the data according to the plan. The results of this analysis are then reviewed and summarized in a manner directly related to the research questions. In the obesity study, the researcher compares the measurements of weight, percentage of body fat, and cholesterol that were taken at the first meeting of the subjects to the measurements of the same variables at the final program session. These two sets of data will be analyzed to determine if there was a difference between the first measurement and the second measurement for each individual in the program. Then, the data will be analyzed to determine if the differences are statistically significant. If the differences are statistically significant, the study validates the theory that was the focus of the study. The results of the study also provide valuable information about one strategy to combat childhood obesity in the community.
As you have probably concluded, conducting studies using the eight steps of the scientific research process requires you to dedicate time and effort to the planning process. You cannot conduct a study using the scientific research process when time is limited or the study is done at the last minute. Researchers who do this conduct studies that result in either false conclusions or conclusions that are not of any value to the organization.
This is an excerpt from Applied Research and Evaluation Methods in Recreation.
Creating instrumentation plans for research studies
The instrumentation plan helps guide the progress of the study to the ultimate goal of gathering data and formulating conclusions to answer the research question.
The instrumentation plan is composed of a number of decisions that need to be made before beginning the study. These decisions are made to determine
what data are needed to answer the research questions,
- how to gather the data,
- when to gather the data,
- where to gather the data, and
- how to analyze the data.
These decisions must be made as part of the instrumentation plan for the study. They help guide the progress of the study to the ultimate goal of gathering data and formulating conclusions to answer the research question.
Selecting Data
The previous decisions help define what data need to be gathered and why the data are important. What is data? Data refers to the information that is gathered to answer the research question. Data can be numbers, words, or actual objects, such as photos, articles, or video. The instrumentation process defines what data need to be collected and the timing of the data collection process.
What data are needed if the researcher is examining the relationship between taking swimming lessons and purchasing a summer session pass? The researcher needs the records of swimming lesson participants and season pass holders. Examining any other type of data, such as instructor evaluations, is a waste of time if it does not focus on answering the research question. If the researcher is examining the factors that lead a person to purchase a summer pass for the pool, interview data are needed to fully understand the factors in the decision-making process of the customer. By using the interview process, the researcher can conduct a two-way conversation in order to explore the factors that lead to purchasing the season pass. This type of data is very comprehensive in exploring the decision-making process, and it is more effective than looking at only one or two variables. In the example of the fitness center evaluating the personal trainers, a written survey will provide the data needed to evaluate the performances of the trainers. By identifying the specific type of data needed to answer the research question, the researchers' efforts are properly focused. Table 8.3 summarizes what data are needed in relation to the previous decisions in the instrumentation plan.
Gathering Data
The next natural question is, how will the data be collected? To answer this question, the researcher needs to identify whether the study is an evaluation, a quantitative study, or a qualitative study. This helps determine how to collect data. What instrument will be used to collect data? The researcher has a variety of options that may be used as a data collection instrument, such as surveys, interviews, observations, or rating instruments. Table 8.4 summarizes the decision of how to gather data along with the previous pieces of the instrumentation plan.
Determining When to Collect Data
Once the researcher decides what data to collect and how to collect them, the researcher must determine when to collect the data, where to collect them, and who should collect them. The results of the study can be influenced by the time that data are gathered, where the data are gathered, and who gathers the data. Studies that only use literature or the records of the agency are not influenced by these decisions. This is the case with the first research question concerning the relationship between taking swimming lessons and purchasing a summer season pass. Once the majority of season passes have been sold, the data should be gathered at the beginning of the summer.
This is not the case when data are being gathered from human subjects. Timing considerations that must be addressed are the month, the day of the week, and the time of day that the data are collected. In the example of identifying what factors lead a person to purchase a season pass to the pool, the interview should be conducted as soon as possible after the purchase of the pass. If the researchers wait weeks or months after the purchase, the subjects may not recall what factors led them to make the purchase. This interview could be done over the phone or in a small group.
For the fitness center evaluating the personal trainers, a specific time frame is necessary in order to ensure a high response rate. The manager decides to give the clients an evaluation form to complete after their last session with their personal trainer (before they leave the facility). This is a better plan than mailing the surveys to the clients' home weeks after the clients' last session with their trainer. The timing of data collection can affect the quality of the data received from the subjects. Table 8.5 presents this information, along with the previous decisions in the instrumentation plan.
Determining Where to Collect Data
The place that data are collected and the person collecting data must be specifically defined in the instrumentation plan. The best plan is to standardize the place and person for the data collection. This standardization helps enhance the truthfulness and validity of the data. The subjects should be in an environment where they feel at ease so that they will answer questions honestly. These two considerations are not an issue for the first research question because the data are coming from the documents of the agency and not from individuals.
In the case of identifying the factors that lead to purchasing a season pass for the pool, the interviews will be conducted by telephone. The people conducting the interviews should be trained in how to conduct an interview and how to record the information accurately. Each person being interviewed will be at home while participating in the interview, which is a comfortable environment for that individual.
In the example of evaluating personal trainers, having the clients' trainer administer the survey in the gym will most likely provide invalid data. The ideal situation would be to have one staff person provide the clients with a quiet room to complete the survey. Then the subject should return the survey to the same staff member who gave the survey to the subject. This standardizes where the data are collected and by whom. Table 8.6 provides a summary of all the instrument pieces and provides examples of each decision that needs to be made in developing the instrumentation plan for a study.
Analyzing Data
The next planning decision to make is how to analyze the data and what to do with the information once the analysis is completed. The type of analysis used with data is determined by whether the data are quantitative or qualitative data. To analyze quantitative data, some type of statistical analysis is used to provide the results. The type of statistical analysis used with data must be thought out and documented in the instrumentation plan.
Some of the most frequently used options for statistical analysis will be covered later in this text. Qualitative data are analyzed through a coding process that identifies themes; these themes become the foundation for the conclusions of the study. This type of data analysis will also be covered later in this text. The results of the data analysis should be summarized and presented in a report to supervisors and other parties for review. Research and evaluation efforts yield a wealth of information that can be used to educate commissioners, city councils, customers, and other decision makers. Studies that document outcomes of the programs and benefits to the community serve as a powerful tool that allows the agency to document its benefits to the community through facts and data, not perceptions and speculations. In today's cost-conscious society, the agency must prove that it is contributing to the quality of life of the community and must document the outcomes from the programs. This type of documentation and evidence is also a requirement for CAPRA accredited agencies and represents an ongoing evaluation process within the agency. Table 8.7 summarizes the key decisions that need to be made in the development of the instrumentation plan.
This is an excerpt from Applied Research and Evaluation Methods in Recreation.
The importance of research and evaluation in recreation
Parks and recreation departments provide a wide variety of programs and services to their local communities. But are these programs accomplishing what they are intended to do for the individual, the organization, or the community?
Foundational Knowledge for Researchers
To become comfortable with research and evaluation, you first need to understand the following:
- The mission and purpose of parks and recreation
- The need for research and evaluation
- How research and evaluation assist an organization
- The expectations for evaluation and research within the industry related to professional certification and agency accreditation
Mission and Purpose of Recreation Agencies
In the United States, the field of recreation and parks has deep roots grounded within societal needs, community needs, and individual needs (see figure 1.1). As you probably remember, the original needs of the community emerged during the Industrial Revolution. With the population shifting from rural areas to urban centers, children needed a safe place to play and socialize. The scope of facilities and services expanded from that point to include providing public parks, public recreation facilities, and instructional classes. The focus of this expansion was to (1) provide places for people to play, (2) offer programs that were educational and considered “wholesome” for the participant, and (3) provide recreational opportunities for individuals, families, and seniors within local communities.
Every park and recreation agency is challenged by—and exists to address—social concerns that become the needs within the community. For example, many American communities are concerned about gang activity, risk-taking behavior of youth, single-parent households, drug and alcohol abuse, and the growing need for sport fields and other public open spaces. Obesity is a primary example of a social concern that park and recreation programs are well positioned to address. This epidemic is prevalent in children, teenagers, and adults within the United States. Researchers predict that many members of the current generation of children will die before their parents because of factors related to obesity (high cholesterol, high blood pressure, diabetes, and a lack of exercise). The National Recreation and Park Association has partnered with other organizations to provide initiatives that address obesity factors, such as Hearts N' Parks, Step Up to Health, and Teens Outside. Each initiative is a community mobilization model that is designed to assist local communities in planning, marketing, and promoting healthy lifestyles related to diet and exercise. In addition, park and recreation agencies provide a wide variety of health and fitness programs, such as swimming lessons, open swim, aerobic classes, sport skill classes, movement classes for preschoolers, weight rooms for working out, and the simple reminder to get outside and play an hour a day. Healthy lifestyles go beyond physical fitness to emotional and psychological wellness. Park and recreation agencies also provide a wide variety of art classes, dance lessons, and concerts. All these kinds of programs and services work toward a healthier community.
Parks and recreation departments provide a wide variety of programs and services to their local communities. But are these programs accomplishing what they are intended to do for the individual, the organization, or the community?
The Need for Research and Evaluation
Every park and recreation agency needs to answer a number of questions about its programs, services, and operations which are noted in figure 1.2. These questions include the following:
- Is the agency meeting the objectives of the programs?
- Is the agency meeting the needs of the customer?
- Is the agency financially stable?
These are only a small sample of the questions that a park and recreation agency needs to answer. How does a park and recreation agency go about answering these questions? The best way is to use the process of research and evaluation. Let's explore these three questions further in order to gain an understanding of the importance of evaluation and research in providing the answers.
Is the Agency Meeting the Objectives of the Programs? All programs provided by park and recreation agencies have a purpose or an objective related to the outcomes for the participants. For example, the objective of a beginning swimming class is to have the participants move through water adjustment skills and learn a basic stroke on their front and back. For the organization, the objective for swimming classes is to have more people learn to swim. This can help reduce the number of rescues or deaths at the organization's aquatic facilities. In addition, the programs should cover costs associated with the programs and generate revenues for the agency. The agency must always consider the cost and revenues generated from the programs. For the community, the objective of swimming classes could be to have a variety of opportunities available for family members to learn to swim and to enjoy the aquatic facilities year-round. Many park and recreation agencies have indoor facilities, outdoor facilities, or both. Swimming is also a physical activity that people can enjoy throughout their lifetime, either in formal classes or individually. Park and recreation agencies must work to gather the information they need to determine if the objectives of the program are being met and whether these are class objectives, agency objectives, or objectives for the community. The research and evaluation process is the primary method for gathering this information and making this assessment. This process gives recreation professionals specific information and evidence to support or negate the notion that their agencies are achieving their objectives. Without research and evaluation, professionals might have only a vague sense of whether objectives are being met, or they may have no idea at all.
Is the Agency Meeting the Needs of the Customer? What are the customer needs and expectations in relation to recreational activities? For swimming classes, the needs of the customers are generally to have a safe environment for the class and to have the class conducted by a qualified instructor. The needs of the participants' parents are to have their children learn to swim in a safe environment so that the children can be safer at pools, lakes, and beaches during the summer. The swimming classes provide one way to meet these needs. Additionally, customers have expectations about their experience or their children's experience in the program. These expectations may be associated with the registration process, how to get to the facility, cleanliness of the locker rooms, cleanliness of the facility, and the quality of the instructor. The agency must gather information to determine if the needs and expectations of the customers are being met during the program.
Is the Agency Financially Stable? Historically, park and recreation departments received a majority of their funding for operations from local taxes. The agency was considered a type of social service that needed tax-based funding for the annual operations. This perception regarding park and recreation agencies has shifted from a social service model to a business-based model. Today, park and recreation agencies are expected to operate as a business. This has resulted in departments receiving very little tax-based funding for operations and has increased the need to generate revenues from programs and services. The agency gathers financial information on a regular basis to determine if the agency is generating sufficient funds to meet the financial needs of the organization. These research activities provide valuable information to assist the managers in making decisions related to the future operations of the agency. The information gained through research and evaluation helps managers decide which programs and services need to be revised, which ones should be added, and which ones should be phased out to ensure that the agency is financially stable. The decision on whether to add or phase out a program cannot be properly made unless the managers have current information about new community needs or needs within the community that no longer exist. The best method for gathering this information is through research and evaluation.
For agencies to answer these fundamental questions concerning their programs, services, and financial operations, research and evaluation have become a necessary activity. Historically, park and recreation agencies did not regularly evaluate program outcomes for the participants. In the past, evaluation efforts centered on the agency's need to know “the numbers”—the number of participants, the hours of the programs, the revenues generated by the programs, and the expenses associated with the programs. This type of evaluation misses the essence and purpose of the park and recreation agency (i.e., meeting the program objectives for the participants and meeting the needs of the customers and community). Evaluating and researching participant outcomes, needs, and expectations are critically important to the future of every park and recreation agency. Agencies that regularly conduct evaluation and research projects gain many benefits that help to sustain the operation of the agency.
This is an excerpt from Applied Research and Evaluation Methods in Recreation.
Steps of the research process
Scientific research involves a systematic process that focuses on being objective and gathering a multitude of information for analysis so that the researcher can come to a conclusion.
Scientific research involves a systematic process that focuses on being objective and gathering a multitude of information for analysis so that the researcher can come to a conclusion. This process is used in all research and evaluation projects, regardless of the research method (scientific method of inquiry, evaluation research, or action research). The process focuses on testing hunches or ideas in a park and recreation setting through a systematic process. In this process, the study is documented in such a way that another individual can conduct the same study again. This is referred to as replicating the study. Any research done without documenting the study so that others can review the process and results is not an investigation using the scientific research process. The scientific research process is a multiple-step process where the steps are interlinked with the other steps in the process. If changes are made in one step of the process, the researcher must review all the other steps to ensure that the changes are reflected throughout the process. Parks and recreation professionals are often involved in conducting research or evaluation projects within the agency. These professionals need to understand the eight steps of the research process as they apply to conducting a study. Table 2.4 lists the steps of the research process and provides an example of each step for a sample research study.
Step 1: Identify the Problem
The first step in the process is to identify a problem or develop a research question. The research problem may be something the agency identifies as a problem, some knowledge or information that is needed by the agency, or the desire to identify a recreation trend nationally. In the example in table 2.4, the problem that the agency has identified is childhood obesity, which is a local problem and concern within the community. This serves as the focus of the study.
Step 2: Review the Literature
Now that the problem has been identified, the researcher must learn more about the topic under investigation. To do this, the researcher must review the literature related to the research problem. This step provides foundational knowledge about the problem area. The review of literature also educates the researcher about what studies have been conducted in the past, how these studies were conducted, and the conclusions in the problem area. In the obesity study, the review of literature enables the programmer to discover horrifying statistics related to the long-term effects of childhood obesity in terms of health issues, death rates, and projected medical costs. In addition, the programmer finds several articles and information from the Centers for Disease Control and Prevention that describe the benefits of walking 10,000 steps a day. The information discovered during this step helps the programmer fully understand the magnitude of the problem, recognize the future consequences of obesity, and identify a strategy to combat obesity (i.e., walking).
Step 3: Clarify the Problem
Many times the initial problem identified in the first step of the process is too large or broad in scope. In step 3 of the process, the researcher clarifies the problem and narrows the scope of the study. This can only be done after the literature has been reviewed. The knowledge gained through the review of literature guides the researcher in clarifying and narrowing the research project. In the example, the programmer has identified childhood obesity as the problem and the purpose of the study. This topic is very broad and could be studied based on genetics, family environment, diet, exercise, self-confidence, leisure activities, or health issues. All of these areas cannot be investigated in a single study; therefore, the problem and purpose of the study must be more clearly defined. The programmer has decided that the purpose of the study is to determine if walking 10,000 steps a day for three days a week will improve the individual's health. This purpose is more narrowly focused and researchable than the original problem.
Step 4: Clearly Define Terms and Concepts
Terms and concepts are words or phrases used in the purpose statement of the study or the description of the study. These items need to be specifically defined as they apply to the study. Terms or concepts often have different definitions depending on who is reading the study. To minimize confusion about what the terms and phrases mean, the researcher must specifically define them for the study. In the obesity study, the concept of “individual's health” can be defined in hundreds of ways, such as physical, mental, emotional, or spiritual health. For this study, the individual's health is defined as physical health. The concept of physical health may also be defined and measured in many ways. In this case, the programmer decides to more narrowly define “individual health” to refer to the areas of weight, percentage of body fat, and cholesterol. By defining the terms or concepts more narrowly, the scope of the study is more manageable for the programmer, making it easier to collect the necessary data for the study. This also makes the concepts more understandable to the reader.
Step 5: Define the Population
Research projects can focus on a specific group of people, facilities, park development, employee evaluations, programs, financial status, marketing efforts, or the integration of technology into the operations. For example, if a researcher wants to examine a specific group of people in the community, the study could examine a specific age group, males or females, people living in a specific geographic area, or a specific ethnic group. Literally thousands of options are available to the researcher to specifically identify the group to study. The research problem and the purpose of the study assist the researcher in identifying the group to involve in the study. In research terms, the group to involve in the study is always called the population. Defining the population assists the researcher in several ways. First, it narrows the scope of the study from a very large population to one that is manageable. Second, the population identifies the group that the researcher's efforts will be focused on within the study. This helps ensure that the researcher stays on the right path during the study. Finally, by defining the population, the researcher identifies the group that the results will apply to at the conclusion of the study. In the example in table 2.4, the programmer has identified the population of the study as children ages 10 to 12 years. This narrower population makes the study more manageable in terms of time and resources.
Step 6: Develop the Instrumentation Plan
The plan for the study is referred to as the instrumentation plan. The instrumentation plan serves as the road map for the entire study, specifying who will participate in the study; how, when, and where data will be collected; and the content of the program. This plan is composed of numerous decisions and considerations that are addressed in chapter 8 of this text. In the obesity study, the researcher has decided to have the children participate in a walking program for six months. The group of participants is called the sample, which is a smaller group selected from the population specified for the study. The study cannot possibly include every 10- to 12-year-old child in the community, so a smaller group is used to represent the population. The researcher develops the plan for the walking program, indicating what data will be collected, when and how the data will be collected, who will collect the data, and how the data will be analyzed. The instrumentation plan specifies all the steps that must be completed for the study. This ensures that the programmer has carefully thought through all these decisions and that she provides a step-by-step plan to be followed in the study.
Step 7: Collect Data
Once the instrumentation plan is completed, the actual study begins with the collection of data. The collection of data is a critical step in providing the information needed to answer the research question. Every study includes the collection of some type of data—whether it is from the literature or from subjects—to answer the research question. Data can be collected in the form of words on a survey, with a questionnaire, through observations, or from the literature. In the obesity study, the programmers will be collecting data on the defined variables: weight, percentage of body fat, cholesterol levels, and the number of days the person walked a total of 10,000 steps during the class.
The researcher collects these data at the first session and at the last session of the program. These two sets of data are necessary to determine the effect of the walking program on weight, body fat, and cholesterol level. Once the data are collected on the variables, the researcher is ready to move to the final step of the process, which is the data analysis.
Step 8: Analyze the Data
All the time, effort, and resources dedicated to steps 1 through 7 of the research process culminate in this final step. The researcher finally has data to analyze so that the research question can be answered. In the instrumentation plan, the researcher specified how the data will be analyzed. The researcher now analyzes the data according to the plan. The results of this analysis are then reviewed and summarized in a manner directly related to the research questions. In the obesity study, the researcher compares the measurements of weight, percentage of body fat, and cholesterol that were taken at the first meeting of the subjects to the measurements of the same variables at the final program session. These two sets of data will be analyzed to determine if there was a difference between the first measurement and the second measurement for each individual in the program. Then, the data will be analyzed to determine if the differences are statistically significant. If the differences are statistically significant, the study validates the theory that was the focus of the study. The results of the study also provide valuable information about one strategy to combat childhood obesity in the community.
As you have probably concluded, conducting studies using the eight steps of the scientific research process requires you to dedicate time and effort to the planning process. You cannot conduct a study using the scientific research process when time is limited or the study is done at the last minute. Researchers who do this conduct studies that result in either false conclusions or conclusions that are not of any value to the organization.
This is an excerpt from Applied Research and Evaluation Methods in Recreation.
Creating instrumentation plans for research studies
The instrumentation plan helps guide the progress of the study to the ultimate goal of gathering data and formulating conclusions to answer the research question.
The instrumentation plan is composed of a number of decisions that need to be made before beginning the study. These decisions are made to determine
what data are needed to answer the research questions,
- how to gather the data,
- when to gather the data,
- where to gather the data, and
- how to analyze the data.
These decisions must be made as part of the instrumentation plan for the study. They help guide the progress of the study to the ultimate goal of gathering data and formulating conclusions to answer the research question.
Selecting Data
The previous decisions help define what data need to be gathered and why the data are important. What is data? Data refers to the information that is gathered to answer the research question. Data can be numbers, words, or actual objects, such as photos, articles, or video. The instrumentation process defines what data need to be collected and the timing of the data collection process.
What data are needed if the researcher is examining the relationship between taking swimming lessons and purchasing a summer session pass? The researcher needs the records of swimming lesson participants and season pass holders. Examining any other type of data, such as instructor evaluations, is a waste of time if it does not focus on answering the research question. If the researcher is examining the factors that lead a person to purchase a summer pass for the pool, interview data are needed to fully understand the factors in the decision-making process of the customer. By using the interview process, the researcher can conduct a two-way conversation in order to explore the factors that lead to purchasing the season pass. This type of data is very comprehensive in exploring the decision-making process, and it is more effective than looking at only one or two variables. In the example of the fitness center evaluating the personal trainers, a written survey will provide the data needed to evaluate the performances of the trainers. By identifying the specific type of data needed to answer the research question, the researchers' efforts are properly focused. Table 8.3 summarizes what data are needed in relation to the previous decisions in the instrumentation plan.
Gathering Data
The next natural question is, how will the data be collected? To answer this question, the researcher needs to identify whether the study is an evaluation, a quantitative study, or a qualitative study. This helps determine how to collect data. What instrument will be used to collect data? The researcher has a variety of options that may be used as a data collection instrument, such as surveys, interviews, observations, or rating instruments. Table 8.4 summarizes the decision of how to gather data along with the previous pieces of the instrumentation plan.
Determining When to Collect Data
Once the researcher decides what data to collect and how to collect them, the researcher must determine when to collect the data, where to collect them, and who should collect them. The results of the study can be influenced by the time that data are gathered, where the data are gathered, and who gathers the data. Studies that only use literature or the records of the agency are not influenced by these decisions. This is the case with the first research question concerning the relationship between taking swimming lessons and purchasing a summer season pass. Once the majority of season passes have been sold, the data should be gathered at the beginning of the summer.
This is not the case when data are being gathered from human subjects. Timing considerations that must be addressed are the month, the day of the week, and the time of day that the data are collected. In the example of identifying what factors lead a person to purchase a season pass to the pool, the interview should be conducted as soon as possible after the purchase of the pass. If the researchers wait weeks or months after the purchase, the subjects may not recall what factors led them to make the purchase. This interview could be done over the phone or in a small group.
For the fitness center evaluating the personal trainers, a specific time frame is necessary in order to ensure a high response rate. The manager decides to give the clients an evaluation form to complete after their last session with their personal trainer (before they leave the facility). This is a better plan than mailing the surveys to the clients' home weeks after the clients' last session with their trainer. The timing of data collection can affect the quality of the data received from the subjects. Table 8.5 presents this information, along with the previous decisions in the instrumentation plan.
Determining Where to Collect Data
The place that data are collected and the person collecting data must be specifically defined in the instrumentation plan. The best plan is to standardize the place and person for the data collection. This standardization helps enhance the truthfulness and validity of the data. The subjects should be in an environment where they feel at ease so that they will answer questions honestly. These two considerations are not an issue for the first research question because the data are coming from the documents of the agency and not from individuals.
In the case of identifying the factors that lead to purchasing a season pass for the pool, the interviews will be conducted by telephone. The people conducting the interviews should be trained in how to conduct an interview and how to record the information accurately. Each person being interviewed will be at home while participating in the interview, which is a comfortable environment for that individual.
In the example of evaluating personal trainers, having the clients' trainer administer the survey in the gym will most likely provide invalid data. The ideal situation would be to have one staff person provide the clients with a quiet room to complete the survey. Then the subject should return the survey to the same staff member who gave the survey to the subject. This standardizes where the data are collected and by whom. Table 8.6 provides a summary of all the instrument pieces and provides examples of each decision that needs to be made in developing the instrumentation plan for a study.
Analyzing Data
The next planning decision to make is how to analyze the data and what to do with the information once the analysis is completed. The type of analysis used with data is determined by whether the data are quantitative or qualitative data. To analyze quantitative data, some type of statistical analysis is used to provide the results. The type of statistical analysis used with data must be thought out and documented in the instrumentation plan.
Some of the most frequently used options for statistical analysis will be covered later in this text. Qualitative data are analyzed through a coding process that identifies themes; these themes become the foundation for the conclusions of the study. This type of data analysis will also be covered later in this text. The results of the data analysis should be summarized and presented in a report to supervisors and other parties for review. Research and evaluation efforts yield a wealth of information that can be used to educate commissioners, city councils, customers, and other decision makers. Studies that document outcomes of the programs and benefits to the community serve as a powerful tool that allows the agency to document its benefits to the community through facts and data, not perceptions and speculations. In today's cost-conscious society, the agency must prove that it is contributing to the quality of life of the community and must document the outcomes from the programs. This type of documentation and evidence is also a requirement for CAPRA accredited agencies and represents an ongoing evaluation process within the agency. Table 8.7 summarizes the key decisions that need to be made in the development of the instrumentation plan.
This is an excerpt from Applied Research and Evaluation Methods in Recreation.
The importance of research and evaluation in recreation
Parks and recreation departments provide a wide variety of programs and services to their local communities. But are these programs accomplishing what they are intended to do for the individual, the organization, or the community?
Foundational Knowledge for Researchers
To become comfortable with research and evaluation, you first need to understand the following:
- The mission and purpose of parks and recreation
- The need for research and evaluation
- How research and evaluation assist an organization
- The expectations for evaluation and research within the industry related to professional certification and agency accreditation
Mission and Purpose of Recreation Agencies
In the United States, the field of recreation and parks has deep roots grounded within societal needs, community needs, and individual needs (see figure 1.1). As you probably remember, the original needs of the community emerged during the Industrial Revolution. With the population shifting from rural areas to urban centers, children needed a safe place to play and socialize. The scope of facilities and services expanded from that point to include providing public parks, public recreation facilities, and instructional classes. The focus of this expansion was to (1) provide places for people to play, (2) offer programs that were educational and considered “wholesome” for the participant, and (3) provide recreational opportunities for individuals, families, and seniors within local communities.
Every park and recreation agency is challenged by—and exists to address—social concerns that become the needs within the community. For example, many American communities are concerned about gang activity, risk-taking behavior of youth, single-parent households, drug and alcohol abuse, and the growing need for sport fields and other public open spaces. Obesity is a primary example of a social concern that park and recreation programs are well positioned to address. This epidemic is prevalent in children, teenagers, and adults within the United States. Researchers predict that many members of the current generation of children will die before their parents because of factors related to obesity (high cholesterol, high blood pressure, diabetes, and a lack of exercise). The National Recreation and Park Association has partnered with other organizations to provide initiatives that address obesity factors, such as Hearts N' Parks, Step Up to Health, and Teens Outside. Each initiative is a community mobilization model that is designed to assist local communities in planning, marketing, and promoting healthy lifestyles related to diet and exercise. In addition, park and recreation agencies provide a wide variety of health and fitness programs, such as swimming lessons, open swim, aerobic classes, sport skill classes, movement classes for preschoolers, weight rooms for working out, and the simple reminder to get outside and play an hour a day. Healthy lifestyles go beyond physical fitness to emotional and psychological wellness. Park and recreation agencies also provide a wide variety of art classes, dance lessons, and concerts. All these kinds of programs and services work toward a healthier community.
Parks and recreation departments provide a wide variety of programs and services to their local communities. But are these programs accomplishing what they are intended to do for the individual, the organization, or the community?
The Need for Research and Evaluation
Every park and recreation agency needs to answer a number of questions about its programs, services, and operations which are noted in figure 1.2. These questions include the following:
- Is the agency meeting the objectives of the programs?
- Is the agency meeting the needs of the customer?
- Is the agency financially stable?
These are only a small sample of the questions that a park and recreation agency needs to answer. How does a park and recreation agency go about answering these questions? The best way is to use the process of research and evaluation. Let's explore these three questions further in order to gain an understanding of the importance of evaluation and research in providing the answers.
Is the Agency Meeting the Objectives of the Programs? All programs provided by park and recreation agencies have a purpose or an objective related to the outcomes for the participants. For example, the objective of a beginning swimming class is to have the participants move through water adjustment skills and learn a basic stroke on their front and back. For the organization, the objective for swimming classes is to have more people learn to swim. This can help reduce the number of rescues or deaths at the organization's aquatic facilities. In addition, the programs should cover costs associated with the programs and generate revenues for the agency. The agency must always consider the cost and revenues generated from the programs. For the community, the objective of swimming classes could be to have a variety of opportunities available for family members to learn to swim and to enjoy the aquatic facilities year-round. Many park and recreation agencies have indoor facilities, outdoor facilities, or both. Swimming is also a physical activity that people can enjoy throughout their lifetime, either in formal classes or individually. Park and recreation agencies must work to gather the information they need to determine if the objectives of the program are being met and whether these are class objectives, agency objectives, or objectives for the community. The research and evaluation process is the primary method for gathering this information and making this assessment. This process gives recreation professionals specific information and evidence to support or negate the notion that their agencies are achieving their objectives. Without research and evaluation, professionals might have only a vague sense of whether objectives are being met, or they may have no idea at all.
Is the Agency Meeting the Needs of the Customer? What are the customer needs and expectations in relation to recreational activities? For swimming classes, the needs of the customers are generally to have a safe environment for the class and to have the class conducted by a qualified instructor. The needs of the participants' parents are to have their children learn to swim in a safe environment so that the children can be safer at pools, lakes, and beaches during the summer. The swimming classes provide one way to meet these needs. Additionally, customers have expectations about their experience or their children's experience in the program. These expectations may be associated with the registration process, how to get to the facility, cleanliness of the locker rooms, cleanliness of the facility, and the quality of the instructor. The agency must gather information to determine if the needs and expectations of the customers are being met during the program.
Is the Agency Financially Stable? Historically, park and recreation departments received a majority of their funding for operations from local taxes. The agency was considered a type of social service that needed tax-based funding for the annual operations. This perception regarding park and recreation agencies has shifted from a social service model to a business-based model. Today, park and recreation agencies are expected to operate as a business. This has resulted in departments receiving very little tax-based funding for operations and has increased the need to generate revenues from programs and services. The agency gathers financial information on a regular basis to determine if the agency is generating sufficient funds to meet the financial needs of the organization. These research activities provide valuable information to assist the managers in making decisions related to the future operations of the agency. The information gained through research and evaluation helps managers decide which programs and services need to be revised, which ones should be added, and which ones should be phased out to ensure that the agency is financially stable. The decision on whether to add or phase out a program cannot be properly made unless the managers have current information about new community needs or needs within the community that no longer exist. The best method for gathering this information is through research and evaluation.
For agencies to answer these fundamental questions concerning their programs, services, and financial operations, research and evaluation have become a necessary activity. Historically, park and recreation agencies did not regularly evaluate program outcomes for the participants. In the past, evaluation efforts centered on the agency's need to know “the numbers”—the number of participants, the hours of the programs, the revenues generated by the programs, and the expenses associated with the programs. This type of evaluation misses the essence and purpose of the park and recreation agency (i.e., meeting the program objectives for the participants and meeting the needs of the customers and community). Evaluating and researching participant outcomes, needs, and expectations are critically important to the future of every park and recreation agency. Agencies that regularly conduct evaluation and research projects gain many benefits that help to sustain the operation of the agency.
This is an excerpt from Applied Research and Evaluation Methods in Recreation.
Steps of the research process
Scientific research involves a systematic process that focuses on being objective and gathering a multitude of information for analysis so that the researcher can come to a conclusion.
Scientific research involves a systematic process that focuses on being objective and gathering a multitude of information for analysis so that the researcher can come to a conclusion. This process is used in all research and evaluation projects, regardless of the research method (scientific method of inquiry, evaluation research, or action research). The process focuses on testing hunches or ideas in a park and recreation setting through a systematic process. In this process, the study is documented in such a way that another individual can conduct the same study again. This is referred to as replicating the study. Any research done without documenting the study so that others can review the process and results is not an investigation using the scientific research process. The scientific research process is a multiple-step process where the steps are interlinked with the other steps in the process. If changes are made in one step of the process, the researcher must review all the other steps to ensure that the changes are reflected throughout the process. Parks and recreation professionals are often involved in conducting research or evaluation projects within the agency. These professionals need to understand the eight steps of the research process as they apply to conducting a study. Table 2.4 lists the steps of the research process and provides an example of each step for a sample research study.
Step 1: Identify the Problem
The first step in the process is to identify a problem or develop a research question. The research problem may be something the agency identifies as a problem, some knowledge or information that is needed by the agency, or the desire to identify a recreation trend nationally. In the example in table 2.4, the problem that the agency has identified is childhood obesity, which is a local problem and concern within the community. This serves as the focus of the study.
Step 2: Review the Literature
Now that the problem has been identified, the researcher must learn more about the topic under investigation. To do this, the researcher must review the literature related to the research problem. This step provides foundational knowledge about the problem area. The review of literature also educates the researcher about what studies have been conducted in the past, how these studies were conducted, and the conclusions in the problem area. In the obesity study, the review of literature enables the programmer to discover horrifying statistics related to the long-term effects of childhood obesity in terms of health issues, death rates, and projected medical costs. In addition, the programmer finds several articles and information from the Centers for Disease Control and Prevention that describe the benefits of walking 10,000 steps a day. The information discovered during this step helps the programmer fully understand the magnitude of the problem, recognize the future consequences of obesity, and identify a strategy to combat obesity (i.e., walking).
Step 3: Clarify the Problem
Many times the initial problem identified in the first step of the process is too large or broad in scope. In step 3 of the process, the researcher clarifies the problem and narrows the scope of the study. This can only be done after the literature has been reviewed. The knowledge gained through the review of literature guides the researcher in clarifying and narrowing the research project. In the example, the programmer has identified childhood obesity as the problem and the purpose of the study. This topic is very broad and could be studied based on genetics, family environment, diet, exercise, self-confidence, leisure activities, or health issues. All of these areas cannot be investigated in a single study; therefore, the problem and purpose of the study must be more clearly defined. The programmer has decided that the purpose of the study is to determine if walking 10,000 steps a day for three days a week will improve the individual's health. This purpose is more narrowly focused and researchable than the original problem.
Step 4: Clearly Define Terms and Concepts
Terms and concepts are words or phrases used in the purpose statement of the study or the description of the study. These items need to be specifically defined as they apply to the study. Terms or concepts often have different definitions depending on who is reading the study. To minimize confusion about what the terms and phrases mean, the researcher must specifically define them for the study. In the obesity study, the concept of “individual's health” can be defined in hundreds of ways, such as physical, mental, emotional, or spiritual health. For this study, the individual's health is defined as physical health. The concept of physical health may also be defined and measured in many ways. In this case, the programmer decides to more narrowly define “individual health” to refer to the areas of weight, percentage of body fat, and cholesterol. By defining the terms or concepts more narrowly, the scope of the study is more manageable for the programmer, making it easier to collect the necessary data for the study. This also makes the concepts more understandable to the reader.
Step 5: Define the Population
Research projects can focus on a specific group of people, facilities, park development, employee evaluations, programs, financial status, marketing efforts, or the integration of technology into the operations. For example, if a researcher wants to examine a specific group of people in the community, the study could examine a specific age group, males or females, people living in a specific geographic area, or a specific ethnic group. Literally thousands of options are available to the researcher to specifically identify the group to study. The research problem and the purpose of the study assist the researcher in identifying the group to involve in the study. In research terms, the group to involve in the study is always called the population. Defining the population assists the researcher in several ways. First, it narrows the scope of the study from a very large population to one that is manageable. Second, the population identifies the group that the researcher's efforts will be focused on within the study. This helps ensure that the researcher stays on the right path during the study. Finally, by defining the population, the researcher identifies the group that the results will apply to at the conclusion of the study. In the example in table 2.4, the programmer has identified the population of the study as children ages 10 to 12 years. This narrower population makes the study more manageable in terms of time and resources.
Step 6: Develop the Instrumentation Plan
The plan for the study is referred to as the instrumentation plan. The instrumentation plan serves as the road map for the entire study, specifying who will participate in the study; how, when, and where data will be collected; and the content of the program. This plan is composed of numerous decisions and considerations that are addressed in chapter 8 of this text. In the obesity study, the researcher has decided to have the children participate in a walking program for six months. The group of participants is called the sample, which is a smaller group selected from the population specified for the study. The study cannot possibly include every 10- to 12-year-old child in the community, so a smaller group is used to represent the population. The researcher develops the plan for the walking program, indicating what data will be collected, when and how the data will be collected, who will collect the data, and how the data will be analyzed. The instrumentation plan specifies all the steps that must be completed for the study. This ensures that the programmer has carefully thought through all these decisions and that she provides a step-by-step plan to be followed in the study.
Step 7: Collect Data
Once the instrumentation plan is completed, the actual study begins with the collection of data. The collection of data is a critical step in providing the information needed to answer the research question. Every study includes the collection of some type of data—whether it is from the literature or from subjects—to answer the research question. Data can be collected in the form of words on a survey, with a questionnaire, through observations, or from the literature. In the obesity study, the programmers will be collecting data on the defined variables: weight, percentage of body fat, cholesterol levels, and the number of days the person walked a total of 10,000 steps during the class.
The researcher collects these data at the first session and at the last session of the program. These two sets of data are necessary to determine the effect of the walking program on weight, body fat, and cholesterol level. Once the data are collected on the variables, the researcher is ready to move to the final step of the process, which is the data analysis.
Step 8: Analyze the Data
All the time, effort, and resources dedicated to steps 1 through 7 of the research process culminate in this final step. The researcher finally has data to analyze so that the research question can be answered. In the instrumentation plan, the researcher specified how the data will be analyzed. The researcher now analyzes the data according to the plan. The results of this analysis are then reviewed and summarized in a manner directly related to the research questions. In the obesity study, the researcher compares the measurements of weight, percentage of body fat, and cholesterol that were taken at the first meeting of the subjects to the measurements of the same variables at the final program session. These two sets of data will be analyzed to determine if there was a difference between the first measurement and the second measurement for each individual in the program. Then, the data will be analyzed to determine if the differences are statistically significant. If the differences are statistically significant, the study validates the theory that was the focus of the study. The results of the study also provide valuable information about one strategy to combat childhood obesity in the community.
As you have probably concluded, conducting studies using the eight steps of the scientific research process requires you to dedicate time and effort to the planning process. You cannot conduct a study using the scientific research process when time is limited or the study is done at the last minute. Researchers who do this conduct studies that result in either false conclusions or conclusions that are not of any value to the organization.
This is an excerpt from Applied Research and Evaluation Methods in Recreation.
Creating instrumentation plans for research studies
The instrumentation plan helps guide the progress of the study to the ultimate goal of gathering data and formulating conclusions to answer the research question.
The instrumentation plan is composed of a number of decisions that need to be made before beginning the study. These decisions are made to determine
what data are needed to answer the research questions,
- how to gather the data,
- when to gather the data,
- where to gather the data, and
- how to analyze the data.
These decisions must be made as part of the instrumentation plan for the study. They help guide the progress of the study to the ultimate goal of gathering data and formulating conclusions to answer the research question.
Selecting Data
The previous decisions help define what data need to be gathered and why the data are important. What is data? Data refers to the information that is gathered to answer the research question. Data can be numbers, words, or actual objects, such as photos, articles, or video. The instrumentation process defines what data need to be collected and the timing of the data collection process.
What data are needed if the researcher is examining the relationship between taking swimming lessons and purchasing a summer session pass? The researcher needs the records of swimming lesson participants and season pass holders. Examining any other type of data, such as instructor evaluations, is a waste of time if it does not focus on answering the research question. If the researcher is examining the factors that lead a person to purchase a summer pass for the pool, interview data are needed to fully understand the factors in the decision-making process of the customer. By using the interview process, the researcher can conduct a two-way conversation in order to explore the factors that lead to purchasing the season pass. This type of data is very comprehensive in exploring the decision-making process, and it is more effective than looking at only one or two variables. In the example of the fitness center evaluating the personal trainers, a written survey will provide the data needed to evaluate the performances of the trainers. By identifying the specific type of data needed to answer the research question, the researchers' efforts are properly focused. Table 8.3 summarizes what data are needed in relation to the previous decisions in the instrumentation plan.
Gathering Data
The next natural question is, how will the data be collected? To answer this question, the researcher needs to identify whether the study is an evaluation, a quantitative study, or a qualitative study. This helps determine how to collect data. What instrument will be used to collect data? The researcher has a variety of options that may be used as a data collection instrument, such as surveys, interviews, observations, or rating instruments. Table 8.4 summarizes the decision of how to gather data along with the previous pieces of the instrumentation plan.
Determining When to Collect Data
Once the researcher decides what data to collect and how to collect them, the researcher must determine when to collect the data, where to collect them, and who should collect them. The results of the study can be influenced by the time that data are gathered, where the data are gathered, and who gathers the data. Studies that only use literature or the records of the agency are not influenced by these decisions. This is the case with the first research question concerning the relationship between taking swimming lessons and purchasing a summer season pass. Once the majority of season passes have been sold, the data should be gathered at the beginning of the summer.
This is not the case when data are being gathered from human subjects. Timing considerations that must be addressed are the month, the day of the week, and the time of day that the data are collected. In the example of identifying what factors lead a person to purchase a season pass to the pool, the interview should be conducted as soon as possible after the purchase of the pass. If the researchers wait weeks or months after the purchase, the subjects may not recall what factors led them to make the purchase. This interview could be done over the phone or in a small group.
For the fitness center evaluating the personal trainers, a specific time frame is necessary in order to ensure a high response rate. The manager decides to give the clients an evaluation form to complete after their last session with their personal trainer (before they leave the facility). This is a better plan than mailing the surveys to the clients' home weeks after the clients' last session with their trainer. The timing of data collection can affect the quality of the data received from the subjects. Table 8.5 presents this information, along with the previous decisions in the instrumentation plan.
Determining Where to Collect Data
The place that data are collected and the person collecting data must be specifically defined in the instrumentation plan. The best plan is to standardize the place and person for the data collection. This standardization helps enhance the truthfulness and validity of the data. The subjects should be in an environment where they feel at ease so that they will answer questions honestly. These two considerations are not an issue for the first research question because the data are coming from the documents of the agency and not from individuals.
In the case of identifying the factors that lead to purchasing a season pass for the pool, the interviews will be conducted by telephone. The people conducting the interviews should be trained in how to conduct an interview and how to record the information accurately. Each person being interviewed will be at home while participating in the interview, which is a comfortable environment for that individual.
In the example of evaluating personal trainers, having the clients' trainer administer the survey in the gym will most likely provide invalid data. The ideal situation would be to have one staff person provide the clients with a quiet room to complete the survey. Then the subject should return the survey to the same staff member who gave the survey to the subject. This standardizes where the data are collected and by whom. Table 8.6 provides a summary of all the instrument pieces and provides examples of each decision that needs to be made in developing the instrumentation plan for a study.
Analyzing Data
The next planning decision to make is how to analyze the data and what to do with the information once the analysis is completed. The type of analysis used with data is determined by whether the data are quantitative or qualitative data. To analyze quantitative data, some type of statistical analysis is used to provide the results. The type of statistical analysis used with data must be thought out and documented in the instrumentation plan.
Some of the most frequently used options for statistical analysis will be covered later in this text. Qualitative data are analyzed through a coding process that identifies themes; these themes become the foundation for the conclusions of the study. This type of data analysis will also be covered later in this text. The results of the data analysis should be summarized and presented in a report to supervisors and other parties for review. Research and evaluation efforts yield a wealth of information that can be used to educate commissioners, city councils, customers, and other decision makers. Studies that document outcomes of the programs and benefits to the community serve as a powerful tool that allows the agency to document its benefits to the community through facts and data, not perceptions and speculations. In today's cost-conscious society, the agency must prove that it is contributing to the quality of life of the community and must document the outcomes from the programs. This type of documentation and evidence is also a requirement for CAPRA accredited agencies and represents an ongoing evaluation process within the agency. Table 8.7 summarizes the key decisions that need to be made in the development of the instrumentation plan.
This is an excerpt from Applied Research and Evaluation Methods in Recreation.