The Oregon Community Foundation: Grant Programs

Environmental Ed photo Why is Evaluation Important?

A considerable amount of time, effort, and other resources go into the development and implementation of environmental education programs and curricula. Quite obviously, our goal is to create effective programs that can serve as models of excellence for Oregon. Whether the program involves incorporating nature activities into early childhood education, taking sixth graders to a resident outdoor education center, or working with high school students on a community service learning project, the aim of providing quality educational experiences remains the same. But, how do we know that the programs we are providing are effective?

Building evaluation into program design and implementation can help us answer some basic questions. Often, in the course of implementing a program, various types of information are gathered. We want to know how many individuals participated in an event, whether workshops were delivered on schedule, or whether participants were generally satisfied with the logistics. Answers to these questions provide useful information. They help us monitor specific aspects of the program.

Lupine Evaluation can also help us answer deeper questions about how our program is working and can help us determine the program’s merit (does it work?). Evaluations can determine whether participants developed a specific set of ecological concepts, whether staff members and volunteers feel confident in their ability to deliver a particular educational experience, or how community members are using natural areas. Evaluation helps us determine if a project should be continued in its current configuration and, if so, suggest ways to improve it. Evaluation documents program accomplishments. If the project has been designed with well-articulated outcomes that specify what we want to accomplish, the evaluation can help us understand whether or not these outcomes are really being met.

We conduct evaluations because we want to understand, in a systematic way, what is and is not working. All too often, how a program works (or doesn’t work) is understood primarily through a combination of instincts, anecdotes and numbers on a balance sheet. The passionate instructor who is wrapped up in the moment of teaching is probably not in the position to make defensible claims about the long-term impact of a lesson on learners’ education. Likewise, a series of anecdotes, although informative and even satisfying, cannot be generalized. Evaluation provides perspective. It provides evidence. It provides the types of information necessary for sound decision-making.

Planning Your Evaluation

Just as each program is different, each evaluation is different. Here are some things to keep in mind as you plan your evaluation:

Typically, approximately 7-10% of your total project budget should be earmarked for your evaluation. This would include the costs of designing the evaluation methods and data collection strategies (e.g., pretest/posttest, interviews, survey instruments), collecting data (e.g., postage and copying costs of mailing a survey to community members), data analysis, and report writing.
Designing and conducting an evaluation requires a degree of expertise. Early in your process, the decision will need to be made whether you will use an internal evaluator or an external evaluator. That is, will someone within your organization design and implement the evaluation or will you need to hire an outside evaluator? As you develop your evaluation plan, think about the type of data you want to collect and the questions you want to answer – consider whether you have the expertise “in house” to implement this plan.
Evaluation results should be used as a tool for program improvement. Select objectives (see below), evaluation questions and methods that will let you know what you have accomplished and provide insights for program improvement.
Evaluations typically include both quantitative and qualitative data. Quantitative data allow us to count events (e.g., workshops held, number of field trips) or number of participants, determine cost/participants, perform statistical analyses (mean, median, mode, standard deviation), and complete other calculations. Qualitative methods allow us to record explanations, perceptions, and descriptions of experiences – often in the participants’ own words. Qualitative methods allow us to create narratives that provide an in-depth view and a more complete understanding of the context of the evaluation. Typically, a group of individuals participate in a qualitative evaluation. Using both quantitative and qualitative methods can strengthen your evaluation. A survey of participants may provide a great deal of information about what activities they liked most (and least); an interview of a group of these participants may provide in-depth information concerning why those activities are most desired (or least desired) and, importantly, what characteristics make a particular type of activity most desired.
Whether quantitative and/or qualitative data are collected, it is essential that data are collected systematically. Create a plan for what data you want to collect (e.g., participants’ ability to perform a particular task), who you will collect the data from (e.g., teachers and volunteer mentors), how you will collect it (e.g., teaching observation checklist), and when you will collect it (e.g., during lessons being delivered at the resident outdoor school). And, importantly, stick to that plan.

Guidelines for Completing Your Evaluation Plan

Evaluation Plan. The evaluation plan describes what you want to evaluate (objectives), who you will collect information from (participants), how you will collect that data (surveys, interviews, reports), and what your targeted outcomes are (projected accomplishments). Your evaluation plan should be submitted with your grant proposal. To simplify this process, complete the attached evaluation plan worksheet. We have provided a template as well as examples.

Evaluation Report. Please keep in mind that you will be required to report the results of your evaluation at the end of the project. Avoid creating an evaluation plan that is too ambitious (not doable given your resources) or one that will not answer important questions about how your program is working. Copies of data collection tools (e.g., surveys, interview questions, focus group questions, observation checklists) should be included in your evaluation report.

Directions for Completing the Evaluation Plan Template:

The evaluation plan template is formatted as a matrix and asks you to supply the following information:
_ Goal(s) addressed by your program _ Program objectives that will be evaluated _ Program activities _ Target audience(s) _ Data collection tool(s) _ Evaluation design _ Expected outcome(s)

Goal: Identify at least one of the four Funding Goals for the Environmental Education Program listed in the Environmental Education Program RFP that your program is addressing:

Strengthening and developing programs that provide outdoor experiences for youth from early childhood through grade 12.
Creating, expanding and improving programs that connect schools with their communities and provide students with practical hands-on experience in addressing environmental issues both locally and globally.
Creating lasting change in education systems to foster improved understanding of and interaction with our natural systems.
Encouraging programs that connect creativity with the natural environment.

Project Activities: List what you will be doing in concrete terms. That is, are you sponsoring workshops, teaching programs at an outdoor resident center, taking participants on field trips, promoting participation in service learning projects, or installing energy efficient windows? Be as specific as possible. For example, sponsor “five, one-day teacher training workshops on how to use an inquiry approach in the outdoors”; lead “10, two-hour field trips to natural areas”; “install 25 energy efficient windows in the main building and 10 energy efficient windows in each of the three dormitories for a total of 55 windows.”

Target Audience(s): List all of the audiences you will be serving directly through your program or curriculum. Who is your target audience? That is, who benefits directly from your project? Are you teaching young children attending a day care center or high school students participating in a stream monitoring program? Are you reaching teachers or community members? Be sure to indicate how many project participants you expect to serve. For example, 75 3-5 year old children attending local day care centers; 150 middle school language arts and social studies teachers from District XX; 250 low-income fifth and sixth grade students from XX counties.

Objectives: Your evaluation should measure the degree to which your program or curriculum meets specific objectives. Measuring multiple objectives provides a deeper understanding of the degree to which you have accomplished what you set out to do. The following list includes objectives (Part I) related to the “reach” of your program (i.e., the number of participants served, the number of infrastructure changes made). The objectives listed in Part II, relate to the degree to which “change” occurred or the effectiveness of your program (e.g., increased understanding, increased ability to use a particular teaching strategy, decreased natural resource use).

In your evaluation plan, you will need to select at least two objectives that will be evaluated. At least one of these objectives should be from Part I (Objectives 1-6) and at least one of these objectives should be from Part II (Objective 7-17). You are encouraged to include any additional objectives that are specific to your program.

Part I (Choose at least one objective from #1-6)
As a result of the program or curriculum, the number of . . .

…individuals participating in nature-oriented experiences will increase.
…teachers and volunteer mentors who lead projects promoting sustainability and
stewardship in their schools and communities will increase.
…individuals participating, on their own, in nature-oriented experiences will increase.
… youth involved in stewardship projects will increase.
…sustainable systems in schools and education facilities (i.e., solar power) will increase.
…natural play places or outdoor, experiential learning spaces on or near school grounds or education facilities will increase.

Part II (Choose at least one objective from #7-17)
As a result of the program or curriculum, participants will increase their…

…understanding of natural systems.
… understanding of the geography of Oregon.
… appreciation of the natural world especially in the Pacific Northwest.
…understanding of community infrastructure (e.g., landfills, wastewater treatment plants) designed to protect our communities and ecosystems.
… understanding of ways to better protect the natural world.
… ability to strengthen civic responsibility in Oregon communities (e.g., around issues of environmental sustainability and restoration).
… understanding of the use of school gardens, community gardens, and local farms and food systems.

As a result of the program, teachers and volunteer mentors will increase their…

… ability to align and integrate effective experiential, outdoor focused, place-based, and inquiry-based learning into the curriculum.
… ability to lead projects (e.g., service learning, stewardship) promoting sustainability in their schools and communities.
… understanding of the field of sustainability.

As a result of your program or infrastructure changes…

…the amount of energy, water, or other natural resource use will be reduced.

Data collection tool(s): Here you will list the methods you will use to gather the information for your evaluation. How will you collect the information necessary to measure success? For example, you might use a survey, conduct interviews, administer a test, or collate end of month attendance records. The type of data collection tools or measurement tools you use will depend on the type of objectives you are addressing.

For Objectives 1-6 listed in Part I (above), data may come from attendance records (e.g., number of students attending a field trip, number of teachers enrolled in a workshop), invoices (e.g., number of windows purchased and installed), and self-report data (e.g., number of parents or caregivers who report on a survey that they have sought out nature-oriented activities).

For Objectives 7-17 listed in Part II (above), data should measure change that has occurred because of your program or provide evidence of the effectiveness of your program or curriculum. For example, tests of knowledge can measure participants’ level of understanding. Surveys can measure the degree to which participants report that they appreciate natural environments. Interviews can measure whether or not teachers feel comfortable using a particular instructional method and observation checklists can measure whether or not teachers are using that instruction method correctly. In addition, energy or water bills might be used to collect information on changes in resource usage rates (Objective 17).

Evaluation design: When and from whom will you collect your evaluation information? That is, will you survey participants at the end of a workshop? Will you give students a pretest and a posttest? Will you collect data from a control group that does not participate in the program? Will you collect attendance records from participating schools at the end of each quarter?

Expected outcome(s): What specifically do you hope to accomplish as a result of this program or curriculum? That is, you will want to identify the specific level of increase you expect. Spell out the test results you expect. Indicate how much energy, water or other resources you expect to save. Outcomes are often described in terms of percentages (e.g., 95% of teachers will take their students on a field trip, 80% of the students will report appreciation of their local natural environment). At times, a score along with a percentage are used to indicate the desired outcome (e.g., 75% of students will pass the knowledge with a score of 80 or better). Under certain circumstances when pretest and posttest scores are compared, statistical tests of significance can be reported (e.g., students will increase their knowledge of natural systems significantly).

Evaluation Plan Template

The evaluation plan template is formatted as a matrix.

List the requested information in the appropriate cell of the matrix. If the requested information is not applicable given your particular program, simply write N/A.
The matrix or evaluation plan should be read from left to right, starting with an objective and ending with an outcome related to that objective.
Start a new row for each objective. Your matrix will include at least two different objectives, at least one from Objectives 1-6 and at least one from Objectives 7-17.
You may include additional objectives that are specific to your program.

Gray Family Fund of the Oregon Community Foundation
Environmental Education Program
Evaluation Plan Template – Example

Goal(s) addressed by your proposed program: (See directions)

• Strengthening and developing programs that provide outdoor experiences for youth from early childhood through grade 12.