Tools

Note: Some of these tools were originally presented in the Evaluation Webinar series on Demonstrating Client Improvement to Yourself and Others: Setting up an Evaluation System to Succeed; Demonstrating Client Improvement to Yourself and Others: Understanding and Using your Outcomes; and Demonstrating Client Improvement to Yourself and Others: Making Sure Client Numbers Reflect Client Reality.

• Excel Tracking Tool, to ensure systematic recording of data.  Reviewed in the webinar “Demonstrating Client Improvement to Yourself and Others: Setting up an Evaluation System to Succeed”, can be modified for your own use.
• Paired T-test Calculator Choose “2-tailed test” for most data, since change can occur in either direction (increase or decrease in symptoms, for example).
• Effect Size Calculator – (from the University of Colorado) to calculate Cohen’s d value.  To use this online application, plug in values for your mean and standard deviation, and click on “compute” to calculate a Cohen’s d value, showing the size of your intervention’s effect.
• Chi-Square Calculation Tool for Dichotomous, categorical Data
• Correlation Calculation– a sample excel file, showing how to calculate a Pearson correlation coefficient.
• Instrument Reliability and Validity (from the University of Connecticut).  Includes link to an Excel spreadsheet, which can be used to calculate a Cronbach’s alpha value, to determine inter-rater reliability.

Resource Centers

• American Evaluation Association (AEA) http://www.eval.org    Extensive resources for planning and conducting Program Evaluation.
• The Center for Outcome Analysis:  a non-profit which provides resources to human services and health care organizations, specifically to provide support for outcomes analysis.  Provides sample surveys and scales.

Listservs

• If you join the American Evaluation Association, you can post questions to their forum and get feedback from other members. (There is an annual fee for membership.)
• The American Evaluation Association also provides links to several Discussion Lists and Listservs that do not require membership.

Program Evaluation

Psychometrics

Data Analysis

Reporting Results

Ethics

Logic Models

Apps

Evidence Based Practices

• Comprehensive list of tools and resources for Program Evaluation complied by the American Evaluation Association.
• W.K. Kellogg Foundation Evaluation Handbook (1998, 120 pages).  This handbook (free download) provides detailed steps you can use to plan and conduct an Evaluation of your organization. The W.K. Kellogg Foundation also includes extensive resources for grant-writing.
  • Essentials of a Good Psychological Test – an overview of various types of reliability and validity.
  • How to choose tools, instruments, & questionnaires for intervention research & evaluation (James Neill, 2004).  A list of factors to consider when choosing a measurement tool or scoring system.
  • Repository of Measurement Instruments and Associated Reviews (site accessible with AEA membership)
  • Factors influencing internal and external validity (Campbell & Stanley, 1963). If your sample does not show internal validity, review this list of factors for possible sources of error. 
  • Three basic considerations for data collection during a Program Evaluation; summary by the Pell Institute.
  • Cultural factors affecting evaluation; from the American Evaluation Association (AEA); cultural factors are reflected in evaluation results; identifies methods to accommodate cultural difference during data collection. 
  • How to Use Excel for Data Entry by the University of Tennessee's Statistical Consulting Center.  This free 4 page (pdf) guideline shows how to create a dataset in Excel and how to import Excel data it into other programs, such as SPSS, SAS, and “R”.
  • Free online SPSS tutorial.   Extensive tutorial shows how to setup datasets, perform analysis, and create graphs.
  • Reporting Statistics in APA Style guideline for reporting results in APA format (based on the 6th edition of the APA Publication manual); includes many sentences as examples.
  • The American Evaluation Association provides resources and guidance on the process of Evaluation for any organizational setting.  They provide a detailed list of principles and ethical standards which apply to any Evaluation process.   These guidelines include ways to inform colleagues and clients of the standards and principles that will be upheld during evaluation.
  • Basic ethical practices in collecting data.   This guide from the University of Texas, Instructional Assessment Resources, provides basic principles of confidentiality when working with data.   Includes a sample consent form, for participants who fill our surveys of provide information during an evaluation.
  • An online course (estimated time 40 minutes) on Logic Models from the Center for Disease Control.  Covers basic methods which can be applied to strategic planning or program evaluation.
  • Extensive resources on Logic Models from the University of Wisconsin, Extension-Program Development & Evaluation; includes a PowerPoint slide overview on logic models, and examples and templates for creating your own models.
  • There are many free apps from the Apple Appstore (for ipads, iphone, ipod touch).  These free interactive apps (such as “Simple Stat” and “LearnStatFree”) can help you visualize statistical concepts and enter sample data. These are useful to learn basic concepts and experiment with values; however, they are not suitable for data analysis.
  • The Guide to Community Preventive Services: Systematic Reviews and Evidence-Based Recommendations (Public Health Resources): http://www.thecommunityguide.org/about/methods.html ; the “Methods” section provides Systematic review methods for evaluating the effectiveness of health interventions.
  • SAMHSA Evidence-Based Practices Resource Center: https://www.samhsa.gov/ebp-resource-center; provides an online registry of more than 300 intervention programs using evidence-based programs. Use this resource to find examples of other evidence-based process evaluation.