Recipient Organization
PENNSYLVANIA STATE UNIVERSITY
408 Old Main
UNIVERSITY PARK,PA 16802-1505
Performing Department
Agricultural Economics, Sociol
Non Technical Summary
The Army Community Service (ACS) provides an encompassing and integrated approach to meet the unique and everyday needs of military-connected individuals throughout their lifetimes. The ACS Relocation Readiness Program (RRP) provides support to Soldiers and their families to help reduce challenges associated with Permanent Change of Station (PCS) moves. The RRP aims to minimize the stress of relocation by providing resources to streamline the moving process, increase preparedness for relocation, decrease financial strain, and facilitate awareness and connections to community and military resources. To date, implementation of key RRP processes and evidence of intended RRP program outcomes or cost-effectiveness have not been assessed; however, examining these components is required under DODI 1342.22. The Clearinghouse has been tasked, by the Army, with addressing these gaps in knowledge regarding program functioning, outcomes, and means of improvement. The Clearinghouse will conduct a process evaluation, outcome evaluation, and cost benefit analysis of the RRP.
Animal Health Component
100%
Research Effort Categories
Basic
0%
Applied
100%
Developmental
0%
Goals / Objectives
This plan will be consistent with standard evaluation practices and will involve a process and outcome evaluation. The Clearinghouse's objectives are to do the following: (1) build a strong, functioning partnership with all Army stakeholders; (2) develop a project plan that enables the Clearinghouse to conduct the most appropriate methodological and analytic approaches for the process and outcome evaluation, and (3) if RRP is found effective, conduct a cost-benefit analysis of RRP.?
Project Methods
First, in collaboration with partners DCS, G9, the Clearinghouse will develop a project plan to achieve all tasks noted below. This plan will be consistent with standard evaluation practices and will involve a process and outcome evaluation. The Clearinghouse's objectives are to do the following: (1) build a strong, functioning partnership with all Army stakeholders; (2) develop a project plan that enables the Clearinghouse to conduct the most appropriate methodological and analytic approaches for the process and outcome evaluation, and (3) if RRP is found effective, conduct a cost-benefit analysis of RRP. Second, the Clearinghouse will use a multi-tiered approach that will be a theory-driven evaluation plan to examine implementation through a process evaluation to determine the extent to which programs are provided to audiences in a manner consistent with how they were intended to be implemented (Munro & Bloor, 2010; Weiss, 1997). The process evaluation will also focus on the utilization of the materials (e.g., educational and marketing) and program components as intended and will determine whether the target population is reached (Weiss, 1997). The RRP process evaluation will include a sample of installations that represent the Army and will address implementation, including consistency and quality of services across installations, an assessment of the evaluability of program components, and coordination of RRP services with other services. Third, within 6 months after launching the process evaluation, an outcome evaluation will be initiated. The outcome evaluation will use information garnered from the initial phase of the process evaluation to identify and employ SMART outcomes (i.e., Specific, Measurable, Attainable, Relevant, and Time bound; Conzemius & O'Neill, 2005; Doran, 2008). Metrics from the outcome evaluation will be selected based on the alignment with the RRP and the ability of metrics to be monetized for a potential cost benefit analysis. A cost benefit analysis will be completed if findings from the outcome evaluation indicate that the RRP is effective. Thus, the outcome evaluation plan, if implemented, would directly inform the "benefit" element of the CBA. The outcome evaluation will be done across select Army installations (i.e., a mixture of sites with varying sizes, commands, and locations) to ensure the representativeness of the result for the Army overall. The outcome evaluation metrics will provide measurement of RRP effectiveness in achieving outcomes of strategic relevance to the Army. For the cost benefit analysis portion, benefits-mapping procedures that were demonstrated in prior work (Belfield et al., 2015) will be employed during the development of outcome metrics in the outcome evaluation. These metrics are identified as part of the outcome evaluation to understand how relevant outcomes should be evaluated/valued for the sake of economic analysis in terms of whether these outcomes can be directly converted to dollar amounts, can be indirectly valued or cannot be monetized. The latter option may include important outcomes that could be targeted through program services or could be better suited for cost effectiveness analyses, if measured. With ACS consultation, the Clearinghouse will proceed as follows - produce a brief report that indicates the Clearinghouse's capability to pursue CBA and specifies whether the findings from the outcome evaluation warrant CBA. CBA can provide crucial information in an economic evaluation by determining net benefits for the program. Potential obstacles include lack of directly measured costs, which could include the use of medical services or increased productivity. However, projection methods may be used to estimate costs based on precedence from prior studies, including cost-of-illness estimates and contingent valuation. Fourth, provided findings from the outcome evaluation indicate RRP effectiveness, a cost-benefit analysis will be conducted. However, if the outcome evaluation does not yield positive findings in terms of program effectiveness, then a comprehensive detailed plan of program refinement will be completed. The CBA will involve two components: (1) cost analyses and (2) benefit mapping and analysis. Cost analyses of ACS programs will be employed using an ingredients-based approach (Levin & McEwan, 2001). Clearinghouse researchers will map all necessary cost resources and inputs to implement the program. This mapping will be documented in figures that will be presented in a brief report and will include representation of the program ingredients as specified in original logic models of the program and inclusive of any new program adaptations. The Clearinghouse will consult with ACS RRP staff regarding the usable, available budget resource data and the non-dollar resources that are critical to the functioning of the program (e.g., space requirements, participant time). Projected value and opportunity costs will be applied where costs were not explicit in budgets. Further, analyses of costs will consider variation across contextual and participant levels. Determinant upon these analyses, the Clearinghouse will provide ranges of costs, for each program, that represent variability across settings. Cost analyses will be crucial to understanding the degree to which necessary resources vary and how efficiencies can be introduced to reduce costs at the participant level. As part of this, the Clearinghouse will also investigate what infrastructure factors are important to consider for program efficiency and sustainability (Crowley et al., 2012). This includes an assessment of staff required to deliver services and consistency of program delivery across implementations and over time. The cost analyses will be completed first in order to provide a full assessment of the resources expended per participant for each program.