Draft EM&V Process for NAESB Retail EE Task Force
(v. 1.2, Tom Simchak, Alliance to Save Energy)

The following is a step-wise approach toward EM&V generally based on major steps identified in the NAPEE evaluation guide.  Varying levels of detail are provided under each step; some are better described than others.  NAESB Retail EE workgroup members should decide level of detail to include under each step, including consideration of additional steps and sub-steps. Citations to existing protocols and guidance should be included as appropriate. Consideration should also be given to the order and arrangement of the steps, with the idea of creating a parallel flow diagram of the steps.  Are there items to add or delete?  Note that several steps refer to issues that the group has agreed to not address in this first phase work but are worth identifying in the MBP as part of EM&V and that may be considered in a future phase of work. Among these, net-to-gross and attribution issues, process evaluation, and market effects evaluation. Have at it.

1.	Identify Program and Portfolio  Objectives
1.1	Establish Objectives of the efficiency program broadly
2.	Set Evaluation Objectives
•	Impacts and co-benefits directly resulting from a program
•	Evaluators may also wish to consider process evaluation (assessing program delivery and means for improving it) and market effects evaluation (influence on market transformation)
2.1	Determine areas for evaluation measurement and verification.
•	An entire program could be evaluated or specific elements of interest could be evaluated. Or certain representative portions could be evaluated for extrapolation to a whole.
2.2	Establish metrics for evaluating program success
•	[[to be developed]]
2.3	Select evaluator(s), if applicable
•	Identifying appropriate certifications for evaluators may be first course of action.
2.4	Pilot innovative EM&V Methodologies and EM&V Methodologies for innovative programs, if planned and within project scope
•	Where new types or approaches of EM&V have been developed for a program, program or M&V administrators may wish to pilot these approaches with a sampling of participants. These may include both methodological and technological innovations.
2.5	Identify co-benefits for measurement, if within project scope
•	Co-benefits, or non-energy benefits, may include any number of categories that have been identified as desirable outcomes that are not directly related to energy use or demand. Greenhouse gas emission reduction, energy security, air quality, and economic development are all possible co-benefits of energy efficiency programs. Some co-benefits may be related to power, for example reliability and power quality. Program stakeholders may identify countless other benefits that could be accounted for in determining program performance. These could be intentional targets built into program design or merely fringe bonuses identified during the EM&V process.
2.6	Identify constraints to achieving objectives
•	For example, establishing whether necessary data is available or if there are barriers to effective measurement.
3.	Select gross savings evaluation approaches
3.1		Define methodology and performance metrics
•	This would be to measure and verify program performance, including data collection approaches.
•	Approaches may include measures of savings or deemed savings, various types of self-reporting, or economic modeling.
•	Means for control of error and uncertainty should be considered, along with determining acceptable levels of error and uncertainty. Ensure error is kept within acceptable limits.
3.2		Establish deemed savings estimates
•	Use of deemed savings estimates for common programmatic activities may greatly simplify M&V activities, but may introduce systematic error. Deemed savings estimates will need to account for regional variation, be it weather, behavior, implementation differences, or other factors. Savings may be deemed at a variety of scales, from individual actions (e.g. one light bulb) up to almost any scale (e.g. building, site, or even towns or cities), dependent on the scale of the overall program. 
3.3		Define M&V timeframe
•	Establish whether M&V activities will continue throughout the program or only establish an initial success rate to be used for subsequent deemed savings estimates; some form of M&V should continue for the duration of the program, even if only in the form of projections. As such, determining a program end point will generally be necessary (though this need not preclude counting of post-program energy savings). Ongoing monitoring programs or persistence studies may be developed.
4.	Define baseline scenarios
•	Baselines should establish pre-program energy savings estimates, including metrics for projected change in program area activity that would have occurred in the absence of the program (viz. the counterfactual). 
4.1		Establish performance, service quality, existing standards, and weather, if applicable.
4.2 	Analyze extant use and demand
4.3		Estimate projected costs and energy savings, particularly if cost-effectiveness is a program criteria or requirement.
5. 	Execute program
5.1		Install monitoring infrastructure, if applicable, at onset of program implementation.
•	Such infrastructure should include both physical and technological infrastructure (e.g. data loggers and other monitoring equipment) as well as the human and knowledge capital to operate and maintain a program monitoring regime. 
•	Depending on the structure and type of program being implemented, monitoring infrastructure may need to be installed before the program begins, or it may simply be more efficient to do so.
6. 	Calculate gross savings
•	Gross savings are those directly attributable to the program itself, exclusive of co-benefits and ignoring issues of free-ridership, etc.
6.1		Including, if applicable:
•	Annual average demand savings
•	Peak demand reductions
•	Coincident peak demand reduction
•	Demand response peak demand reduction
6.2	Quantify and analyze co-benefits, if within project scope, as per 2.4
6.3	Calculate net savings, if applicable (though out of the scope of this process document, some notes on net savings are included at the end)
6.4	Calculate uncertainty and evaluate error
•	Uncertainty should be reviewed in instrumentation, modeling, sampling, interactive effects, and estimation.
•	Systematic Error: the extent to which results are biased by systemic issues to over- or underreport results. 
•	Random Error: sampling error resulting from the fact that only a portion of total participants may have been included in measurement.
7.	Evaluate program based on M&V findings
7.1 	Compare objectives and actual, including M&V objectives vs. actual
8.	Report evaluation results
9.	Evaluate and respond to feedback
10.	Identify possible program and operational improvements for future implementation, if applicable
10.1	Coordinate and collaborate with administrators to implement program improvements, if applicable.
11.	Evaluate savings persistence, if within project scope
•	Savings persistence is determined by evaluating energy savings over a period of time after implementation, relative to savings determined initially. This would normally require repeated M&V activities over some periodic interval, e.g. annually. Methodology for persistence could mirror original project M&V, either exactly or based on smaller representative sampling. Savings persistence is measured as a percentage of first year savings. 
o	Ongoing commissioning-type activities could be undertaken to maintain program savings, adapted to evaluation of savings persistence M&V.

---

A note on Net Savings

Though outside the scope of this NAESB process document, determining net savings and net-to-gross ratios may be desirable or required for program evaluation. This would come into the planning process in step 3 and implemented once gross savings were determined in step 6.
•	Net savings is the total load or demand reduction that is attributable to the program. In calculating net savings, consideration should be given to program externalities which may increase or reduce the attributable savings. 
o	Freeridership – The percentage of program savings that would have occurred even if the program had not – generally as a result of actors participating in a program when they would have undertaken efficiency or conservation measures anyway.
o	Spillover – Energy savings that occur outside the scope of the program or program participants, but which occurs as a result of the program’s existence. E.g., after participation in a program, a participant may seek out other energy saving opportunities in areas not covered by the program or a non-participant may choose to undertake such activities on their own without being directly connected to the program’s activities.
o	Rebound – Behavioral changes in response to a program which offset the energy saving benefits of the program. E.g. a participant using an efficient product more because they know that it is more efficient. The so-called ‘Jevons paradox’ suggests that efficiency tends to encourage greater usage of a fuel resource.
o	Other externalities such as may have positive or negative impact on additionality.